Affiliate Network Automation
How much time are you wasting on data entry? Automation saved this client more than 25+ hours per week.
Overview:
A client in the affiliate marketing vertical came to us help solve a major problem for them. Their team was in a pickle; they were stuck manually importing data from more than a dozen affiliate networks. Not only was this task mundane and error prone, it was eating up more than 25+ hours a week to try to stay on top of it all. The process involved someone authenticating into each affiliate network, then running reports. Each report required filters to be applied (status, date ranges, etc) before running, exported out to a CSV, and then loaded into their system. Needless to say, this manual effort was a real time sink and exposed the potential for a lot of unforced errors.
To add even more urgency, the amount of work and effort was rapidly expanding as the client was in growth mode. With new merchants coming on weekly, they were constantly in catch-up mode. Not to mention what the costs of having stale data meant to the site's success. Missing out on the latest merchant and time sensitive deals was losing money for the business. Without timely updates, visitors would be lost to competitors with the latest data.
Devliverables on this project included:
- Automate affiliate network data pulls.
- Improve data quality.
- Save 25+ hours of manual work per week.
Read on to see how ThunderSteed freed up staff to concentrate on more important tasks and elevated user experience with fresh, timely data.
Issues:
The client was drowning in data. More than a dozen affiliate networks representing over 4,000 stores was a struggle to maintain due to the sheer amount of volume. This is not taking into account the coupons and deals for each merchant. Despite their best efforts, they were unable to keep up with the demand. Getting behind on the data feed was costing the business big time. In the ultra-competitive world of affiliate marketing, visitors won't hesitate to bail to a competitor if they can't find what they are looking for.
A list of challenges they were facing follows:
- Diverse API Structures: Distinct endpoints and parameters, data schemas and authentication protocols.
- Data Inconsistency: Variations in data fields across the networks required extensive mapping.
- Scalability: Manual imports are time-intensive and error-prone.
- Timely Updates: Fresh data was critical to maintaining user trust and site reliability.
With mounting pressure to get off of this hamster wheel, the client turned to us to help them tackle each challenge.
Solutions:
The challenges had taken a toll on the staff as they attempted to keep up with demand. The additional work load was leading to burnout, along with introducing errors from the manual processing. Management was feeling the pressure and stress, which only intensified their desire to find an optimal solution...and fast! But the good news?
All of the issues the client was experiencing could be solved with the help of automation. Most networks have rich API capabilities or at least, end points to retrieve reports, making it a perfect candidate for automation. The effort to streamline data ingestion would be well worth the investment. The automation would free up time for team members to focus on growing the business. Users would benefit greatly with fresh and accurate data, keeping them on the site to make their purchases.
Once we clearly defined the objectives and scope per the client, we performed a thorough API analysis and documentation review for each affiliate network. The purpose was to identify commonalities and differences between each network. Noted key differences in authentication methods for each network. Data formats varied by network between JSON, XML, and CSV. We came up to speed on the endpoints and parameters. Made note of error codes for appropriate handling.
Our goal was to automate the entire process. From fetching new data from the affiliate API endpoints to ETL processing to prepare the data for database ingestion. The API client development featured a modular approach, focusing on creating clean, robust, and reusable code. Pagination was common, which required logic to ensure the entire dataset was processed. Rate limits were also common, which required adherence to prevent being blocked from the service.
Data formats varied, requiring specific parsers to handle data the three common formats: JSON, XML and CSV. Merchant-specific attributes were mapped to align with the client’s data model. After parsing the data, it was cleansed, normalized and standardized before handed off to the database.
Summary of improvements ThunderSteed made:
- Automated affiliate data import.
- Ensured data integrity by removing manual intervention.
- Deep API integration
- Data mapping from source to target system
- Enhanced scheduling and orchestration.
Results:
- Increased Efficiency
- Increased Data Integrity & Accuracy
- Scalabile Across Networks
- Enhanced User Experience
- Asset Reallocation
To summarize, our automated process streamlined data imports leveraging API integration. As a result, the project achieved data consistency, integrity and accuracy by standardizing fields and performing validation checks. Due to the modular integration layer, the solution easily scales across a set of disparate affilite networks. The system was flexible to add or remove any number of networks. Job scheduling and orchestration allowed for customizing the update frequency as required. Replace stale data with fresh results, user experience increased, driving better user engagement by making latest merchants and deals available to the site's visitors. And the cherry on top? Boosted morale by saving more than 25 hours of mundane work per week, allowing them to focus on more important areas of the business.
