Objective: Building a Live Product Feed with Automated Valuations
Has a task ever seemed impossible to you? At Maple we like a challenge and this is why we have been most excited by our latest ongoing project. We have been tasked with valuing hundreds of thousands of electronic items and their different variances, and keeping this pricing ‘current’; in other words live, and up to date. No easy feat.
Unless you have a team of dozens of people devoted entirely to valuing items 12 hours a day, to achieve this task with just humans is virtually impossible, and so in good old 2024 fashion, we have had to turn our heads to AI, and custom software. Given that we are in the AI revolution, it all seems rather fitting…
After the first 60,000 lines of code developed, and still going, we now have a sophisticated scripted algorithm which uses AI to reliably value Apple items (and other Brands on the way) with absolute precision.
Essentially this means we have built a reliable custom software which employs AI to value electronic items, and this has enormous advantages and potential for us and our customers.
Example: Valuing an Apple MacBook Pro
Valuing items in reverse commerce and e-commerce is one of the most complex tasks due to the sheer volume of products and the time required to determine accurate valuations, as well as ever fluctuating demand and pricing.
To illustrate, let’s consider valuing an Apple MacBook Pro 16-inch i9 2.4 GHz Space Grey Retina Touch Bar from 2019 with 16GB of ram and 512GB storage. I deal with Macs day in day out, so I’d have as fair a chance as most to provide an accurate idea – I could give a rough estimate off the top of my head but it would probably be out by £100, and what about those electronics I’m not so familiar with. A precise valuation requires a thorough process.,
- Market Analysis: Checking numerous retail outlets across the market to gather current price data.
- Depreciation Assessment: Evaluating depreciation data to understand the item’s value loss over time.
- Data Cleansing: Removing outliers and faulty items from the dataset to ensure accuracy.
- Nearest Neighbours Analysis: Identifying and analysing similar products to provide context and comparative data.
Nearest Neighbour Concept
The concept of a nearest neighbour involves comparing a product to similar items with slight variations in attributes. For example, if we lack data for the Space Grey model, we might look at the prices of the Silver model. This method helps in creating a more accurate valuation by using closely related products as reference points.
Challenges and Solutions
Valuing items accurately at scale presents several challenges:
- Data Volume: Handling large datasets efficiently.
- Dynamic Pricing: Keeping up with constantly changing market prices.
- Attribute Variability: Managing numerous product attributes and variations.
Our algorithm addresses these challenges through:
- Automated Data Collection: Continuously scraping and updating data from various retail sources.
- Advanced Filtering: Utilising AI and machine learning to cleanse data and remove outliers.
- Comparative Analysis: Implementing nearest neighbour comparisons to enhance valuation accuracy.
The complexity and scale of e-commerce and reverse commerce require robust automated solutions. Our algorithm not only simplifies the valuation process but also ensures precision and reliability, making it a valuable tool for navigating the intricate landscape of item valuation, providing an incredibly useful service for our customers.
Time Constraints and Automation Necessity
Initial Manual Process
We initiated the task by setting a stopwatch to zero, aiming to definitively value an item and produce corresponding buy and sell prices, all manually. For this task, we were given a model name and its specifications.
Data Collection and Preliminary Analysis
Against the clock, we began by browsing all relevant sites, collating the necessary data into a spreadsheet. This initial step, including browsing and data entry, took 7 minutes and 14 seconds. Next, we filtered out outliers, such as faulty or damaged items, and calculated customer acquisition fees to sensibly produce the buy and sell prices. This additional step, based on predefined formulas, took another couple of minutes. Overall, it took approximately 10 minutes to value one item, assuming all necessary tools were readily available.
Scalability Challenges
To put this manual process into perspective, our goal was to value items across more than 100 product categories, many containing over 1,000 variations of product specification variation spanning the past decade. This equates to:
- 1,000,000 minutes
- 16,666 hours
- Over 2,000 days at 8 hours per day
Given this scale, this was potentially time to quit. It would take over five years to manually value the entire product feed. Clearly, the value of the first item would likely change long before the entire valuation process could be completed. This exercise highli6ghted the impracticality of manual valuation and underscored the necessity for an automated approach.
Conclusion
The sheer volume and dynamic nature of the data demonstrated that manual valuation is not a feasible solution. Automation became essential to efficiently and accurately value items, ensuring timely and relevant pricing across the extensive product range.
Items are now valued in less than 1 second!
We discovered earlier that it takes a human approximately 10 minutes to confidently value an item manually.Â
Once we mapped out the manual process of valuation, we looked towards automating this entire process, including reading descriptions of items to rule out items which were faulty or damaged.Â
We can explain more about this process later, but as you can see in our video at the top of this post which demonstrates the valuation of a sample of MacBook Pros, the time of valuing is dramatically reduced to less than 0.1 second per item using our software, is more accurate than the human eye and makes far fewer mistakes.Â
To value one hundred thousand individual item SKUs with our scripted software equates to:
- 10,000 seconds
- 166 Minutes
- 2-3 HoursÂ
- This approximately 1/6000th of the time to value an item and is constantly live looking for the latest data.Â
While we can't give away our entire process, we will explain the core concept.
The Process – How exactly does it work?
Step 1: Writing Pseudocode
The initial phase in developing code involves drafting pseudocode. This serves as a blueprint, written in plain English, that delineates the intended steps and logic the final script will implement. Pseudocode provides a clear and structured approach to solving the problem before diving into actual coding.
Step 2: Data Collection
Our process commenced with an extensive search across the internet to aggregate all pertinent data. This foundational step involved using various custom scripted web scraping techniques and APIs to ensure comprehensive data acquisition. We understood that this needed to be constantly live, obtaining the most current data.
Step 3: Data Cleaning and Validation
In the manual data processing stage, our objective was to meticulously verify and cleanse the collected data. We aimed to eliminate any entries that were incomplete, defective, or contained discrepancies such as missing components or physical damages. To automate and enhance this process, we deployed AI algorithms in conjunction with Python regular expressions to systematically exclude any flawed data entries.
Step 4: Data Preparation and Transformation
Once we had a repository of verified data, the next task was to prepare and transform this data for seamless integration with our product feed. This stage involved sophisticated data parsing to identify and standardise key attributes. For instance, the code was designed to equate different textual representations of the same attribute, such as recognising that “13-inch” and “13”” refer to the same dimension. Additionally, the script could distinguish between storage capacities and other specifications, ensuring accurate attribute classification. Advanced looping constructs were employed to automate the completion of any missing data fields.
Step 5: Data Integration and Synchronization
With the cleansed and standardized data prepared, we proceeded to integrate it with our existing product feed. This complex task required continuous synchronization to ensure that any new or updated data entries were accurately reflected in the product feed. The integration process involved sophisticated matching algorithms to correlate the prepared data with specific products, ensuring that the correct prices and attributes were applied.
Step 6: Algorithmic Pricing
In the final phase, we processed the integrated dataset using a bespoke algorithm designed to calculate optimal buy and sell prices. This algorithm took into account various market factors and data trends to generate prices that we deemed reliable and competitive. The culmination of this process ensured that we had a robust and accurate pricing model ready for deployment.
Data Collection
Read Data (AI)
Clean and Mining Data (AI)
Merge Data
Process and Prepare Final Pricing
Browse through our AI generated priced Limited-Time Offers! Daily Deals You Can't Ignore:
Explore Our Wide Range of Apple Products: Browse Our Categories Now
Application and Future Scope
Our valuation tool is currently optimised to accurately and reliably assess the value of refurbished Apple products. However, it has the capability to extend its valuations to all major categories and brands of refurbished electronics and we are currently developing this.
This data is crucial for the development of our reverse commerce tool, allowing us to generate instant buy prices for all major electronic items and their variants.
Our eventual product will provide a buy and sell price for all electronics, allowing customers to seamlessly buy and sell their devices in the knowledge the pricing is very competitive and accurate. This will eventually aim to mimic the Bid and Ask price in trading, as we hope to commoditise electronics.
