Are Your Sneakers Still Trending?
A short story about how and why I created this app to figure out what the trendy sneakers are.
Inspiration
When I was in college, one of the thing’s I used to do to make money was resell clothing and sneakers. A few friends and I would go to all the FootLocker’s, FootAction’s, and Supreme stores in New York City on a Saturday to collect raffles. Then after a weekend of hunting, we would get a call for one or two sneakers (like Yeezys’s) to be permitted to make a purchase. On average, I would make a 200% or more return on investment with the sneaker and clothing reselling.
With my ear close to the streets, I often knew what the hottest and most in demand sneaker/clothing brands would be. But ever since pivoting to my second hustle as a problem solver in the technology industry, I’ve fallen out of touch with what the streets want. So I thought, wouldn’t it be nice to know what the top trending sneaker and clothing brands were?
System Design
So you might be thinking, how did I decide what tools to use for this project? Well since this was really a quick adhoc project, I was quick to build and I didn’t think toooo deeply about them. I already knew that I wanted to work with a tool called BeautifulSoup (a library that makes it easy to get data from a website) so from there I only needed to think about where I would get data on trendy sneakers and clothing.
In the past I’ve used 2 platforms to offload products: FlightClub and StockX. Since I figured StockX was a bit trendier amongst my friend group, I figured I’d start with whatever data they had. With my initial investigation of StockX, I wasn’t able to find any public facing API’s. That being said, they had a trending page for the most popular sneakers and clothing that seemed to be updated regularly. Perfect!
So from there, I decided I would build an API that provides the user with the top trending sneakers. When called, it would request the top trending sneakers page on StockX. Then I would use BeautifulSoup to parse through the data and just give me a list of sneakers. Then I could write some method that would figure out the most commonly used sneaker brand amongst the top X sneakers.
But wait, what if several users want to access this API back to back? Having worked at big tech companies like Etsy, I know that if a company notices heavy traffic coming from an IP address they might just block the IP address from requesting data. With that knowledge, I wanted to implement some sort of caching solution. The caching would need to be server side so I could use the same data for anyone requesting it from my server as opposed to client side where each client would get their own custom data. From there, I decided to store this data in a NoSQL data store. This is because I could do very fast lookups based on a date as a key. And if I wanted to do some type of analysis with the history of trends, I would already have that data persisted.
The following system design diagram came out of this thinking.
Steps I Took to Complete
- I always like to get some type of Hello World on the screen with a new technical project. I might’ve gotten this from Intro to Computer Science way back when. So I started out by getting some boilerplate Flask code with an endpoint that returns the string “Hello World” when you successfully call it.
- I imported this library called urllib.request into my project. This was used to request the html for the top trending sneakers page. https://stockx.com/sneakers/most-popular. From there, I just logged the output to see if it worked.
- Once I got that working and was able to see something in the logs. I began to inspect the html of the page in my browser. I was looking for what common container StockX used to store data about the sneaker. This could be done easily by using the inspector tools on Google Chrome. I was able to find that the sneaker data was found in divs on the page with class called “product-tile”.
- With that, I imported the library called BeautifulSoup into my project. With this library, I was able to get a list of all the products on the most popular sneakers page for StockX. (Of course logging to verify)
- After this, I decided it was a good point to persist some of this data. I figured that it was a good SLA to tell users that the data would only be fresh for up to 24 hours. Given that, I decided to store the top sneakers calculated based on StockX in Cassandra. This of course means I needed to set up a Cassandra db cluster locally. Then I needed to connect to it in code so I could store and retrieve data from it.
- After doing this, I added some timers so I could measure the amount of time it took to get the top sneakers from the StockX webpage vs the Cassandra db. It was magnitudes faster retrieving the top sneakers from the database of course.
- Lastly, I wrote an algorithm that got the most common brand out of the sneakers and returned that via the API.
How You Can Contribute
The code for this project is publicly available! So if you read this and see some potential to this project and want to do something more with it, reach out to me and let’s chat! Or alternatively, you could just fork off it and build your own custom thing. Lastly, you could contribute to this directly. There are several bugs and new features that could be worked on. Below I’ll list out a few ideas I had on future changes that could be made to this project.
- Fix how we retrieve data from Cassandra. Right now I just get all top sneakers in the cluster but I should probably just get them by the date
- Create an interface that will display the top sneaker brands per day
- Create an algorithm for finding the most popular sneaker brands in the last X days
- Allow the API to take in a phone number or email so it sends a message with the top sneaker brand
- …
Thank you for taking the time out to read! If you found this story interesting, please follow me for more content and share with your network.
