First of all: this is no stock availability update. Sorry if I got any hopes up, but since this little Sunday project kinda took off overnight there are some things I’d like to talk about. If you’re not interested in future updates regarding the bot itself you can simply ignore this and just follow the bot to get notifications for stock change updates.
I’ve been completely overwhelmed by the feedback of this project. I wanted to create myself a tool to prevent me from smashing F5 all day long - as the idea progressed I thought that actually many people could benefit from that and this ultimately lead to the bot as it is in its current form.
Within the last 24 hours I’ve received hundreds of messages, tons of feedback, requests and the bot gained more than 4000 followers. Thank you!
Within a few hours I’ve got many requests regarding adding more shops - from italian Amazon over Newegg and shops I’ve never heard before.
Adding more shops is of course technically possible but there are some things we need to keep in mind.
First of all: every shop added increases required performance of the bot. To no ones surprise servers are not free and my personal server is not even meant to run it in its current state. But it also didn’t die today, so I guess we’re good so far.
Another thing is that many people asked for adding entire shops, like “italian Amazon”. That’s not how my bot works: my bot requires to have a product page or any kind of stock info regarding a single GPU. I can’t just add the italian Amazon - I explicitly need a product page.
Last but not least: what shops to add? Answering that requires a bit of research and actually evaluating requests per shop. I don’t care if I add Newegg before some UK Shop - but if there’s more people who’d like to see the UK shop added before I’ll simply do as the majority wants.
That being said: I will add more shops this week. I don’t know which ones yet, but I’ll let you know.
Crawling is always a critical thing; it puts additional pressure on servers. Requests by such bots cost server performance which cannot be converted to revenue. Hence serving such bots is not a high priority for shops and can even be seen as an annoyance.
When I initially started the bot I set the delay between each stock scan to 30 seconds - or, in other words, every 30 seconds the bot visits the shops and scans for stock updates.
Today at around 8 in the morning this straight led to the bot hitting the rate limit of Caseking. Simplified this means that the bot was no longer able to gather stock information on Caseking and I had to disable Caseking crawling for the rest of the day. As of now it’s up and running again, but to prevent this from happening again some things had to be done:
First of all the delay between scans had to be increased: I’ve increased it from 30s to 60s. It might be increased even more, but this is a subject of testing, since there’s - understandably - rarely any information about web server confirmation of shops.
The next thing that needs to be done is improving the implementation. In case of Caseking this means that instead of crawling every GPU one by one I could just crawl the product overview page - which would result in just one request instead of 19 every 60 seconds.
The sad thing about hitting the rate limit earlier this day: nobody even noticed - because there’s no way to notice. That’s the next thing I’m going to implement: a way to see the crawler status and if everything is fully operational.
I’m most likely not using the Bot itself to tweet about failures since this may lead to useless feed entries and instead setup a dedicated status page for it.
Also within the first day there has been multiple GitHub pull requests to add more features. I’m really happy to see that and will take a look at all of them as soon as I find time for it! Thanks again for this kind of help!
Please use the Tweet about this post for comments!
A project by nehalist.io
Legal Notice / Privacy Policy