Version 0.10.0 Released

The latest bot update mostly fixes bugs and does a lot of “behind of scenes” stuff. Please read the detailed explanation below for understanding why this maintenance was necessary (and, with that, why adding new shops is not “as easy” as you might think).

Changelog

• Fixed Sony Direct Crawler (Broken due to the shop switched to API calls)
• Fixed Gamestop Crawler (Broken due to broken stock recognition)
• Fixed Alternate Crawler (Broken due to the shop switched to API calls)
• Fixed Notebooksbilliger Crawler (Broken due to broken stock recognition)
• Disabled: MediaMarkt, Equippr

Please check the status page to see what products and shops are being crawled.

Detailed explanation (and why adding shops is not that easy)

Before talking about new shops let me briefly explain how my bot works in a simple way: it’s just a piece of software that “visits” a website and “reads” the given stock status. This means someone (aka me) has to exactly define how to get a stock status (“Look exactly at THIS position for THIS EXACT characters to be SOMETHING LIKE THIS”).

In short: my bot is pretty dumb. It looks exactly for what I’ve told to look for and reacts accordingly (e.g. announcing a stock change).

Shops are constantly being worked on and if a shop changes just a slight thing, like the physical location of where the stock status is positioned or how the “In Stock” label is called, my bot still looks for what is has been programmed for - and might no longer be able to get the proper stock status. Maybe I initially told the bot “Look if the ADD TO CARD button is there” and the shop suddenly changes and the button now reads “BUY NOW” - the bot, dumb as it is, still looks for the “ADD TO CARD” button. There’s no intelligence behind it, it just follows a set of instructions - it doesn’t understand what a stock even is, what it’s looking for or why it is doing that.

What I’ve done this weekend was I went through all the shops and checked if the bot still is able to acquire stock properly - and as you can see by the changelog many crawlers broke within the last few weeks. For example, in case of Alternate, they updated their entire website which works completely different from what my bot initially did to get the stock status. I had to rewrite the entire instructions for this shop to be able to get the stock status again.

I need to do this from time to time to make sure that I don’t crawl shops unnecessarily - but this takes a lot of time. To give you an idea: it took more than two hours to fix just the Alternate crawler.

By now you should understand why adding shops is not something I can “just do”. New shop crawlers need maintenance and every shop needs to be validated, refactored, fixed and tested which consumes a lot of time. If I’d add all shops on the “list of shops people want me add” just testing functionality of all of them would require probably multiple days - and we’re not even talking about fixing them in case they’re broken.

Last but not least: that’s also the reason I had to disable MediaMarkt and Equippr. Sometimes shops change in a way which makes fixing the bot really difficult. Again, an example: I’ve tried to fix the MediaMarkt crawler for nearly 3 hours without success. Since the bot is offline (which means ALL crawlers are doing nothing) I need to choose between disabling a single crawler until I was hopefully able to fix or to keep the bot offline for longer. The weekend is pretty much over and with monday on the horizon (and hopefully new stock to be coming in) I went for disabling them and come back to fixing them when all other crawlers are back online again.

I know that you guys would like to have all the shops out there to be tracked - and believe me when I say I’d like to do that. It’s just something which would exponentially increase maintenance and since I’ve got lots of other projects that also needs maintenance and development I need to be careful not to exaggerate.

• Support this project
• Follow me on Twitter

Please use the Tweet about this post for comments!