28 January 2017

BookFetch scanner tested in the wild


After deciding to ditch Drupal 8 and completely rebuild BookFetch in Laravel 5 it took just 2 days to get the MVP online and ready so I could test the Raspberry Pi scanner in a real situation (not just a stack of books on my desk).

Heading to 3 local charity shops only 1 of them were not happy to let me record what books they had in stock.  I think the difference was my opening line to the staff. In 2 of the shops I introduced myself with something like "I'm doing a project to help charity shops sell books..." but in the 3rd vendor I opened by asking "Are you the manager?..." which I think must have instantly cast a mood of unauthorised activity over what I was doing. In the case of the non-cooperating shop the assistant disappeared out the back, leaving me to stand by the shelf of books which it was now assumed I was forbidden to touch. I overheard the concerned tone with which she described me to her superior, a male voice that mimicked her distressed argot and whose face never ventured from the stock room. She returned several minutes later with a phone number for head office and I left. Lesson 1: Selling isn't easy and your opening line is absolutely critical! Even when your product is free and still just an idea.

In the 2 shops that did let me test the system I was able to establish that all the books in a single shop could be scanned in about 20 minutes. Most charity shops have more in the stock room but I focused on the 200 or so that were on the shelf as these are usually a curated selection representing the cream of the crop. Lesson 2: 1 person using the Raspberry Pi scanner can scan at a rate of 10 books per minute.

A handful of books did not have a barcoded ISBN number so I skipped them. This project needs to have low-overheads and be fast for it to be viable so only the books with barcodes got processed.

Upon returning home and giving the Raspberry Pi an internet connection it started to post the data to the server. It got stuck after about 60 when an unusually long barcode was being rejected by the API giving a 500 Internal Server Error. I had to add some sanitation code that rejects barcodes that are obviously not valid ISBN numbers. Lesson 3: Some 2nd hand books have other barcodes, often on stickers, which can be accidentally scanned when working at speed.

Once the scan data was on the server the scheduled task started to lookup the titles on the ISBNdb.com API. Sadly, this final step was the most disappointing part of the test. Of the 389 books that I scanned across both the shops the system managed to lookup the details of just 154 of them, leaving 235 ISBN numbers that could not be found in the ISBNdb.com database. That's only a 40% success rate. Lesson 4: ISBNdb.com is far from a definitive source of data and other resources will need to be leveraged.

A few little bugs revealed themselves but they were easy fixes. Improving the Python script on the scanner so that it eases off the posts to the API when it receives a 429 Too Many Requests error was a good addition. It's as important to build a polite internet of things as it is to build a fast one.

All things considered today's experiment was very positive. It looks like the project could work.

Time for some shoutouts. A big Thank You to Matt Thurling for helping with the test and taking the short video above. Also thanks to the North Street branches of Tenovus and Children's Hospice South West, both of whom were happy to let me stand by their bookshelves making beeping noises for 20 minutes.