I came across the bit of code posted below today while browsing Stack Overflow. The user who posted the question was asking what this bit of code actually did. He was aware that it was malicious due to the fact that it was on his server without his knowledge, and obfuscated. Unfortunately the question was marked as off topic, “Questions asking us to recommend or find a book, tool, software library, tutorial or other off-site resource are off-topic”.

At the moment my blog doesn’t have all that many posts on it, and I really don’t consider myself a serious blogger. I write when I feel like it, and in whatever tone I’m feeling like writing in at the moment. Odd as it may seem, I’m not normally writing with the intent of being read. This doesn’t mean that I don’t care when people read my articles. It’s especially good to receive comments and engage in discussion, but I’m not motivated to find as many readers as possible. I seldom share links to my blog posts on other sites, I simply write posts and visitors find them on search engines, or don’t find them at all.

I’ve been intending on migrating my blog to a static site for quite some time, for security, speed, and generally just being more minimalist. The primary issue preventing me from making the change was fear of commitment. Hugo looked like an excellent option, as did Hyde (not to be confused with the Jekyll Template). I decided to go with Jekyll simply because of the size of its community, and because of GitHub Pages (which I’m still not using until they support HTTPS for custom domains).

This article isn’t meant to discuss what web scraping is, or why it’s valuable to do. What I intend to focus on instead, is how modern web application architecture is changing how web scraping can/must be performed. A nice article discussing traditional web scraping just appeared in Hacker Newsletter #375 by Vinko Kodžoman. His article tipped my motivation to write this.

Traditional Scraping

Up until recently, data was typically harvested by parsing a site’s markup. Browser automation frameworks allowed this to be achieved in various ways, and I’ve used both Beautiful Soup and Selenium to achieve what I needed to in the past. Vinko discusses in his article another library lxml, which I’ve not tried. His explanation of lxml and how it interacts with the DOM is good enough to allow general understanding of the way scraping is performed. Essentially, your bot reads the markup, and categorizes relevant data for you.

It’s been about a year, a little over actually, since I started work on my main side project. The app is a motocross track directory, which isn’t something that doesn’t exist already, but I felt existing track directories were lacking a lot of features. This lead to me creating MapMoto.

An Idea

I ride motocross a lot, not as much as a few years ago, but a lot. I’m always looking up weather before I ride, looking for hot-line numbers to call to confirm days to ride, and looking for new tracks all together, especially when traveling. I wrote down everything I wished a motocross track directory would have, and came up with the follow list.