Webscraping awful JavaScript, part II

I edited the previous post to remove the link to the website I'm trying to scrape data from, in order to protect the guilty. What I am dealing with there is about 8000 lines of JavaScript, of which, I think, roughly 6000 lines comprise multiple blocks of code that are identical except for a loop condition, comparison, or other minor change. It's a classic if ugly C idiom, and sometimes unavoidable in a language so primitive. In a higher-level language like Java, C++, or JavaScript, it's the mark of a tyro—especially in JavaScript, which has first-class functions, which could be passed as arguments to a single instantiation of the repeated code.¹

Luckily I can sidestep the mess and just figure out the data-parsing code.

1. Douglas Crockford, in JavaScript: The Good Parts (hilariously, a little less than one-fifth the length of JavaScript: The Definitive Guide), argues that first-class functions are the best and most important thing about it: they make the language essentially "LISP in C's clothing." [back]

Reverse-engineering dynamically-created JavaScript

This is interesting: a page I want to webscrape some options price data from appears to be entirely created dynamically by JavaScript code which itself is created dynamically by an unknown CGI backend (probably PHP). That seems a little bit kludgey, but I understand the reasoning behind it; the page is interactive, but there's a lot of data that can potentially be displayed. This way there is one big server hit when the page is first loaded (to get a snapshot of all the options data)—presumably that CGI code is querying a database—and the JS just displays the data or not as the user clicks show/hide for each stat or each block of options prices.

I can get the data I need by parsing the JavaScript code, if I can figure out how that code parses its data strings (i.e. the data "passed" to it by the underlying CGI code) for display; luckily, the JavaScript string-manipulation methods seem to be modeled closely on Perl's.

I'll then have the data in an elaborate Perl data structure, and can manipulate it as I see fit.