You’ve got a strong back—get a job!

Here’s the current roster of personal projects I’m working on in an effort to become more employable:

• A realtime, multithreaded Scala program which I'm building in stages but which I envision will probably end up at 10-20,000 lines of code--involving realtime computations against a NoSQL database and interaction with a SOAP service.
• A build-your-own content-management website á là mywedding.com—I’m tinkering with the Django, Drupal, and Catalyst MVC frameworks, trying to decide which will work best while getting some exposure to PHP 5, Python 3, and Moose in the bargain. This will include a MySQL backend, so I’m brushing up on SQL as well.
• Some Arduino toys including a couple of retro, 8-bit-style Gameduino games to polish my rusty C skills.
• A slightly more ambitious microcontroller project, this one using an ARM board and the mbed platform to do some real-world data acquisition and control of a mechanical system (that’s deliberately vague: more details will come when I start to get an idea whether it’ll actually work!).

(A gold star for those who know what classic ’90s movie the title of this post is a quote from.)

Determinism and market madness

The past few months of misery in the stock market have been so obviously driven by fundamentals that there’s something a little bit lightheaded and silly about saying a rebound is on the way because of technical factors. Consider the following chart (click to enlarge), which doesn’t even take into account the political and military complexities in North Africa:

Yes, I know, technical analysis is supposed to (depending whose explanation you read about why it works) operate independent of events external to the price of the stock, or perhaps to incorporate those factors in such a way that there’s no need to look at anything but the price; but there are philosophical holes in that explanation the size and stickiness of the La Brea Tar Pits. Leaving aside whether those holes can be patched (and, also, my apologies for the layers of mixed metaphor I seem to be building up here like a stale rhetorical napoleon), it’d be nice if we could look at this market death spiral we’re in and see it as something regular and predictable (and finite!). People operate in the everyday world by matching patterns and proceeding as if those patterns will continue, what Sherlock Holmes always called “deductive logic” but is really inductive; I don’t think they do it by following anything remotely like the rules of formal logic (I think modeling the brain as a computer running a program puts the cart pretty far out in front of the horse), but they do do it in a fairly consistent way. By making that distinction, though, I'm at odds with a lot of people in cognitive science, philosophy, and artificial intelligence.

Three hundred years ago, around the same time that Adam Smith was laying the ground work for those researchers by asserting that we were all members of the supremely logical species Homo economicus, his empiricist contemporaries elsewhere in Western Europe were coming out pretty strong in favor of a pure determinism: the world as giant clock. It’s a seductive way to think, and I admit to having the following quote from the Marquis de Laplace on my fridge:

We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at any given moment knew all of the forces that animate nature and the mutual positions of the beings that compose it, if this intellect were vast enough to submit the data to analysis, could condense into a single formula the movement of the greatest bodies of the universe and that of the lightest atom; for such an intellect nothing could be uncertain and the future just like the past would be present before its eyes.

The thing about Laplace’s brand of determinism is that it’s compelling on a really fundamental level because it’s how we live our lives day-to-day; if it weren’t so compelling, and weren’t so familiar, then why would the head-scratching, insomnia-inducing philosophical discontinuity between it and the concept of free will even be worth talking about? The steady encroachment, warranted or not, by scientists onto the traditional grounds of metaphysics has muddied the popular conception of what the free-will debate is about, by introducing the idea of quantum particles with unknowable and/or unpredictable states. But, woo-woo stuff like that aside, where determinism impacts our lives isn’t in the question of whether you or I, as conscious beings riding around in our heads and looking out of our eye sockets, act predictably to satisfy our needs, but whether all those other eye-socketed things around us act in a way that we can predict well enough to function. And it seems that they do. (This is a finer-grained version of the truism that we all, like it or not, are in practice practicing Rousseauians: without the Social Contract, how would you ever make it through five minutes on the freeway alive?)

In 1980 I was an avid reader of Creative Computing magazine, which, like other great magazines of the early home-computer culture of the late 1970s and early ’80s, published in each issue complete programs (not just illustrative snippets of code) that you could type into your computer, save, and run. Most were games, but many were serious applications, such as primitive databases; Compute! even published a full-featured word processor. Believe it or not, this was until 1990 or so (I think PC World was the last holdout, or maybe one of the Atari- or Amiga-specific magazines) one of the chief ways that non-commercial software was distributed (the other being BBSs). So, late in 1980 Creative Computing printed a program, with accompanying explanatory article, that was supposed to predict the outcome of that year's Presidential election. As a kid I thought this was magic. Via simulation games that had you playing a bullfighter or king of Sumer, I had gained an intuitive grasp of the way that software could model a small part of the real world; but I couldn't make the leap of generalization from those toy models to one that would, accurately enough to be interesting, simulate the behavior of 100 million adults. (Never mind that like most people under the age of 25 or so, I found the behavior of even a single adult totally incomprehensible, period.)

I'm reconstructing this from memory, of course, and I'd love to see a copy of the magazine again, or have someone email me saying that they wrote that program. I imagine that the algorithms used were predicated on some sort of deterministic optimism about voter behavior in the 1980 race, i.e., an assumption that similar factors were present as in prior elections. (Or were there? Carter’s single term is now remembered mostly for an unusually heated level of strife in the Middle East and an economy so screwed-up that a relatively new term of art from economics became common currency just so people could grasp what was happening.) Probably there was also a dash of statistics added in, since if I remember right the program was supposed to narrow in on its best guess as you entered numbers from the earlier-reporting states, just like the TV networks do. (“As X goes, so goes the nation.”) It’s this latter algorithm, or group of algorithms, that most intrigues me. It’s pattern matching of a sort; and pattern matching, no more and no less, is what technical analysis of capital markets is all about. When we see that A usually precedes B, after a while we begin to expect B when we see A; if we get what we expect, our predictive model is strengthened. It’s a little bit circular, but so is all inductive reasoning.

That circularity becomes a problem only when we mistake an A-prime, or a C, or even a Z, for an A, and have become complacent enough that we mistake the B-prime or D or unrecognizable squiggle that follows for a B—and then act as if a B is guaranteed. On that path lies madness, and something worse than madness: losing money.

"Shadow IT" is a symptom, not a disease

...specifically, it’s a symptom of an inability of an organization’s IT department to meet business needs. And not all shadow IT is created equal: the term lumps together (for instance) staff circumventing security controls on their workstations to install unauthorized software, with people creating complex Excel spreadsheets to answer their business-intelligence questions (rather than asking IT to develop or purchase a new software package). So a little bit of the handwringing by old IT salts that you’ll see if you Google the term is justified, but for the wrong reasons, as I’ll explain.

If people need to install software and have shown that they can do so without bricking their machines, then IT should relax security controls, encourage them to keep informal records of changes they’ve made to their configuration, but most of all stay out of their way and allow them to be more productive. Similarly, don’t keep people from solving their problems with Excel; if they’re solving their problems (period), ipso facto that’s a good thing and IT staff (not users) are the ones who need to adjust their attitude to fit reality.

Let’s focus on the latter example for a bit. Here’s where IT really has a good chance to support rather than obstruct. As I’ve argued before, Excel has long been less a spreadsheet program than a development environment, with more flexibility and usable power than many dedicated business intelligence tools and ERP systems. Consider Juice Analytics, my favorite gang of Excel wizards: If the sample code they give away for free in their blog is anything to go by, the stuff they charge for is a wonder of transparency, structure, and scalability—just like the best "real" software.

But if it’s really a development environment, however, Excel needs to be treated as such; and in turn its power users need to be treated as developers. What are the three most important support tools for developers in a mature programming shop? Source control, source control, source control. Excel source control tools exist—there’s a short but sweet discussion of them here at StackOverflow—but they certainly aren’t mature. The technical problem here is that Excel stores data and business logic (the latter in the form of formulas and VBA code) as one binary file; same thing for the BI and ERP tools I’ve used, which make you navigate through dialog box after menu to create a query, report, or whatever that’s then stored somewhere mysterious inside the software. If all such software stored everything the user created in a text file in a standardized parsable format (XML being the obvious candidate) those files could be manipulated just like the source code files in a traditional development environment (i.e. not just versioned but interactively or programmatically edited using the user’s favorite tools), and they’d be exposed within the organization to be shared, catalogued, backed up, etc. Again, a win-win for users and IT, provided the latter opens its mind to the possibilities.

The article that sparked this post? Ironically, it's on TechRepublic, the epicenter of an awful lot of that handwringing I mentioned above. "Decade of the Developer," indeed. Open source reshapes the organization itself?

Reverse-engineering dynamically-created JavaScript

This is interesting: a page I want to webscrape some options price data from appears to be entirely created dynamically by JavaScript code which itself is created dynamically by an unknown CGI backend (probably PHP). That seems a little bit kludgey, but I understand the reasoning behind it; the page is interactive, but there's a lot of data that can potentially be displayed. This way there is one big server hit when the page is first loaded (to get a snapshot of all the options data)—presumably that CGI code is querying a database—and the JS just displays the data or not as the user clicks show/hide for each stat or each block of options prices.

I can get the data I need by parsing the JavaScript code, if I can figure out how that code parses its data strings (i.e. the data "passed" to it by the underlying CGI code) for display; luckily, the JavaScript string-manipulation methods seem to be modeled closely on Perl's.

I'll then have the data in an elaborate Perl data structure, and can manipulate it as I see fit.

Another financial-data-mining/web-scraping/scripting exercise

Continuing the theme of the previous couple of entries, I'm thinking of writing a program, almost certainly in Perl, to scrape the options price and open interest data for any given stock from the Morningstar web site (example here) and analyze it in various ways. It seems to me that there are more ways, and sometimes simpler ways, than the traditional "greeks" to evaluate the resulting data set as a predictor of short- and medium-term stock prices.

Given the numbers on the Morningstar page, I should be able to compute:

• A simple put/call ratio: total open interest of puts over total open interest of calls.


• (Here's where my own ideas start) A put/call ratio where the open interest is weighted according to time until expiration—i.e., near-term options are given more weight since (maybe) they represent traders who have a larger stake in the game and thus are paying more attention to whether their bet will pay off.


• A similar ratio, but with the open interest weighted by how far the strike price is out of the money.


• Change in the price of the option and of the underlying stock. (Call these delta-o and delta-s to avoid confusion with the options greek called "delta.")

The last should be graphed over time. There's a nice module in CPAN already to build 2-D graphs (line charts, bar charts, the usual) given an array of data, and output them as image files ready to be tagged on a simple web page like I did in the previous entry. But you have undoubtedly already noticed, Careful Reader, that there are really two dependent variables in the graph I just described. (Yes, the difference and the ratio between delta-o and delta-s are interesting, but the individual stats are also interesting in themselves.) This calls for a three-dimensional graph. I don't see any relevant library on CPAN, but I'm sure a little digging will turn up something I can adapt. (In fact, I already have the book Perl Hacks, which has a nice explanation of how to do bitmapped graphics in a window with SDL; so all I really need, if I'm recalling Computer Graphics 101 correctly, are the equations for a projection of a three-dimensional point onto a two-dimensional plane from a particular relative viewpoint). Having done this, I've got another idea for which a 3-D graph—and preferably a dynamic one that you can "fly around" and look at from all sides—would be not just nice to have but mandatory: distance out of the money (y) and price (z) versus time to expiration (x). If the points are colored or shaded appropriately to indicate the sign and magnitude of delta-o, then a large solid-colored area would be a tipoff that a certain set of options with a similar strike price and time until expiration have gone up or down in asking price—quite possibly a significant leading indicator of the price trend of the underlying security.

The analysis of options is always—and this is a truism, but a deep and important one—complicated by the fact that for every buyer of options, there's also a seller. The line "Most options expire worthless" is often given as an argument that options buyers are mostly ignorant speculators whose bets don't pan out. I don't buy that. (If that were the case, the put-call ratio is universally interpreted in reverse; we ought to expect that the buyers of puts are mostly wrong in their predictions, so the price of the underlying stock will go up; and the inverse for calls.) I'd bet that most options are in fact both bought and sold not in order to speculate on the options themselves but in order to hedge a trade of the underlying securities. If that's the case, then something would be wrong if the options mostly didn't expire worthless. Most people rarely if ever make claims against their car insurance, too.

(Hey, there's even already a Perl module to do Black-Scholes options pricing...)

"Now seems like a good time," I said to myself...

..."to get those rusty programming skills going."

I had found myself wanting to do some analysis in Excel of price behavior of a large list of stocks.

I glanced at the first few pages of Perl and LWP, and then at the Regular Expressions Pocket Reference; I opened Firebug on the Yahoo Finance "summary page" for a stock I was interested in, so that I could see the raw HTML I was dealing with; and wrote the following:¹

---

#!/usr/bin/perl
use LWP::Simple;

# Expects a list of security symbols on standard input, one per line.

print("Symbol\tPrevClo\tOpen\tLast\n");

while ($sym = <>)
{
    chop $sym;
    $summary=get("http://finance.yahoo.com/q?s=$sym");
    die "Couldn't get Yahoo Finance Quote Summary page for symbol $sym!"
        unless defined $summary;

    $summary =~ m/>Prev Close:<.*?>(\d+\.\d+)</;
    $prevclose = $1;
    $summary =~ m/>Open:<.*?>(\d+\.\d+)</;
    $open = $1;
    $summary =~ m/>Last Trade:<.*?>(\d+\.\d+)</;
    $last = $1;

    print("$sym\t$prevclose\t$open\t$last\n");
}

---

It worked the first time—not bad for not having done any programming whatsoever for about five years and nothing of significant size for ten. (Yes, I know it's not very idiomatic Perl—combining the match regexps and doing a few other things would probably cut the line count in half.) That code took a few hours to produce, but subsequent similar programs to web-scrape other pages took much less time, now that I was in the groove.

For example, more exciting was the following, which expects the same list of symbols:

---

#!/usr/bin/perl
use LWP::Simple;

print "<html>";

while ($sym = <>)
{
    chop $sym;

    print "<font size=5>$sym</font><br>";

    foreach $period ("1d","1w","1m")
    {
        getstore("http://ichart.finance.yahoo.com/z?" .
            "s=$sym&t=$period&q=l&l=on&z=m&p=e5,e20&" .
            "a=p12&lang=en-US&region=US",
            "$sym$period.png");
        print "<img src=\"$sym$period.png\"/>";
    }

    print "<br><br>\n";
}

print "</html>";

---

What I'm doing here, if it isn't clear, is scraping a number of security price charts from Yahoo Finance, saving the image files locally, and building a crude but effective web page to make them viewable in one place. Beats looking at each stock by hand for price trends, let me tell you!

Now, all of this may seem like "Hello World" stuff to anyone reading this who's had any programming experience beyond Computer Science 101. But what I think shouldn't be taken for granted here is the amazing ability to (in the first script, as the simpler example) in just a few lines of code, suck an entire web page into a string variable, search that string in a complex way, and output the result in a universally readable (i.e. by humans or other programs) format. We used to want applications to have built-in programming languages—now (and here's the takeaway!) we have programming languages with built-in applications: very-high-level functionality to do things that only applications used to be able to do. And we can do them in a scriptable, redirectable, programmatic way. Admittedly much of this is due to the straightforward API of the LWP module (Perlspeak for "library"); but I'd argue that that accessibility is a function of the design of Perl; there's obviously stuff going on there behind the scenes that would be much harder to write in a language without such integral support for string manipulations (C, say).

I was weaned as a programmer on 1980s consumer 8-bit machines, the multimedia powerhouses of their day, on which even in a high-level language (built-in BASIC), to do anything interesting you had to twiddle bits. And most of the software I've been paid to write has been low-level stuff in C—device drivers and the like. So I'm easily impressed and easily seduced by VHL (very-high-level) languages that let you do so much with so little typing. Of course there's danger inherent in only knowing high-level languages. When you don't understand what's really going on at the machine level, optimization can be much more difficult, for example. Ironically, though, even as undergraduate computer science programs deemphasize C and assembler skills and move their students towards Java, C#, .NET, PHP, and so on—preparing them more effectively for the kind of web-back-end-database-interface work 9 out of 10 of them will face as new programmers—even as this huge and largely unremarked shift in what it means to be a professional computer programmer takes place, hobbyists tinker with microcontrollers, programmed at as low a level as you want, to recapture some of that early-80s frontier-machine-code feeling. Some will call this retrograde or Luddite-ish but the truth is, I think, that controlling hardware directly with one's code fulfills some kind of deep need in the engineering personality to exercise maximum control over one's immediate universe; and there's nothing wrong with the practical experience gained thus: few programmers will ever write an operating system, true, but there will always be lesser software that needs to run "close to the metal." (A $5 pocket calculator will never run a Java interpreter, for instance. I think...)

Getting back to my own programming for my own use and profit, far more complicated and wonderful things will come in time. I'm comfortable using Perl for this kind of stuff, but have never written a program of any serious size in it. What little user-level software development I've done has been fairly strictly object-oriented code in C++. I only know how to use Perl procedurally; understanding the OOP features of the language, which seem to be highly regarded, would be a good thing to have under my belt.

On the other hand, I have had a strong hankering to learn Python, thanks to what seems to me to be a very elegant syntax. And I have the book A Primer on Scientific Programming with Python which—while I'm quite sure that somewhere on CPAN there's a module to support in Perl the same kind of computations I need to do—numerical integration and differentiation, curve fitting, linear regression, etc.—is an excellent tutorial for Python in general besides describing the appropriate libraries in detail. Lastly, the Beautiful Soup library looks like an even cleaner way to do webscraping.

1. How the heck do you format code nicely (i.e. not just in a non-proportional font but also indented correctly, lines that overrun the margin indicated clearly, and with symbols correctly escaped) in idiomatic HTML these days? Yeah, I know there's the <pre> tag, but it doesn't help you with lines that run past the right edge of your text frame (or wherever your body text is going), and you still have to festoon your code with &whatever-entity tags to escape all the non-alphanumeric characters.[back]