Determinism and market madness

The past few months of misery in the stock market have been so obviously driven by fundamentals that there’s something a little bit lightheaded and silly about saying a rebound is on the way because of technical factors. Consider the following chart (click to enlarge), which doesn’t even take into account the political and military complexities in North Africa:

Yes, I know, technical analysis is supposed to (depending whose explanation you read about why it works) operate independent of events external to the price of the stock, or perhaps to incorporate those factors in such a way that there’s no need to look at anything but the price; but there are philosophical holes in that explanation the size and stickiness of the La Brea Tar Pits. Leaving aside whether those holes can be patched (and, also, my apologies for the layers of mixed metaphor I seem to be building up here like a stale rhetorical napoleon), it’d be nice if we could look at this market death spiral we’re in and see it as something regular and predictable (and finite!). People operate in the everyday world by matching patterns and proceeding as if those patterns will continue, what Sherlock Holmes always called “deductive logic” but is really inductive; I don’t think they do it by following anything remotely like the rules of formal logic (I think modeling the brain as a computer running a program puts the cart pretty far out in front of the horse), but they do do it in a fairly consistent way. By making that distinction, though, I'm at odds with a lot of people in cognitive science, philosophy, and artificial intelligence.

Three hundred years ago, around the same time that Adam Smith was laying the ground work for those researchers by asserting that we were all members of the supremely logical species Homo economicus, his empiricist contemporaries elsewhere in Western Europe were coming out pretty strong in favor of a pure determinism: the world as giant clock. It’s a seductive way to think, and I admit to having the following quote from the Marquis de Laplace on my fridge:

We may regard the present state of the universe as the effect of its past and the cause of its future. An intellect which at any given moment knew all of the forces that animate nature and the mutual positions of the beings that compose it, if this intellect were vast enough to submit the data to analysis, could condense into a single formula the movement of the greatest bodies of the universe and that of the lightest atom; for such an intellect nothing could be uncertain and the future just like the past would be present before its eyes.

The thing about Laplace’s brand of determinism is that it’s compelling on a really fundamental level because it’s how we live our lives day-to-day; if it weren’t so compelling, and weren’t so familiar, then why would the head-scratching, insomnia-inducing philosophical discontinuity between it and the concept of free will even be worth talking about? The steady encroachment, warranted or not, by scientists onto the traditional grounds of metaphysics has muddied the popular conception of what the free-will debate is about, by introducing the idea of quantum particles with unknowable and/or unpredictable states. But, woo-woo stuff like that aside, where determinism impacts our lives isn’t in the question of whether you or I, as conscious beings riding around in our heads and looking out of our eye sockets, act predictably to satisfy our needs, but whether all those other eye-socketed things around us act in a way that we can predict well enough to function. And it seems that they do. (This is a finer-grained version of the truism that we all, like it or not, are in practice practicing Rousseauians: without the Social Contract, how would you ever make it through five minutes on the freeway alive?)

In 1980 I was an avid reader of Creative Computing magazine, which, like other great magazines of the early home-computer culture of the late 1970s and early ’80s, published in each issue complete programs (not just illustrative snippets of code) that you could type into your computer, save, and run. Most were games, but many were serious applications, such as primitive databases; Compute! even published a full-featured word processor. Believe it or not, this was until 1990 or so (I think PC World was the last holdout, or maybe one of the Atari- or Amiga-specific magazines) one of the chief ways that non-commercial software was distributed (the other being BBSs). So, late in 1980 Creative Computing printed a program, with accompanying explanatory article, that was supposed to predict the outcome of that year's Presidential election. As a kid I thought this was magic. Via simulation games that had you playing a bullfighter or king of Sumer, I had gained an intuitive grasp of the way that software could model a small part of the real world; but I couldn't make the leap of generalization from those toy models to one that would, accurately enough to be interesting, simulate the behavior of 100 million adults. (Never mind that like most people under the age of 25 or so, I found the behavior of even a single adult totally incomprehensible, period.)

I'm reconstructing this from memory, of course, and I'd love to see a copy of the magazine again, or have someone email me saying that they wrote that program. I imagine that the algorithms used were predicated on some sort of deterministic optimism about voter behavior in the 1980 race, i.e., an assumption that similar factors were present as in prior elections. (Or were there? Carter’s single term is now remembered mostly for an unusually heated level of strife in the Middle East and an economy so screwed-up that a relatively new term of art from economics became common currency just so people could grasp what was happening.) Probably there was also a dash of statistics added in, since if I remember right the program was supposed to narrow in on its best guess as you entered numbers from the earlier-reporting states, just like the TV networks do. (“As X goes, so goes the nation.”) It’s this latter algorithm, or group of algorithms, that most intrigues me. It’s pattern matching of a sort; and pattern matching, no more and no less, is what technical analysis of capital markets is all about. When we see that A usually precedes B, after a while we begin to expect B when we see A; if we get what we expect, our predictive model is strengthened. It’s a little bit circular, but so is all inductive reasoning.

That circularity becomes a problem only when we mistake an A-prime, or a C, or even a Z, for an A, and have become complacent enough that we mistake the B-prime or D or unrecognizable squiggle that follows for a B—and then act as if a B is guaranteed. On that path lies madness, and something worse than madness: losing money.

"Shadow IT" is a symptom, not a disease

...specifically, it’s a symptom of an inability of an organization’s IT department to meet business needs. And not all shadow IT is created equal: the term lumps together (for instance) staff circumventing security controls on their workstations to install unauthorized software, with people creating complex Excel spreadsheets to answer their business-intelligence questions (rather than asking IT to develop or purchase a new software package). So a little bit of the handwringing by old IT salts that you’ll see if you Google the term is justified, but for the wrong reasons, as I’ll explain.

If people need to install software and have shown that they can do so without bricking their machines, then IT should relax security controls, encourage them to keep informal records of changes they’ve made to their configuration, but most of all stay out of their way and allow them to be more productive. Similarly, don’t keep people from solving their problems with Excel; if they’re solving their problems (period), ipso facto that’s a good thing and IT staff (not users) are the ones who need to adjust their attitude to fit reality.

Let’s focus on the latter example for a bit. Here’s where IT really has a good chance to support rather than obstruct. As I’ve argued before, Excel has long been less a spreadsheet program than a development environment, with more flexibility and usable power than many dedicated business intelligence tools and ERP systems. Consider Juice Analytics, my favorite gang of Excel wizards: If the sample code they give away for free in their blog is anything to go by, the stuff they charge for is a wonder of transparency, structure, and scalability—just like the best "real" software.

But if it’s really a development environment, however, Excel needs to be treated as such; and in turn its power users need to be treated as developers. What are the three most important support tools for developers in a mature programming shop? Source control, source control, source control. Excel source control tools exist—there’s a short but sweet discussion of them here at StackOverflow—but they certainly aren’t mature. The technical problem here is that Excel stores data and business logic (the latter in the form of formulas and VBA code) as one binary file; same thing for the BI and ERP tools I’ve used, which make you navigate through dialog box after menu to create a query, report, or whatever that’s then stored somewhere mysterious inside the software. If all such software stored everything the user created in a text file in a standardized parsable format (XML being the obvious candidate) those files could be manipulated just like the source code files in a traditional development environment (i.e. not just versioned but interactively or programmatically edited using the user’s favorite tools), and they’d be exposed within the organization to be shared, catalogued, backed up, etc. Again, a win-win for users and IT, provided the latter opens its mind to the possibilities.

The article that sparked this post? Ironically, it's on TechRepublic, the epicenter of an awful lot of that handwringing I mentioned above. "Decade of the Developer," indeed. Open source reshapes the organization itself?

Not reinventing the wheel (again)

As I mentioned in two previous posts, I’ve been doing some 3-D graphics programming using the Perl interface to SDL. SDL is quite fast for 2-D graphics, but it doesn’t do 3-D on its own; so there’s a stack on my desk over a foot high of used textbooks with titles like Introduction to Computer Graphics Algorithms from which I’ve extracted the matrix transforms for various kinds of projections of a given pixel from threespace to twospace (orthographically onto a planar "window" given by a certain equation, seen in perspective by an eye located at given points, and so on). It’s been educational, but the results I’m getting are unusably slow, and I think they would still be too slow even if I were using a faster language. So, it’s time to start using a graphics library that supports 3-D—that is, one where someone else has already done the hard part of writing optimized code to do those transforms. It’s one of the hardest lessons to learn as a new software engineer transitioning into the workforce from school: it’s OK to use other people’s code; it’s better, in fact, to use other people’s code in the form of using existing libraries, rather than reinventing the wheel. (Jeff Atwood wrote an interesting contrarian blog post on why reinventing the wheel is sometimes, in special cases, the right thing to do: in general, the preference for reinvention is proportional to the need for performance; but note that there’s no consideration of cost there. Ironically, reinventing the wheel in the software sense is more likely to be favored by the ill-informed or non-technical manager, who doesn’t understand [the accepted wisdom] that it’s "always" better to buy than to build).

The obvious choice, one that’s been around for years with lots of support in the form of books, tutorials, and sample code online, is OpenGL with the GLUT libraries. The GLUT sample code I’ve skimmed seems to have (at a very high level) a two-part structure—you first set up the parameters for a 3-D rendering (the minimal case being an object and a view, with no light sources, texturemapping, or other complicated/cool stuff), and then you spawn a thread or another process to do the actual rendering on the fly until told to terminate. I’m not sure yet how this will square with the fact that I want the rendering to refresh in real time per the new data I’m collecting from the server. Let’s pull some numbers out of a hat: say I have a Perl program that runs all day long, sleeping most of the time, but waking up every five minutes to web-scrape and munge some derivatives-pricing data; say the latter takes about a minute, allowing for the bottleneck of my mediocre broadband connection. It sends that data on to the graphics code, which creates an interesting 3-D graph or visualization¹ in n minutes. Then that 3-D image pops up in a window, rotating slowly so that you can see it well enough to interpret it. If n<4, then the whole thing can repeat ad infinitum. If n>4, then the rendering is going to lag behind (and more behind, and more behind...) the data collection, and become useless. Depending on how GLUT works, I might be able to kill two birds with one stone—minimize n, and minimize CPU load—by avoiding the real-time render entirely: I’m hoping that I can actually have OpenGL render the frames of a smooth 360-degree rotation of the graph from start to finish as image files. Then these could be converted to an animated GIF (if they aren’t already) and shown on a web page containing other useful data (which itself could get updated every five minutes, continuing the example, or more often). Meanwhile, the Perl code sleeps again, and then wakes up and continues the cycle...

I want to have the possibility of running this code on OS X, where I spend most of my time, or on Linux, or on Windows 2000 or Windows XP (which I run in virtual machines here and there for various reasons). Toward that end, in turn, I’m requiring myself to write the most portable code possible: since OpenGL and GLUT are (intended to be) entirely platform-independent within the limits of the hardware they’re run on, there’s no sense in doing anything else. To refamiliarize myself with the Microsoft Visual Studio IDE² I’ll be coding my GLUT app as a Win32 application in C++, if necessary using some simple but not too kludgy wrapper classes around C code full of function pointer callbacks that’s sadly inherent to the Win32 libraries (well, I guess I can give them a break... it was the ‘90s.). However, if the instructions I’ve found for writing cross-platform-safe GLUT code for Linux and Windows using VS 2005 are accurate, that shouldn’t be a problem.

But I’m also going to try writing the whole thing in POGL, the Perl OpenGL/GLUT bindings, which are claimed to be just as fast as using OpenGL via C or C++ or <insert compiled language here> That makes the parsing that I referred to here easier since I won’t have to call Perl from a compiled C++ program (obviously a performance hit to be avoided) and I won’t have to find a regular expression library for C++ (of which I'm sure there are plenty, but the differences in regexp syntax between them and Perl would probably be unpleasant).

1. The form of which is a secret—I can’t give away all my tricks, can I? However, one of the books in that big pile is a collection of academic papers called something like Topics in Scientific Data Visualization, so you can guess I’m headed into some pretty wild territory. (Here be dragons!)[back]


2. One of the few truly great pieces of programming to come out of Redmond, it pains me somewhat to admit, along with Excel 2003 and the old QuickBASIC compiler (the distant MS-DOS ancestor of Visual BASIC, for you young whippersnappers). [back]

My take on "best practices" for hiring programmers

Do the majority of managers hiring for programming jobs think like Justin James?

I can't find a page to link to right now that sums it all up, but I have the impression that the way the wind is blowing right now in the programming job market, there are a lot more jobs than there are qualified people. The number of computer science graduates is way down now from a peak ten years ago during and shortly after the so-called dot-com boom, and while the slack has somewhat been taken up by people who, however well-intentioned, think they should be instantly hireable because they whizzed through Teach Yourself Javascript in 24 Hours in only eight, that's little consolation.

As a hiring manager, I assume you'd want first of all to screen the latter group out; optimally, you'd want to select from what remained (perhaps 25% of applicants, charitably) the tiny fraction (less than 1% overall) of truly smart, capable, competent people. But James' approach in the blog post I linked to above seems designed to hire not the great but only the good enough. What I'm bothered by in particular is James' expectations for the candidate's understanding of technologies inside and outside of the mainstream. It's a long quotation but I think it's worth it:

I am not hiring Lisp, Prolog, Erlang, APL, Scheme, Clipper, PowerBuilder, Delphi, Pascal, Perl, Ruby, Python (forgive me for including those four in this list), Fortran, Ada, Algol, PL/1, OCaml, F#, Spec#, Smalltalk, Logo, StarLogo, Haskell, ML, D, Cobra, B, or even COBOL (which is fairly mainstream) developers. If you show these on your resume, I will want to interview you just for the sake of slipping in a few questions about these items. I am serious. As part of my secret geekiness, I am really into obscure and almost obscure languages and technologies. I know that a lot of those items take better-than-industry-average intellect and experience to do; they also provide a set of experiences that gives their practitioners a great angle on problems. While you will never directly use those skills in my shop, you will be using those ways of thinking, and it will give us something to talk about on your first day.

There's more than a whiff of ambivalence here, isn't there? Anything in that list is "obscure" and "you'll never directly use those skills," although "you will be using those ways of thinking." The subtext here is that a good programmer, a hireable programmer, is one who knows the languages and technologies that are currently the most popular (prima facie the stuff that isn't in James' list in the quotation above); anything even a little bit outside of the mainstream isn't a marketable skill, though it might make for good talk around the water cooler. And James is probably about as liberal and accommodating as it gets. Another manager might be more plain-spoken: "Well, it's nice if you know LISP, 'cause that shows you're a geek who programs for fun when he goes home; but you really need to know dot-NET 'cause that's where the money is."

As I suggested a few paragraphs back, I object to this mentality because it places a premium on being a cut-'n'-paste programmer over being one who actually thinks about what he or she is doing; on being a good-enough programmer who understands enough of the current flavor-of-the month technology to get by, rather than being something qualitatively different, the great programmer who knows that technology and its limits, but also knows others and their limits; it's like the difference—yeah, I know these programming-as-cabinetry-or-whatever-physical-craft analogies have been done to death, but bear with me—between someone who puts together furniture on an assembly line, versus someone who hand-crafts each piece individually; maybe it's even the distinction between knowledge and wisdom.

Grand claims? Impractical stuff? Expectations with nothing to do with the reality of the job market? Well, go read Joel Spolsky's blog post on the same subject, and get back to me.

Done? OK. What I wanted you to get from that is that Spolsky also hires programmers; and he takes one sentence to dismiss James' emphasis on job candidates' knowing the flavor-of-the-month tech. For Spolsky, being a good programmer is all about aptitude. What skills should a good programmer have? Wrong question. Distinguish "skills" from "familiarities," and discard the latter; what programmers need, among other things, is a knack for simultaneous application of logic (the real "guts" of programming whatever language or library you're using) and creativity (because the best solution to a software problem is so often outside the box). This doesn't mean lack of hands-on technical knowledge— quite the contrary: it's a wide knowledge of tools, and when to use the right one for the job, that makes that famous 10x difference in productivity that a great programmer has over a good one ¹ manifest. It's what distinguishes software designers from software engineers; programmers from code monkeys; IT from CS (pick your terminology). As in the old saying, it's the state of being a man who's been taught how to fish.

So, let's say I'm a hiring manager. What kind of concrete test can I use in the real world to determine whether an applicant is a great programmer, or has what it takes to become one (probably an even more preferable case since I'll then have a hand in their professional growth and can shape it to my needs—oh dear,that sounds more sinister than I meant it to!)? Interestingly, Spolsky in the post I linked to above dismisses out of hand the kind of brainteasers that Microsoft interviews used to be (maybe still are?) famous for (so much so that a micro-industry sprang up with its own corner in the job-searcher section at Barnes & Noble, dedicated to help prepare you for the inevitable). He does offer a couple of positive suggestions: make the candidate write some real code on the spur of the moment (although I'd be a little less stringent about the requirements than he is, and a little more forgiving of bugs); ask the candidate to discuss (not "solve") a story problem, what Jon Bentley in Programming Pearls referred to as a back-of-the-envelope calculation.²

Those are good, but I can think of a couple of screening tests that perhaps bring Spolsky down to earth—that is, they speak to the candidate's approach to programming, but they also explicitly involve the tools of software development.

• One would be something like—if hiring for a web frontend position—"What text editor would you use for making some quick changes in a CSS file?" The answer "What's a text editor?" would be an immediate fail. But so would "Notepad." Why? Because it indicates a lack of the programming "gene" in a couple of different ways: a lack of understanding of the toolset available to today's programmer (what? you mean there are other text editors?); an indifference to version control (it won't break anybody else's code if I just make this little change); a lack of sufficient creativity to see that a tool like Notepad could be vastly improved for the specific needs of programmers (source code coloring? paren matching? huh?).
• Another would be to test the applicant's willingness to use appropriate technology specifically in their code. Again, this is job- and applicant-skill- dependent to an extent, but here's an example: Applicant has recently graduated from a well-regarded CS program. I describe to him/her a scenario in which a client, having been happy with a custom-built interactive software package for manufacturing process control sold them by our company, now wants to be able to automate the process further by integrating some kind of scripting language. Does the interviewee say, "Well, if it's a command-line tool, a shell script should do it; if it's an interactive text-based application, that's what Expect is for; if it's a graphical application things get more complicated." Or does he/she say, without missing a beat, "Lex and Yacc!"? Or, "Well, how complicated a language are we talking about? Lex and Yacc are probably overkill; I could build a little parser more quickly by hand, and then just traverse an abstract syntax tree in memory... [followed by a flurry of thinking aloud how that might be done]." Any of those three could be right, depending. But anything that smacks of reinventing the wheel would be wrong.³
• A third would be to ask the applicant to describe a hobby or other spare-time interest that involves problem-solving, and prompt until you hear a sufficient level of detail that indicates that this person has his logic hat on not only when programming but at other times, and that it feels natural. For instance, I like to work on old cars, specifically European cars from the '80s, for fun; and if you get me started I will talk your ear off about how those kind of cars are modern enough to have electronic engine controls but not modern enough to have the kind of computerized self-diagnostic abilities that new cars have, which presents a challenge, when the car doesn't run right, that is a lot like debugging code. But I'm not sure questions like that about the candidate's life outside the job can be presented in a non-EOE-violating way.

1. Originally from The Psychology of Computer Programming, I believe, this number has been pretty much taken as fact since the first edition of McConnell's Code Complete. In other words, don't blame me. [back]


2. Besides the Bentley book (and its sequel, More Programming Pearls), a great source of these is John Paulos' book Innumeracy. [back]


3. At a former job I had in which several programmers were working on a C++/MFC app, the need arose to do some complicated pattern matching in an input string. One programmer went home and spent three hours after dinner writing some character-by-character algorithms, basically C string manipulation at its goriest. The other added a public-domain regular expression library to the project, then wrote, debugged, and thoroughly tested a regexp to do the same thing—in 15 minutes. There's your order-of-magnitude difference in productivity. [back]

I'm overdue to learn Java

Part of my reason for writing this blog is to document my reëducation in programming (and, reciprocally, to encourage myself to take that process seriously). I've just ordered a general Java handbook for experienced programmers and a book specifically on graphics and audio programming. I'd like to port (or write shameless copies of, depending on your point of view) some classic 8-bit games to Java, with the aim of making them sufficiently platform-agnostic that they will run on mobile devices just as well as on desktop OSs. That little trackball on the BlackBerry is just crying out to be used to play Centipede, Missile Command, or maybe something a bit less well known like Crystal Castles:

I think my first significant Java program, however, is going to be a machine that plays Terry Riley's modern-classical composition In C (differently every time, of course). And there'll be some kind of kaleidoscopic graphical accompaniment showing which notes each "musician" is playing when. Yes, I'm aware of how ridiculous this may seem, but what's wrong with it as an exercise in generating graphics and sound in sync?

The last time I was paid to write software, it was a couple of medium-sized programs in C++ that ran in user space on Windows NT but worked hand-in-hand with a custom driver running (of course) in kernel space. One of those programs that I'm particularly proud of was an interpreter for a "little language" that enabled automatic testing of some proprietary vertical-market hardware. After that job, I taught C++ to undergrads for a while, with an emphasis on OOP fundamentals and how to apply them properly, not just on the syntax of the language. So I don't expect any trouble becoming fluent in Java, since the concepts of C++ are still rattling around in my head: multiple inheritance, operator overloading, virtual functions, references, and all that cool stuff. (I'm rusty on C++ syntax, but that will come back quickly with exposure if necessary.) Still, this will be far more of a brain-stretcher than the Perl code in the previous couple of posts—if the two can even be compared: in the Java case we're talking about brushing up on the effective design of a medium-sized program, while the tiny Perl web-scrapers I presented had a trivial structure (and what little structure there was, was procedural, and I've been doing procedural programming for thirty years).

Coming soon: my resume

Just what the title says! I'm open to offers of permanent or contract positions doing challenging, interesting work in any of the areas I touch on in this blog (see the subtitle above if what I mean isn't clear), as well as in related fields such as education, training, technical writing, etc.

I am particularly interested in interdisciplinary work such as business intelligence software development, data mining, or instructional design, but the bottom line is this: if, after reading some or all of this blog, it sounds like I might be interested in doing something for you, then I probably will be.