Java and the growth of the software business: The more things change...

I’m at a position career-wise where it’s to my advantage to broaden my technical skills before I deepen them any further, so Java is in the to-be-learned queue for real this time (as opposed to in previous posts in which I wrote “I really ought to learn Java” and then went back to banging out backwards ifs in Perl). I’ve been balking at putting Java on the front burner for a while, and for a while I didn’t know why.

Now the reason has finally bubbled to the surface: it’s that Java seemed like a toy back in 1998 or so when I was first programming professionally. Its speed was an issue; the programmers I hung around with were hardcore old-school bare-metal types, and our attitude was that you wrote systems code in C or assembler, applications code in C or C++ (using the Win32 or Microsoft Foundation Classes libraries), and anything else was too slow for serious work. Java’s promise of platform agnosticism seemed like marketing smoke and mirrors, and we couldn’t get past the fact that it was interpreted. (Interpreted? Interpreters were for scripting languages, and you couldn’t write a real application in those, either—more words to eat as Perl and Python matured!)

I’ll get back to the speed issue in a bit. What’s really changed my mind the most about Java is the sheer amount of support for the language in the form of third-party libraries, on top of the staggeringly large, well-thought-out platform that Sun provided from the get-go and Oracle has continued to enlarge and refine. I say “well-thought-out” deliberately; the Microsoft Foundation Classes sounded like a good idea in 1999, but in practice their functionality overlapped with the C++ standard library and the older APIs that Windows programmers were already familiar with, and the typical MFC program ended up being a mashup of all three. (Which is why MS did a good and brave thing when they scrapped all the old cruft of MFC and Win32 and built .NET from scratch.)

The particular example that tipped the balance isn’t so important as such, except that it illustrates the fine-grained libraries that are out there: it’s the jzy3d package, which provides the kind of three-dimensional data visualization that a year ago I was trying to build by hand. And this brings us back to the question of speed: jzy3d is built on top of JOGL, a Java API for OpenGL; and one of the example programs linked from the JOGL site is a port of the game Quake2 that runs in the browser. I was flabbergasted to see that. Quake, for you young-uns, was a ground-breaker in immersive 3-D graphics on its release in 1994 or so; to run it smoothly in high resolution took a high-end CPU (I had a 100 MHz Pentium, pretty good for the time) and a $300 graphics card; it ran best in DOS, needing to hammer on the metal directly for maximum speed; and it’s still a benchmark for low-powered systems (is there a port to the iPhone? I don’t know, but I’d be impressed if there were). As far as I’m concerned, if Java is fast enough to run Quake, then it’s fast enough to run any typical desktop application.

And I think that’s what the majority of the software business thinks, too. Department of Labor stats and the Wikipedia page on software engineering show a doubling in SE jobs since 2004, and that’s despite outsourcing and the 2008 economic crash; what’s more, that statistic doesn’t account for the number of programmers who don’t have the title of software engineer. I’d suspect the true level of growth to be quite a bit greater.

And though I can’t find any information as to what language the majority of these people are coding in most of the time (and if I could I’d suspect it of having been rigged by someone with a vested interest, like Oracle or MS... though here is an attempt from 2006—look at the numbers at the end of each section), I bet Java accounts for much of that growth. The language is just that good, and that effective, and that well-supported. It's enabled many more programming jobs to be created, because it's lowered the barrier to entry. Let me stress that I don’t mean that as a euphemism for “Now you don't have to be as smart to be a programmer, so a lot of stupid people are getting good white-collar jobs that they wouldn't have otherwise.” Rather, a flatter learning curve for mainstream software development is a universal win: it means more jobs, better-working software, and a more robust world economy, thanks to, for one, the continuing adoption of general-purpose hardware running custom software in industries that previously relied on custom and/or archaic hardware for process control. The more things change, the more they change!

Ten years ago people were calling Java “C++ for dummies.” Well, guess what? That isn’t an insult any more. People are coming to realize that C++ can be a real pain in the butt. It’s hard to write maintainable, reliable code in C++. You can do it; I can do it, and I have done so in the past. And I want to make clear that where speed is of the absolute essence, C++ is obviously a better choice (although I think the complexity-vs.-features tradeoff of using C++ vs. good ol’ C ought to be be carefully considered in those cases). There's at least one excellent API for user-level software in general use that works best with C++, Qt, and if there's one thing you can say about Qt programs it's that they are snappy! And, undoubtedly no coincidence, the API is a delight, very clean and lightweight—it reminds me of programming BeOS. (With the speed and memory of the average consumer-level computer being what they are these days, it’s ridiculous that we still have to put up with sluggish “click-and-wait” UIs... but that's a topic for another post.)

In practice, most C++ code, I would bet, is written using a subset of the language with which the programmers on any given project feel more comfortable, ignoring the really tricky features (no multiple inheritance, for example; maybe no operator overloading) and using some sort of integrated garbage collection scheme (overloaded new, for example) rather than relying on the programmer to manage dynamically allocated memory. I didn’t realize until I took a good look at Java (if you must know, I’ve been reading the first 200 pages or so of Java in a Nutshell a bit at a time before bed for a while now) that that comes very close to describing what Java syntax is: a slightly more verbose, less complicated version of C++. (And of course it has garbage collection, although not tunable for the needs of real-time programming, as far as I know.) QED: There’s nothing wrong with that. (Really, who could possibly take offense at replacing =0 with virtual, or : with extends?) Unless you need that last bit of speed that only a true compiled-to-assembler language can give, I can’t see anything wrong with Java as a baseline for applications programming.

Perl Ate My Brain!

Looking back at my earlier posts here I see a lot of optimism about rebooting (an irresistible pun) my ability to write software; a year or so ago, I had a plan to learn some new languages, notably Java, and refresh my knowledge of existing ones, notably C++, by embarking on a few different programming projects. It seems I’ve taken a slightly different route. I’ve done a lot of programming over the past year, but it’s almost all been in Perl. Hardcore Perl fans would say this was inevitable—that the language is so powerful and flexible that, once you start using it, there’s no need to use any other. People with a lower opinion of Perl would be more likely to call it a sickness.

What’s so seductive about Perl? A few things, to start, off the top of my head:

• The astonishing variety of freely available modules (Perl-speak for libraries) for highly specialized tasks: more than a few times I’ve girt my loins for a difficult bit of coding, only to check CPAN and find that someone else has already done it for me.


• Flexible built-in data structures and the ability to easily create more elaborate ones when the built-ins won’t do. Perl’s built-in lists, hashes, and references seem to capture the best features of the C++ Standard Template Library, Lisp lists, and hand-built C data structures, with performance (thanks to a highly evolved and optimized interpreter) fast enough for most purposes.


• Syntax that, once a slight learning curve is overcome, feels natural and is (surprise!) easy to read. Perl’s reputation for looking like line noise comes from two things: the idiomatic use of regular expressions as the most effective way to parse string (and often numeric as well) data, and the way that non-alphanumeric characters are used to mark variable types. The former did encourage “write-only” code in Perl 4, but in the current implementation of the language, you can interpolate comments in regexps to make them as readable as you want. The latter becomes more rational once you realize that the dollar, percent, and at symbols are used to identify a variable’s type not just at declaration, but at use, in context; they aren’t just an uglier cousin of Hungarian notation but a concise expression of an extremely elegant kind of dynamic casting. There are a few rough edges—for example, the backslash and curly braces are overloaded for references (sort of like C pointers) in a way that makes me uncomfortable—but on the other hand, there are only so many non-alphanumeric characters in ASCII. Perhaps Perl 6 should move on to use some interesting Unicode symbols, or (I’m only mostly joking here) something like APL… (In the meantime, the plethora of dollar signs in Perl code gives me a bit of a warm fuzzy feeling, as it reminds me of my childhood programming in 8-bit home computer BASIC, where the dollar sign indicated a string variable.)
  

I still fully intend to learn Python. Even though he was contrasting immature versions of both languages, open-source guru Eric Scott Raymond’s classic essay on the elegance and ease of Python compared to Perl still makes as big an impression on me now as it did when I read it ten years ago. Perl’s become more elegant since then, and its suitability for large software projects has been greatly enhanced by the Moose project, which transforms Perl’s OOP features to such a degree that it almost creates a separate dialect of the language. On the other hand, Python has loud cheerleaders in the scientific community: my impression is that the NumPy libraries have in just a few years supplanted Fortran as the development environment of choice for numerical analysis of large data sets, something that is a critical programming task for me. And somewhere near the top of my to-do list is to learn one of the new platform-independent Web 2.0 programming frameworks; Ruby on Rails seems to be the most popular, although I privately suspect that’s partly because of its oddly catchy name—but Python-based systems are close behind.

"Shadow IT" is a symptom, not a disease

...specifically, it’s a symptom of an inability of an organization’s IT department to meet business needs. And not all shadow IT is created equal: the term lumps together (for instance) staff circumventing security controls on their workstations to install unauthorized software, with people creating complex Excel spreadsheets to answer their business-intelligence questions (rather than asking IT to develop or purchase a new software package). So a little bit of the handwringing by old IT salts that you’ll see if you Google the term is justified, but for the wrong reasons, as I’ll explain.

If people need to install software and have shown that they can do so without bricking their machines, then IT should relax security controls, encourage them to keep informal records of changes they’ve made to their configuration, but most of all stay out of their way and allow them to be more productive. Similarly, don’t keep people from solving their problems with Excel; if they’re solving their problems (period), ipso facto that’s a good thing and IT staff (not users) are the ones who need to adjust their attitude to fit reality.

Let’s focus on the latter example for a bit. Here’s where IT really has a good chance to support rather than obstruct. As I’ve argued before, Excel has long been less a spreadsheet program than a development environment, with more flexibility and usable power than many dedicated business intelligence tools and ERP systems. Consider Juice Analytics, my favorite gang of Excel wizards: If the sample code they give away for free in their blog is anything to go by, the stuff they charge for is a wonder of transparency, structure, and scalability—just like the best "real" software.

But if it’s really a development environment, however, Excel needs to be treated as such; and in turn its power users need to be treated as developers. What are the three most important support tools for developers in a mature programming shop? Source control, source control, source control. Excel source control tools exist—there’s a short but sweet discussion of them here at StackOverflow—but they certainly aren’t mature. The technical problem here is that Excel stores data and business logic (the latter in the form of formulas and VBA code) as one binary file; same thing for the BI and ERP tools I’ve used, which make you navigate through dialog box after menu to create a query, report, or whatever that’s then stored somewhere mysterious inside the software. If all such software stored everything the user created in a text file in a standardized parsable format (XML being the obvious candidate) those files could be manipulated just like the source code files in a traditional development environment (i.e. not just versioned but interactively or programmatically edited using the user’s favorite tools), and they’d be exposed within the organization to be shared, catalogued, backed up, etc. Again, a win-win for users and IT, provided the latter opens its mind to the possibilities.

The article that sparked this post? Ironically, it's on TechRepublic, the epicenter of an awful lot of that handwringing I mentioned above. "Decade of the Developer," indeed. Open source reshapes the organization itself?

Webscraping awful JavaScript, part II

I edited the previous post to remove the link to the website I'm trying to scrape data from, in order to protect the guilty. What I am dealing with there is about 8000 lines of JavaScript, of which, I think, roughly 6000 lines comprise multiple blocks of code that are identical except for a loop condition, comparison, or other minor change. It's a classic if ugly C idiom, and sometimes unavoidable in a language so primitive. In a higher-level language like Java, C++, or JavaScript, it's the mark of a tyro—especially in JavaScript, which has first-class functions, which could be passed as arguments to a single instantiation of the repeated code.¹

Luckily I can sidestep the mess and just figure out the data-parsing code.

1. Douglas Crockford, in JavaScript: The Good Parts (hilariously, a little less than one-fifth the length of JavaScript: The Definitive Guide), argues that first-class functions are the best and most important thing about it: they make the language essentially "LISP in C's clothing." [back]

I'm overdue to learn Java

Part of my reason for writing this blog is to document my reëducation in programming (and, reciprocally, to encourage myself to take that process seriously). I've just ordered a general Java handbook for experienced programmers and a book specifically on graphics and audio programming. I'd like to port (or write shameless copies of, depending on your point of view) some classic 8-bit games to Java, with the aim of making them sufficiently platform-agnostic that they will run on mobile devices just as well as on desktop OSs. That little trackball on the BlackBerry is just crying out to be used to play Centipede, Missile Command, or maybe something a bit less well known like Crystal Castles:

I think my first significant Java program, however, is going to be a machine that plays Terry Riley's modern-classical composition In C (differently every time, of course). And there'll be some kind of kaleidoscopic graphical accompaniment showing which notes each "musician" is playing when. Yes, I'm aware of how ridiculous this may seem, but what's wrong with it as an exercise in generating graphics and sound in sync?

The last time I was paid to write software, it was a couple of medium-sized programs in C++ that ran in user space on Windows NT but worked hand-in-hand with a custom driver running (of course) in kernel space. One of those programs that I'm particularly proud of was an interpreter for a "little language" that enabled automatic testing of some proprietary vertical-market hardware. After that job, I taught C++ to undergrads for a while, with an emphasis on OOP fundamentals and how to apply them properly, not just on the syntax of the language. So I don't expect any trouble becoming fluent in Java, since the concepts of C++ are still rattling around in my head: multiple inheritance, operator overloading, virtual functions, references, and all that cool stuff. (I'm rusty on C++ syntax, but that will come back quickly with exposure if necessary.) Still, this will be far more of a brain-stretcher than the Perl code in the previous couple of posts—if the two can even be compared: in the Java case we're talking about brushing up on the effective design of a medium-sized program, while the tiny Perl web-scrapers I presented had a trivial structure (and what little structure there was, was procedural, and I've been doing procedural programming for thirty years).

"Now seems like a good time," I said to myself...

..."to get those rusty programming skills going."

I had found myself wanting to do some analysis in Excel of price behavior of a large list of stocks.

I glanced at the first few pages of Perl and LWP, and then at the Regular Expressions Pocket Reference; I opened Firebug on the Yahoo Finance "summary page" for a stock I was interested in, so that I could see the raw HTML I was dealing with; and wrote the following:¹

---

#!/usr/bin/perl
use LWP::Simple;

# Expects a list of security symbols on standard input, one per line.

print("Symbol\tPrevClo\tOpen\tLast\n");

while ($sym = <>)
{
    chop $sym;
    $summary=get("http://finance.yahoo.com/q?s=$sym");
    die "Couldn't get Yahoo Finance Quote Summary page for symbol $sym!"
        unless defined $summary;

    $summary =~ m/>Prev Close:<.*?>(\d+\.\d+)</;
    $prevclose = $1;
    $summary =~ m/>Open:<.*?>(\d+\.\d+)</;
    $open = $1;
    $summary =~ m/>Last Trade:<.*?>(\d+\.\d+)</;
    $last = $1;

    print("$sym\t$prevclose\t$open\t$last\n");
}

---

It worked the first time—not bad for not having done any programming whatsoever for about five years and nothing of significant size for ten. (Yes, I know it's not very idiomatic Perl—combining the match regexps and doing a few other things would probably cut the line count in half.) That code took a few hours to produce, but subsequent similar programs to web-scrape other pages took much less time, now that I was in the groove.

For example, more exciting was the following, which expects the same list of symbols:

---

#!/usr/bin/perl
use LWP::Simple;

print "<html>";

while ($sym = <>)
{
    chop $sym;

    print "<font size=5>$sym</font><br>";

    foreach $period ("1d","1w","1m")
    {
        getstore("http://ichart.finance.yahoo.com/z?" .
            "s=$sym&t=$period&q=l&l=on&z=m&p=e5,e20&" .
            "a=p12&lang=en-US&region=US",
            "$sym$period.png");
        print "<img src=\"$sym$period.png\"/>";
    }

    print "<br><br>\n";
}

print "</html>";

---

What I'm doing here, if it isn't clear, is scraping a number of security price charts from Yahoo Finance, saving the image files locally, and building a crude but effective web page to make them viewable in one place. Beats looking at each stock by hand for price trends, let me tell you!

Now, all of this may seem like "Hello World" stuff to anyone reading this who's had any programming experience beyond Computer Science 101. But what I think shouldn't be taken for granted here is the amazing ability to (in the first script, as the simpler example) in just a few lines of code, suck an entire web page into a string variable, search that string in a complex way, and output the result in a universally readable (i.e. by humans or other programs) format. We used to want applications to have built-in programming languages—now (and here's the takeaway!) we have programming languages with built-in applications: very-high-level functionality to do things that only applications used to be able to do. And we can do them in a scriptable, redirectable, programmatic way. Admittedly much of this is due to the straightforward API of the LWP module (Perlspeak for "library"); but I'd argue that that accessibility is a function of the design of Perl; there's obviously stuff going on there behind the scenes that would be much harder to write in a language without such integral support for string manipulations (C, say).

I was weaned as a programmer on 1980s consumer 8-bit machines, the multimedia powerhouses of their day, on which even in a high-level language (built-in BASIC), to do anything interesting you had to twiddle bits. And most of the software I've been paid to write has been low-level stuff in C—device drivers and the like. So I'm easily impressed and easily seduced by VHL (very-high-level) languages that let you do so much with so little typing. Of course there's danger inherent in only knowing high-level languages. When you don't understand what's really going on at the machine level, optimization can be much more difficult, for example. Ironically, though, even as undergraduate computer science programs deemphasize C and assembler skills and move their students towards Java, C#, .NET, PHP, and so on—preparing them more effectively for the kind of web-back-end-database-interface work 9 out of 10 of them will face as new programmers—even as this huge and largely unremarked shift in what it means to be a professional computer programmer takes place, hobbyists tinker with microcontrollers, programmed at as low a level as you want, to recapture some of that early-80s frontier-machine-code feeling. Some will call this retrograde or Luddite-ish but the truth is, I think, that controlling hardware directly with one's code fulfills some kind of deep need in the engineering personality to exercise maximum control over one's immediate universe; and there's nothing wrong with the practical experience gained thus: few programmers will ever write an operating system, true, but there will always be lesser software that needs to run "close to the metal." (A $5 pocket calculator will never run a Java interpreter, for instance. I think...)

Getting back to my own programming for my own use and profit, far more complicated and wonderful things will come in time. I'm comfortable using Perl for this kind of stuff, but have never written a program of any serious size in it. What little user-level software development I've done has been fairly strictly object-oriented code in C++. I only know how to use Perl procedurally; understanding the OOP features of the language, which seem to be highly regarded, would be a good thing to have under my belt.

On the other hand, I have had a strong hankering to learn Python, thanks to what seems to me to be a very elegant syntax. And I have the book A Primer on Scientific Programming with Python which—while I'm quite sure that somewhere on CPAN there's a module to support in Perl the same kind of computations I need to do—numerical integration and differentiation, curve fitting, linear regression, etc.—is an excellent tutorial for Python in general besides describing the appropriate libraries in detail. Lastly, the Beautiful Soup library looks like an even cleaner way to do webscraping.

1. How the heck do you format code nicely (i.e. not just in a non-proportional font but also indented correctly, lines that overrun the margin indicated clearly, and with symbols correctly escaped) in idiomatic HTML these days? Yeah, I know there's the <pre> tag, but it doesn't help you with lines that run past the right edge of your text frame (or wherever your body text is going), and you still have to festoon your code with &whatever-entity tags to escape all the non-alphanumeric characters.[back]