Software engineering methodology: my, how you’ve grown!

“Hey, is anyone working on foo.c in the Whatsit build? I need to make some changes in it.”

That’s what source control consisted of at my first job as a software engineer, fifteen years ago. Bug tracking took place somewhere between the brain of the lead developer for each project and his email inbox. Code reviews consisted of the smirking face of another programmer appearing over the top of my cube, saying, “I found a place where you dereferenced a null pointer. Want me to show you?”

I learned about the classic academic work in software engineering in school as part of my computer science degree: Fred Brooks, the waterfall model, unit testing, and so on. But as far as I knew in 1996, outside of large companies with an established research culture (Intel, IBM, Xerox) none of those methods were being applied. Neither was the general idea behind them, that software development needed planning, structure, and organization, things that could not to be trusted to individual programmers working in isolation.

It’s amazing how much things have changed in the past fifteen years. The classic book The Psychology of Computer Programming was published in the '70s but its effect was purely academic. It took years for the lessons described there to filter down to the level of actual practice, and it wouldn’t have happened without Steve McConnell’s Code Complete. From the latter book grew the agile development movement, a reaction both against the waterfall model (which makes it so easy to postpone testing until it gets pushed off the bottom of the release schedule) and against cowboy development culture (which makes for a host of problems—poor bug tracking, poor source control, reinvention of the wheel as every programmer builds his own libraries, and on and on.).

And I never would have expected the massive growth that has taken place in what I’ll call the collaborative programmer culture. Just as the free exchange of information in general has exploded due to the Internet, so has in particular the exchange of information on how to program better. There’s something going on that’s greater than just more message boards and more acceptance of the open-source “more eyes mean fewer bugs” idea. I remember when the entire O’Reilly library fit on just one of those old-fashioned revolving wire book stands at the best technical bookstore in the world; now it takes four or five stands, not just because there are more programming languages and software technologies to cover but also because the coverage of each is deeper (e.g. compare the matched set of Llama and Camel Books that used to make up the entire printed documentation of Perl, to the dozens of books now published on the language). Code Complete spawned an entire genre of “software as a craft” books like the Pragmatic Programmer series.

All these factors mean, I hope, that a new generation of software developers is growing up with the idea that agile development methodology is not just the correct way to write software but the natural, comfortable one. This is probably the biggest change ever, with the greatest lasting effect, in the way software is created—bigger than the widespread adoption of OOP over straight procedural programming, bigger than the shift to dynamic languages and automatic garbage collection, bigger than Linux making a professional development environment available to students. It’ll be interesting to see its long-term effect... has the looming dragon of the Software Crisis finally been slain?

Java and the growth of the software business: The more things change...

I’m at a position career-wise where it’s to my advantage to broaden my technical skills before I deepen them any further, so Java is in the to-be-learned queue for real this time (as opposed to in previous posts in which I wrote “I really ought to learn Java” and then went back to banging out backwards ifs in Perl). I’ve been balking at putting Java on the front burner for a while, and for a while I didn’t know why.

Now the reason has finally bubbled to the surface: it’s that Java seemed like a toy back in 1998 or so when I was first programming professionally. Its speed was an issue; the programmers I hung around with were hardcore old-school bare-metal types, and our attitude was that you wrote systems code in C or assembler, applications code in C or C++ (using the Win32 or Microsoft Foundation Classes libraries), and anything else was too slow for serious work. Java’s promise of platform agnosticism seemed like marketing smoke and mirrors, and we couldn’t get past the fact that it was interpreted. (Interpreted? Interpreters were for scripting languages, and you couldn’t write a real application in those, either—more words to eat as Perl and Python matured!)

I’ll get back to the speed issue in a bit. What’s really changed my mind the most about Java is the sheer amount of support for the language in the form of third-party libraries, on top of the staggeringly large, well-thought-out platform that Sun provided from the get-go and Oracle has continued to enlarge and refine. I say “well-thought-out” deliberately; the Microsoft Foundation Classes sounded like a good idea in 1999, but in practice their functionality overlapped with the C++ standard library and the older APIs that Windows programmers were already familiar with, and the typical MFC program ended up being a mashup of all three. (Which is why MS did a good and brave thing when they scrapped all the old cruft of MFC and Win32 and built .NET from scratch.)

The particular example that tipped the balance isn’t so important as such, except that it illustrates the fine-grained libraries that are out there: it’s the jzy3d package, which provides the kind of three-dimensional data visualization that a year ago I was trying to build by hand. And this brings us back to the question of speed: jzy3d is built on top of JOGL, a Java API for OpenGL; and one of the example programs linked from the JOGL site is a port of the game Quake2 that runs in the browser. I was flabbergasted to see that. Quake, for you young-uns, was a ground-breaker in immersive 3-D graphics on its release in 1994 or so; to run it smoothly in high resolution took a high-end CPU (I had a 100 MHz Pentium, pretty good for the time) and a $300 graphics card; it ran best in DOS, needing to hammer on the metal directly for maximum speed; and it’s still a benchmark for low-powered systems (is there a port to the iPhone? I don’t know, but I’d be impressed if there were). As far as I’m concerned, if Java is fast enough to run Quake, then it’s fast enough to run any typical desktop application.

And I think that’s what the majority of the software business thinks, too. Department of Labor stats and the Wikipedia page on software engineering show a doubling in SE jobs since 2004, and that’s despite outsourcing and the 2008 economic crash; what’s more, that statistic doesn’t account for the number of programmers who don’t have the title of software engineer. I’d suspect the true level of growth to be quite a bit greater.

And though I can’t find any information as to what language the majority of these people are coding in most of the time (and if I could I’d suspect it of having been rigged by someone with a vested interest, like Oracle or MS... though here is an attempt from 2006—look at the numbers at the end of each section), I bet Java accounts for much of that growth. The language is just that good, and that effective, and that well-supported. It's enabled many more programming jobs to be created, because it's lowered the barrier to entry. Let me stress that I don’t mean that as a euphemism for “Now you don't have to be as smart to be a programmer, so a lot of stupid people are getting good white-collar jobs that they wouldn't have otherwise.” Rather, a flatter learning curve for mainstream software development is a universal win: it means more jobs, better-working software, and a more robust world economy, thanks to, for one, the continuing adoption of general-purpose hardware running custom software in industries that previously relied on custom and/or archaic hardware for process control. The more things change, the more they change!

Ten years ago people were calling Java “C++ for dummies.” Well, guess what? That isn’t an insult any more. People are coming to realize that C++ can be a real pain in the butt. It’s hard to write maintainable, reliable code in C++. You can do it; I can do it, and I have done so in the past. And I want to make clear that where speed is of the absolute essence, C++ is obviously a better choice (although I think the complexity-vs.-features tradeoff of using C++ vs. good ol’ C ought to be be carefully considered in those cases). There's at least one excellent API for user-level software in general use that works best with C++, Qt, and if there's one thing you can say about Qt programs it's that they are snappy! And, undoubtedly no coincidence, the API is a delight, very clean and lightweight—it reminds me of programming BeOS. (With the speed and memory of the average consumer-level computer being what they are these days, it’s ridiculous that we still have to put up with sluggish “click-and-wait” UIs... but that's a topic for another post.)

In practice, most C++ code, I would bet, is written using a subset of the language with which the programmers on any given project feel more comfortable, ignoring the really tricky features (no multiple inheritance, for example; maybe no operator overloading) and using some sort of integrated garbage collection scheme (overloaded new, for example) rather than relying on the programmer to manage dynamically allocated memory. I didn’t realize until I took a good look at Java (if you must know, I’ve been reading the first 200 pages or so of Java in a Nutshell a bit at a time before bed for a while now) that that comes very close to describing what Java syntax is: a slightly more verbose, less complicated version of C++. (And of course it has garbage collection, although not tunable for the needs of real-time programming, as far as I know.) QED: There’s nothing wrong with that. (Really, who could possibly take offense at replacing =0 with virtual, or : with extends?) Unless you need that last bit of speed that only a true compiled-to-assembler language can give, I can’t see anything wrong with Java as a baseline for applications programming.

Is SQL obsolete?

Lately I've been studying databases and SQL. Not long ago I purchased a large corpus of financial data (historical stock market indices), which came with a front-end application to output the numbers in various ways for easy importation into other software. What’s going on behind the scenes, most likely, is that a relational database is being queried and various views are being produced as flat tables—there's nothing surprising there. But using this software has made me want to know more about database theory, and to understand SQL better, since the latter is basically the lingua franca of databases in the corporate world and everywhere else.

Hence the question in the title of this blog entry. RDMSs are a fact of life—almost universally agreed upon as the best way to store large data collections—but the way we use them has changed: with increased computer power we are now able to query a database in far more complex ways, and far more frequently; this is what makes data mining possible. And general-purpose programming languages have changed equally radically since SQL was developed. In short: if SQL was developed to support a certain kind of database access, a kind that was not easy to do with the programming languages of the time, and a kind that is not much like the kind of access we need now, has its time passed? And if the answer is “yes,” what comes next?

One fact that supports a “yes” vote: First, not only have programming languages changed, but the environments in which we use them have, as well. The designers of SQL didn’t anticipate point-and-click interfaces, let alone the WWW; they probably expected front-end interfaces of some kind, but were thinking only of the green-screen mainframe fill-and-submit forms of the 1970s. The typical web application is a mess. It’s a Frankenstein’s monster of barely adequate tools assembled into an effective but painfully ugly whole: peek behind the curtain of the Great and Powerful Oz of your typical e-commerce site or a Web-based ERP system sitting on a corporate intranet, and you’ll see dynamic HTML full of JavaScript, presenting to the user some buttons that execute PHP code, which queries an RDMS using SQL code, which was written inline in the PHP and is then handed off to the server as a string variable. It works, but, again, it’s a crude, ugly mess. This is the reason behind the push towards more integrated development approaches for Web 2.0 applications—I’m thinking of Ruby on Rails as a particularly fine example, though as I mentioned in the previous post here, I’m leaning towards Catalyst or perhaps a Python-based solution for my own work.

Another point: the languages in which better database front-ends can and should be written lend themselves naturally to direct access to the database. I’ll anticipate objections to the point in the last paragraph by saying that SQL has changed over the past 40 years. Stored procedures are a major difference, and one that can, in part, avoid the clunkiness of having to pass every query to the database as a long sequence of string literals. Extensions like Transact-SQL make stored procedures able to process data in a much smarter way, reducing the number of individual queries the front end has to do. One result is a greatly reduced load on the network between the server and the client, which I'm not saying isn't important! But, in terms of abstraction, stored procedures, I argue, are the wrong direction. (T-SQL in particular falls far short of being a modern programming language: it looks like nothing so much as Fortran77. Wikipedia points out—damning with faint praise!—that it’s Turing-complete. A ball of yarn and a Sharpie are also Turing-complete, but that doesn't mean I'm going to write code to access a database with them.) A far better design is to build the functionality of SQL directly into the front-end language. This is a simple but incredibly powerful idea. Modern programming languages have the capability (i.e., which older languages did not) to effortlessly create and manipulate data structures of essentially unlimited complexity. There’s no reason why the programmer couldn’t specify queries and retrieve data via a library that translates native-language idioms into SQL transparently; going a step further, I don’t see why a language library shouldn’t allow access to the RDMS server directly, exposing the results of a query as a data structure native to the language, without SQL in the middle at all. Neither of these is pie-in-the-sky stuff: for implementations of the former idea, see Rails’ Active Record Query interface, Python’s SQLAlchemy, or Django. Nobody seems to have actually implemented the complete elimination of SQL yet, but there’s one brief but thought-provoking discussion here. And Microsoft's LINQ goes a long way towards breaking down the distinction between different kinds of data storage, providing a unified interface; as do the various wizards in Visual Studio 2010 which allow you to just drag and drop tables from Access into your project.

Part of the reason for the continued use of explicit SQL code constructed by a front-end app and submitted as strings, I suspect (again, sidestepping the speed issue) has to do with security. Stored procedures provide security via encapsulation; explicit code can be more easily checked for attempted exploits such as SQL injection. But the latter is an issue precisely because of the exposure of the database structure at an unnecessarily high level; and the former is what object-oriented languages are all about. (Again, a design feature that SQL needed to provide at the time of its conception but which has now been implemented with far more versatility and elegance in programming languages: OOP and other techniques preach the gospel of “maximize cohesion, minimize coupling” to improve not just design and maintainability of software but also data security.)

I want to make clear that I’m not advocating the elimination of relational databases as such. (Even though arguably most current implementations don’t really follow relational database theory in practice.) What I don’t like is the crude way that we interact with them. I want to see programmer-space access that goes beyond building SQL code with string manipulation operators, then handing it off to the database; I want to see end-user-space access that goes beyond clunky wizards that build a limited subset of SQL queries (always almost, but not quite, the one you really need). The relational model has been adopted almost universally as the standard way to store data sets of arbitrary size, for very good reasons that held true thirty years ago and still hold true today. But let’s make sure our reasons for the way we interact with the relational model still hold true as well.

Perl Ate My Brain!

Looking back at my earlier posts here I see a lot of optimism about rebooting (an irresistible pun) my ability to write software; a year or so ago, I had a plan to learn some new languages, notably Java, and refresh my knowledge of existing ones, notably C++, by embarking on a few different programming projects. It seems I’ve taken a slightly different route. I’ve done a lot of programming over the past year, but it’s almost all been in Perl. Hardcore Perl fans would say this was inevitable—that the language is so powerful and flexible that, once you start using it, there’s no need to use any other. People with a lower opinion of Perl would be more likely to call it a sickness.

What’s so seductive about Perl? A few things, to start, off the top of my head:

• The astonishing variety of freely available modules (Perl-speak for libraries) for highly specialized tasks: more than a few times I’ve girt my loins for a difficult bit of coding, only to check CPAN and find that someone else has already done it for me.


• Flexible built-in data structures and the ability to easily create more elaborate ones when the built-ins won’t do. Perl’s built-in lists, hashes, and references seem to capture the best features of the C++ Standard Template Library, Lisp lists, and hand-built C data structures, with performance (thanks to a highly evolved and optimized interpreter) fast enough for most purposes.


• Syntax that, once a slight learning curve is overcome, feels natural and is (surprise!) easy to read. Perl’s reputation for looking like line noise comes from two things: the idiomatic use of regular expressions as the most effective way to parse string (and often numeric as well) data, and the way that non-alphanumeric characters are used to mark variable types. The former did encourage “write-only” code in Perl 4, but in the current implementation of the language, you can interpolate comments in regexps to make them as readable as you want. The latter becomes more rational once you realize that the dollar, percent, and at symbols are used to identify a variable’s type not just at declaration, but at use, in context; they aren’t just an uglier cousin of Hungarian notation but a concise expression of an extremely elegant kind of dynamic casting. There are a few rough edges—for example, the backslash and curly braces are overloaded for references (sort of like C pointers) in a way that makes me uncomfortable—but on the other hand, there are only so many non-alphanumeric characters in ASCII. Perhaps Perl 6 should move on to use some interesting Unicode symbols, or (I’m only mostly joking here) something like APL… (In the meantime, the plethora of dollar signs in Perl code gives me a bit of a warm fuzzy feeling, as it reminds me of my childhood programming in 8-bit home computer BASIC, where the dollar sign indicated a string variable.)
  

I still fully intend to learn Python. Even though he was contrasting immature versions of both languages, open-source guru Eric Scott Raymond’s classic essay on the elegance and ease of Python compared to Perl still makes as big an impression on me now as it did when I read it ten years ago. Perl’s become more elegant since then, and its suitability for large software projects has been greatly enhanced by the Moose project, which transforms Perl’s OOP features to such a degree that it almost creates a separate dialect of the language. On the other hand, Python has loud cheerleaders in the scientific community: my impression is that the NumPy libraries have in just a few years supplanted Fortran as the development environment of choice for numerical analysis of large data sets, something that is a critical programming task for me. And somewhere near the top of my to-do list is to learn one of the new platform-independent Web 2.0 programming frameworks; Ruby on Rails seems to be the most popular, although I privately suspect that’s partly because of its oddly catchy name—but Python-based systems are close behind.

Not reinventing the wheel (again)

As I mentioned in two previous posts, I’ve been doing some 3-D graphics programming using the Perl interface to SDL. SDL is quite fast for 2-D graphics, but it doesn’t do 3-D on its own; so there’s a stack on my desk over a foot high of used textbooks with titles like Introduction to Computer Graphics Algorithms from which I’ve extracted the matrix transforms for various kinds of projections of a given pixel from threespace to twospace (orthographically onto a planar "window" given by a certain equation, seen in perspective by an eye located at given points, and so on). It’s been educational, but the results I’m getting are unusably slow, and I think they would still be too slow even if I were using a faster language. So, it’s time to start using a graphics library that supports 3-D—that is, one where someone else has already done the hard part of writing optimized code to do those transforms. It’s one of the hardest lessons to learn as a new software engineer transitioning into the workforce from school: it’s OK to use other people’s code; it’s better, in fact, to use other people’s code in the form of using existing libraries, rather than reinventing the wheel. (Jeff Atwood wrote an interesting contrarian blog post on why reinventing the wheel is sometimes, in special cases, the right thing to do: in general, the preference for reinvention is proportional to the need for performance; but note that there’s no consideration of cost there. Ironically, reinventing the wheel in the software sense is more likely to be favored by the ill-informed or non-technical manager, who doesn’t understand [the accepted wisdom] that it’s "always" better to buy than to build).

The obvious choice, one that’s been around for years with lots of support in the form of books, tutorials, and sample code online, is OpenGL with the GLUT libraries. The GLUT sample code I’ve skimmed seems to have (at a very high level) a two-part structure—you first set up the parameters for a 3-D rendering (the minimal case being an object and a view, with no light sources, texturemapping, or other complicated/cool stuff), and then you spawn a thread or another process to do the actual rendering on the fly until told to terminate. I’m not sure yet how this will square with the fact that I want the rendering to refresh in real time per the new data I’m collecting from the server. Let’s pull some numbers out of a hat: say I have a Perl program that runs all day long, sleeping most of the time, but waking up every five minutes to web-scrape and munge some derivatives-pricing data; say the latter takes about a minute, allowing for the bottleneck of my mediocre broadband connection. It sends that data on to the graphics code, which creates an interesting 3-D graph or visualization¹ in n minutes. Then that 3-D image pops up in a window, rotating slowly so that you can see it well enough to interpret it. If n<4, then the whole thing can repeat ad infinitum. If n>4, then the rendering is going to lag behind (and more behind, and more behind...) the data collection, and become useless. Depending on how GLUT works, I might be able to kill two birds with one stone—minimize n, and minimize CPU load—by avoiding the real-time render entirely: I’m hoping that I can actually have OpenGL render the frames of a smooth 360-degree rotation of the graph from start to finish as image files. Then these could be converted to an animated GIF (if they aren’t already) and shown on a web page containing other useful data (which itself could get updated every five minutes, continuing the example, or more often). Meanwhile, the Perl code sleeps again, and then wakes up and continues the cycle...

I want to have the possibility of running this code on OS X, where I spend most of my time, or on Linux, or on Windows 2000 or Windows XP (which I run in virtual machines here and there for various reasons). Toward that end, in turn, I’m requiring myself to write the most portable code possible: since OpenGL and GLUT are (intended to be) entirely platform-independent within the limits of the hardware they’re run on, there’s no sense in doing anything else. To refamiliarize myself with the Microsoft Visual Studio IDE² I’ll be coding my GLUT app as a Win32 application in C++, if necessary using some simple but not too kludgy wrapper classes around C code full of function pointer callbacks that’s sadly inherent to the Win32 libraries (well, I guess I can give them a break... it was the ‘90s.). However, if the instructions I’ve found for writing cross-platform-safe GLUT code for Linux and Windows using VS 2005 are accurate, that shouldn’t be a problem.

But I’m also going to try writing the whole thing in POGL, the Perl OpenGL/GLUT bindings, which are claimed to be just as fast as using OpenGL via C or C++ or <insert compiled language here> That makes the parsing that I referred to here easier since I won’t have to call Perl from a compiled C++ program (obviously a performance hit to be avoided) and I won’t have to find a regular expression library for C++ (of which I'm sure there are plenty, but the differences in regexp syntax between them and Perl would probably be unpleasant).

1. The form of which is a secret—I can’t give away all my tricks, can I? However, one of the books in that big pile is a collection of academic papers called something like Topics in Scientific Data Visualization, so you can guess I’m headed into some pretty wild territory. (Here be dragons!)[back]


2. One of the few truly great pieces of programming to come out of Redmond, it pains me somewhat to admit, along with Excel 2003 and the old QuickBASIC compiler (the distant MS-DOS ancestor of Visual BASIC, for you young whippersnappers). [back]