"Shadow IT" is a symptom, not a disease

...specifically, it’s a symptom of an inability of an organization’s IT department to meet business needs. And not all shadow IT is created equal: the term lumps together (for instance) staff circumventing security controls on their workstations to install unauthorized software, with people creating complex Excel spreadsheets to answer their business-intelligence questions (rather than asking IT to develop or purchase a new software package). So a little bit of the handwringing by old IT salts that you’ll see if you Google the term is justified, but for the wrong reasons, as I’ll explain.

If people need to install software and have shown that they can do so without bricking their machines, then IT should relax security controls, encourage them to keep informal records of changes they’ve made to their configuration, but most of all stay out of their way and allow them to be more productive. Similarly, don’t keep people from solving their problems with Excel; if they’re solving their problems (period), ipso facto that’s a good thing and IT staff (not users) are the ones who need to adjust their attitude to fit reality.

Let’s focus on the latter example for a bit. Here’s where IT really has a good chance to support rather than obstruct. As I’ve argued before, Excel has long been less a spreadsheet program than a development environment, with more flexibility and usable power than many dedicated business intelligence tools and ERP systems. Consider Juice Analytics, my favorite gang of Excel wizards: If the sample code they give away for free in their blog is anything to go by, the stuff they charge for is a wonder of transparency, structure, and scalability—just like the best "real" software.

But if it’s really a development environment, however, Excel needs to be treated as such; and in turn its power users need to be treated as developers. What are the three most important support tools for developers in a mature programming shop? Source control, source control, source control. Excel source control tools exist—there’s a short but sweet discussion of them here at StackOverflow—but they certainly aren’t mature. The technical problem here is that Excel stores data and business logic (the latter in the form of formulas and VBA code) as one binary file; same thing for the BI and ERP tools I’ve used, which make you navigate through dialog box after menu to create a query, report, or whatever that’s then stored somewhere mysterious inside the software. If all such software stored everything the user created in a text file in a standardized parsable format (XML being the obvious candidate) those files could be manipulated just like the source code files in a traditional development environment (i.e. not just versioned but interactively or programmatically edited using the user’s favorite tools), and they’d be exposed within the organization to be shared, catalogued, backed up, etc. Again, a win-win for users and IT, provided the latter opens its mind to the possibilities.

The article that sparked this post? Ironically, it's on TechRepublic, the epicenter of an awful lot of that handwringing I mentioned above. "Decade of the Developer," indeed. Open source reshapes the organization itself?