Are You Running Your IT Shop Like a Caveman?

I'm finally back from TechEd North America 2011, following a brief stop in Denver and Seattle to promote my new book. My final session at TechEd was a Birds of a Feather discussion on Active Directory change auditing. There were around 50 IT pros and managers in the room, and there were some revelations that, to me, were truly astounding.

One gent said his company pretty much had auditing figured out. They consolidated their event logs into a single database, knew how to report from that database, generated near-real-time alerts from it, and so forth. This was all done using a home-grown solution, too – zero cost! Well, not zero. That solution has been under development and maintenance for 10 years. A decade. In terms of manpower, that has to have cost that company something like a million dollars (literally) in total.

Other folks aren't so fortunate: They don't have the resources for that kind of home-grown solution, so they're cobbling something together themselves.We talked about using Microsoft Audit Collection Service (ACS; hardly free since it requires you to buy System Center Operations Manager, but if you already have SCOM then ACS is at least bundled). We talked about Windows Server 2008 R2's event log forwarding capability (which nobody was using in production). We talked about third-party solutions, too, and the one common thread is that almost nobody in the room could buy a third-party solution. Images ran through my head of IT pros bounding away at stone tablets using stone hammers, huddled around a campfire in front of their cave. I mean, the sheer primitiveness of what these folks were being asked to do – all so the company could save a few bucks.

The highlight of the hour was when one fellow mentioned that his company wanted him and his team to provide auditing details about some specific event. "We couldn't do it," he said, "because we hadn't been capturing that information." I asked if they subsequently started capturing that information. "No," he told me, "we didn't. Cranking up that level of auditing on our domain controllers was a performance nightmare. We would have needed more DCs to spread the load, and nobody wanted to pay for them. So they just can't have what they want."

Finally, some reality: Everything in IT costs something. It either costs time, or it costs software, or it costs hardware. Sometimes, you can only purchase something in hardware or software – simply throwing time at the problem won't help. The fellow's situation was a perfect example: They knew how to capture what the company wanted, but the cost would have been more domain controllers. Weirdly, companies are often hesitant to buy hardware or software, but they're willing to spend time as if it springs from a never-ending supply.

Here's a little IT truth for you: Time, hardware, and software all cost about the same thing. That is, having your own on-staff developer produce a solution will cost about the same, in the long run, as buying something ready-made (provided what you bought will fill your need in the same way a custom solution would). If your developer has nothing better to be doing, then you spend time and have the developer write the solution. If your developer could be working on something that isn't available prepackaged, then that's a better use of that time – since buying software isn't an option in that case.

Here's another little IT truth: Admins aren't developers. You cannot have an IT pro produce something that would otherwise be available as third-party software without spending a lot more in the long run. You'll spend it in time, but you'll spend more.

I don't know of a single major company that would rather than their administrators custom-build servers using white-box parts from NewEgg or TigerDirect. Servers come from HP, or Dell, or IBM, or someone like that – even though that hardware costs more than the home-built version would, and even though that high-end hardware might have the same specs on paper as the DIY version. Why is this? Because the pro-made hardware is usually a better value in the long run. It's better-made, better-configured, and better-supported. So why do those same companies ask their IT Pros to build hacked-together, DIY, scripted "solutions" to things like change auditing, rather than buying pro-made software that's well-made, supported, and so forth? It boggles my mind.

Posted by Don Jones on 05/31/2011 at 1:14 PM