HeathCare.gov Debacle: Biggest IT Failure Ever?

The botched rollout of the Web site built to let customers enroll in an insurance plan under the controversial Affordability Care Act -- aka Obamacare -- will go down as one of the most high profile IT disasters to date.

That's a high bar if you consider all of the major debacles over the years (think of widely publicized E-Trade, Schwab, eBay and Victoria Secret site meltdowns over a decade ago) to more recent Black Friday retail site failures, and over the past year outages that have knocked off Facebook, Twitter and major advertisers of the Super Bowl. Still fresh on many minds was last year's major meltdowns that shut down major portions Windows Azure, Office 365 and Amazon Web Services, which brought down many key sites with it including Netflix last Christmas Eve.

Yet those seem to pale in comparison to the failed launch of HealthCare.gov, the signature effort of the Obama presidency that has become a lightning rod for opponents that played a key role in this month's shutdown of the federal government. Regardless where you stand on Obamacare, you never want the president telling the world how mad he is about how poorly your IT project was planned and implemented, which is what happened yesterday in the White House Rose Garden.

"Nobody's madder than me about the fact that the website isn't working as well as it should, which means it's going to get fixed," the president said. But that may not be easy. According to a report in The New York Times yesterday, Healhcare.gov is a text-book-case study in how not to manage a critical IT project of this magnitude and importance. The site may need 5 million lines of new code, according to yesterday's report. Overall, it's built with 500 million lines of code --five times the amount needed to run a large bank's computer systems, according to the report.

One of the key factors that may have led to this failure was the Centers for Medicare and Medicaid Services, the government agency overseeing the exchange, taking the unusual step (for a federal agency) of managing the 55 contractors and overseeing the effort to ensure they can properly integrate the apps and ensure the databases work together.

While the site has seen incremental improvements, the extensive code rewrite could take several weeks, at the very least. According to The Wall Street Journal, HealthCare.gov's registration application developed by lead contractor CGI, transfers data gathered from registrants creating accounts and transfers that data to an the Medicare agency's Enterprise Identification Management app developed by Quality Software Services, a subsidiary of United Health Care, which submits data to the credit reporting service Experian to confirm user identities.

Apparently the data hasn't interfaced well with Oracle Identity Manager. Oracle reportedly has sent engineers to help remediate the problem and add capacity but a company spokeswoman told the Journal that OIM is not the root of the problem. "Our software is the identical product deployed in most of the world's most complex systems" according to the spokeswoman's statement.

Better preparation in the form of load testing to simulate the anticipated traffic may have helped avoid much of the problems that surfaced at launch as well, observers note. Some critics are calling for the head of Health and Human Secretary Kathleen Sebelius, who announced today that Jeff Zients, a recent acting director of the Office of Management and Budget to advise on the project, will be adding a "surge" of tech support to remediate the problem.

"These reinforcements include a handful of Presidential Innovation Fellows," Sebelius said in a blog post. "This new infusion of talent will bring a powerful array of subject matter expertise and skills, including extensive experience scaling major IT systems.  This effort is being marshaled as part of a cross-functional team that is working aggressively to diagnose parts of HealthCare.gov that are experiencing problems, learn from successful states, prioritize issues, and fix them."

As the facts of this story unfold, the failed launch of this site that millions awaited for years is a painful reminder of how a poorly planned development and IT deployment effort can doom a key strategic initiative. While such IT failures happen all too frequently, this one could go in the annals of all time flops.

Posted by Jeffrey Schwartz on 10/22/2013 at 2:54 PM