In-Depth

The Project from Hell

Three continents, incompetent management and an impossible deadline is a case study in how to NOT do a major upgrade.

It all started so well; I had no inkling that things would turn sour so quickly. Although I’d repressed the memory of my project from hell, I’ve forced myself to recall all the lurid details so you can learn from my pain. The project was a Lotus Notes/Domino upgrade, but as you’ll soon see, the project issues encountered are universal.

The first I heard of this project was during a conversation with my practice leader in the consulting firm in which I worked. “Greg, we’re working on a cool proposal right now to upgrade to the latest version, and if we get it, we’ll need you to go back to Australia during part of the implementation.”

This all sounded fantastic—I was going to be utilized (the golden rule of consulting!), do my first upgrade to this new version of Domino, and better yet, get a trip back to Australia paid for by someone else.

Lesson 1
Reputable firms (and people) realize the folly of committing to something they know they can’t deliver. It’s much better to admit this and advise what’s possible up front, for the sake of all concerned.

The reality of the project hit us when our assembled staff started work on the client site. The client had business imperatives that forced a delivery date that was straight-out unattainable. More respectable consulting firms hadn’t wanted this work because they knew it would end in tears. It appears that our firm was so desperate for the business, it had agreed to deliver something it knew we couldn’t do. But the expectation was that once we got started, it would be too late to remove us; and besides, eventually things would probably turn out OK…

As we progressed through the project, it became clear that to get anywhere near the desired schedule, the team was going to have to work around the clock. However, the project was being run (“project managed” is too strong a term) by a practice leader—I’ll call him Bob—who made no attempt to engage us to committing to the project schedule. Thus, it became his schedule and not our schedule. This increased our frustration and anger, turning lunchtime breaks into mutiny sessions where we we’d voice our discontent.

With our assembled team of five, we were certainly capable of handling the task at hand, but the unrealistic schedule meant there was little commitment within the team to making the date. The other guys were contemplating resigning there and then (this was a few years ago, when jobs were plentiful), although the prospect of the free trip to Australia trip kept me from similar considerations. Bob clearly was no leader—that would imply that people would willingly follow him—but he was great at selling to clients. There was a real death march feeling within the team that we were headed for doom, and nothing we could do would change that.

Lesson 2
People have a variety of motivators, but expecting a super-human effort from all is asking for trouble. I believe it would have been far better for the practice manager to come straight out and ask for our commitment to the schedule, or perhaps have us develop our own schedule. It would also have helped if he’d been upfront about the challenging timeframes and perhaps promising future rewards for successful project completion. People are usually happy to work hard achieving a goal they believe can be met (see Lesson 1), but they prefer to be asked.

After a time planning the upgrade and collecting information about the client’s environment, our implementation proceeded in earnest. We built the first servers, migrated some of the data and were ready to proceed. Then the fun really started.

The team split up to travel to the main client sites to perform the majority of the upgrade—Phoenix, Arizona; Dublin, Ireland; and Sydney, Australia (where I was). We were migrating 1,000 users to the new environment built on Windows NT 4.0 and Lotus Domino 5.01. At this time, our new manager—I’ll call him Abe—joined the project. In terms of the management hierarchy, he came somewhere between us and Bob, our practice lead. We hadn’t worked alongside him before, but we knew he had an extensive technical consulting background. But unfortunately, we’d soon learn Abe wasn’t much of a people manager, either.

The next sign of trouble was when I called after noticing some configuration changes in the environment with which I was unfamiliar. When I asked Abe about this, he told me he’d made some design changes. When I asked him if he could send me some details about what had changed, he said if there was anything I wanted to know I could call him and ask!

Lesson 3
In a leadership role, you have to have faith in your people—even when you’re sure you know better. In that case, you’d do much better coaching your staff by asking open questions about the situation, rather than straight-out dismissing their technical judgment. Not only do they feel they’re part of the agreed solution, they’ll know better next time.

The 18-hour time difference between Sydney and Phoenix makes that suggestion impractical, and the arrogance of coming in at the last minute and changing everything was astounding. Abe did have a strong technical background, and it’s likely that these were sound ideas; at the same time, this lack of confidence in his own staff wasn’t conducive to an ongoing productive relationship.

As we proceeded further, we were engulfed by the internal politics of the client. The majority of the technical research and development folks were in Dublin, with the marketing staff based in Phoenix. Our project sponsor for the company that hired us was also in Phoenix and had made project decisions without knowing or caring how this would affect users. Instead, he elected to hide behind the hired consultants (in other words, our team) and have us deliver bad news. We weren’t able to act on the complaints from those whose ability to work was affected by the changes; we just had to do the dirty work and listen to the resulting profanity. Any further changes were going to have to be debated internally in the company. But given that these were usually directives from “on high,” nothing was going to be changed without a fight.

For example, the technical staff in Ireland would have much less system access in the new environment. This sounds great from a CIO viewpoint, but some of this access was needed for them to do their jobs. (On the other hand, many of them currently had more access than they really needed, and some of their complaints were more ego- than need-driven.)

Having our project team communicate these decisions onsite to the Dublin staff wasn’t much fun. The familiar refrain of “(Bleeping) Americans” was all-too-frequently uttered by the Irish staffers. They had little respect for decisions made at the Phoenix headquarters, and we were stuck in the middle, attempting to implement the already-agreed-to project.

Lesson 4
For an ambitious project like this to succeed, it’s incumbent on the project sponsor to take a leadership role to explain to the project stakeholders (that is, everyone likely to be affected by the project) what’s happening and why. Being invisible and leaving this to the project team to deliver the news—good and bad—is a recipe for disaster.

As we went through our migration, the time zone issue really started to bite us. The different time zones of the three global locations made it impossible to have a single time during the day for a status review meeting. With team members working long days to perform the migration, having one of us wake at 2 a.m. for a teleconference was never a practical idea. The best we could hope for were updates between two of the locations at the start and end of each day, and hoping the information didn’t get mangled as it was passed around, like a game of “telephone.” There’s nothing worse than trying to be part of a team implementation when you don’t know what everyone else is doing. With no regular updates, we had no idea of what progress had been made in the other locations, what decisions had been made, what problems had been encountered or how they could be fixed. This was particularly important because this was a new version of the product, and for all of us it represented our first production deployment of it.

Ultimately, we kind of met the committed live upgrade date. But it was another two weeks until they had a completely stable environment, and for some staff, this interim period had to be frustrating, as their productivity would have been severely affected. E-mail mostly worked from the beginning; it was the applications, most of which were developed internally, that took the extra time to be completely migrated. Of course, e-mail’s important, but for many of the company’s employees, these other applications were critical to their jobs. From our consulting firm’s point of view, more hours meant more dollars for us, since this was a time-and-materials contract.

Lesson 5
This lack of project planning was unforgivable given the global nature of the project to be delivered. Before we all went off in our separate directions, we should have had a plan for how we all would keep updated on the project status. One answer might have been staggered teleconference status meetings, so that staff at different locations had a chance to participate in every second meeting. In between times, having minutes distributed from these status meetings would have been useful.

Within three months of project completion, most of us had left the company. Not long after that, the consulting firm itself had gone into bankruptcy and folded. But I did get my trip to Australia, I guess. Alas, my poor workmates that were originally excited about their trip to Ireland saw little more than the Dublin airport, their hotel room and the client site.

It was formative experiences like this that convinced me to take a management role myself. I’m not saying that I haven’t made—or won’t make—mistakes, but I’d like to think that these will be nowhere near the magnitude of those I experienced on that job. Seeing things done so poorly was source of motivation for me to demonstrate that perhaps I could do better.

comments powered by Disqus

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.