Break Out the Case Studies

Yes, vendor-sponsored case studies and whitepapers are notorious for not-too-honest marketing take on projects. However, if you know how to use them to your advantage, they can become one of the best research tools for any interop project.

• By Emmett Dulaney
• 10/19/2006

If you're a carpenter, you know to "measure twice, cut once." If you're in electrical, it's "better safe than sorry." And if you're planning a interop project, I have an adage for you: "Never stop listening to as many sources as you can."

Maybe it's not as pithy as others out there, but it's just as important. Note the use of the word “listening” as opposed to “talking” – it's not meant to make the act passive, but to shift the focus to where it truly belongs. You can listen to what others say, you can listen to what others write and, probably the most important, you can listen to what is implied in what is not said. The more you listen, the more you can learn about the successes and failures of those who have tried similar tasks before you, and the better prepared you are to face those tasks yourself.

One of the first places to turn for information is case studies and whitepapers. They are a great source of preliminary information, but only -- and I stress only -- if you understand what they represent: A marketing tool that's just one step above a press release. This is not meant to diminish their importance, but only to put them in perspective. No vendor would ever publish a case study about what went wrong with its product and why a migration effort failed, as doing so probably wouldn't help sell the product. As long as you understand that, you can start focusing on the value the case studies and white papers hold.

Most vendors publish case studies and related information on their sites. Red Hat, for example, has white papers, success stories, webcasts, and quite a few other useful tools (the white paper on Unix to Linux Migration is a great starting point. Novell has success stories, videos, and so on for all their products available here. The list goes on and on -- rare is the vendor who doesn't offer similar information.

The first thing to consider when reviewing a case study is how similar the companies profiled are to yours. If a vendor goes to the expense and trouble of writing a detailed success story on how they helped a mom-and-pop shop with a point of sale system, then there's a strong possibility that they don't have any larger customers or successes that they should be touting (a Russian proverb reads, “Tell me who's your friend and I'll tell you who you are”). Now, it may not be the case that this is their biggest success – there's always the possibility the vendor wants to focus on a profitable niche market -- but you'll feel a lot more comfortable if you can read about a company similar to (or larger than) yours, and how the vendor worked with that company to overcome problems there.

Look at how closely the company being discussed matches yours in terms of such things as the amount of data and type of hardware but don't stop there. Is it organized in departments similar to your organization's, or are is it completely different? Does it have to adhere to the same rules about data access and retainment (Sarbanes-Oxley, HIPAA, etc.)? Not all of this information will be available in every case study, but the more you can find, the more useful that information will be to you.

The next thing to ponder is how much sugar coating there is? Is there a lot of flowery text about how the customer was amazed, blown away, overjoyed, etc., or is there a frank discussion of some of the issues and obstacles that surfaced?   Always check to see if the document was created internally or by a third party – this is usually a major indication of the level of bias you'll get. This is just a rule of thumb, and not always a surefire guarantee of frankness, but it often holds true. Additionally, it's usually helpful if background information about the industry is included; otherwise, if the company is different from everyone else in the industry, not a lot of the information will be relevant to you.

One of the most important steps in reviewing a case study is to determine what is not included. Does the text jump from installation to “Four weeks later…”? Or does it walk through “After changing the configuration settings and rebooting three times…”? Obviously, the more detailed the information included, the more useful the paper will be. There seems to be a trend toward only posting small case studies in text form and putting longer ones in video. While the video is helpful (you have to love how everyone thinks choppy interview shots are unique and super cool), there is something to be said for written words that you can refer back to time and time again.

Next on the list? Dates. How recent are the events being discussed? Did the actions take place within the last year -- or the last decade? Sometimes a vendor will want to make sure that the company is truly satisfied before writing the case study and that can unduly delay the release of the information. Even so, you are reading about a successful migration to Linux 2.2, or even 2.4, you might as well spend your time reading a history book. Try to focus on whitepapers profiling integration projects that took place within the past year or so.

Look for quotes. While quotes are meant to show that someone other than the vendor thinks the products are great, they serve another purpose: They provide you with a name of someone at the profiled company who went through a process similar to the one you are contemplating. Take that information and use it to contact the person directly (e-mail, phone, whatever works). Tell them that you read the case study and you'd like to ask them a few additional questions. Hopefully, the person you are talking with will offer information outside of what the text or video relays. You might find out that things went better than expected, and that they are a true evangelists for the vendor, or you might find out that there are some other issues they encountered that weren't what the vendor wanted to talk about, so they were left out. You might also find out that, months after the case, they reverted back to what they were using before because stumbling blocks emerged that made the whole idea a terrible undertaking.For example, I once called someone featured in a video and learned that their testimonial had acutally been taken out of context. The conversation went on for almost 30 minutes, and I learned more from it than I ever would have from any other source. No matter what, you will have more information than when you started with and will be better equipped to assess your own situation.

Case studies and whitepapers are not the only places to turn for information, but they provide an excellent source for preliminary research. In coming months, we will look at some other sources as well, but next month we will delve into a single case study and walk through what the white papers and vendors don't tell you.