Never Again

Tale of the Vanishing E-Mails

IT folks are left scratching their heads as messages disappear into e-mail limbo.

It began as a normal day for our small (we have three staff members) IT shop, with the usual helpdesk calls from various users. One such call was from a user who was not receiving e-mails from a certain AOL sender. What puzzled this user was that she would get some e-mails from this sender as long as they were not sent as part of a group mailing or did not have any attachment in them.

Our helpdesk went through the cursory troubleshooting techniques of checking for a firewall block and looking in the quarantine and junk e-mail folders, but nothing was found. We told her we would keep the troubleshooting ticket open.

A Growing Problem
The following week the CEO of our company complained that he was not receiving mail from a certain sender when that sender e-mailed him something containing HTML contents in the body of the e-mail. This was followed by a complaint from another user group not being able to receive e-mails containing attachments. Again, we looked at the usual culprits that would block the e-mail, but came up empty-handed. It was beginning to get rather annoying.

We called the senders' IT department to seek its help in determining this mystery. They checked their logs to make sure the e-mails in question had indeed been sent. They also confirmed that they had not received any non-delivery report (NDR) in their servers.

We were using Postini as our filtering service, so we called them to see if their server was blocking such mails. We sent them the Internet headers of other e-mails received from the same sources so they could diagnose them. They sent us details of logs of the "missing" e-mails that showed that Postini's server had forwarded the e-mails to our Exchange server.

Vanishing Act
So now the question was: "Where did all these e-mails go?" We could trace the path of these e-mails all the way up to our server and yet they weren't getting distributed to the appropriate recipients. We even opened port 25 on our PIX firewall that was initially set to allow mail on our Exchange Server from Postini only.

This action produced a disastrous result. Now our server was open to the whole wide world and we were deluged with spam.

Finally, we looked at the Exchange System Manager on the Exchange Server and activated the Message Tracking Center. It showed the message from AOL being received, so we went into the Message History to look for details. It showed the message being submitted to Categorizer but there was no indication as to what happened to the message after that.

What's Your Worst IT Nightmare?
Write up your story in 300-600 words and e-mail it to Editor Ed Scannell at escannell@redmondmag.com. Use "Never Again" as the subject line and be sure to include your contact information for story verification.

Seeing Double
Normally this would have shown the message to be queued for local delivery, but that wasn't the case. It was then that we came upon a stunning realization: Our mail was not only being checked by Postini, but we were also running Trend Micro's ScanMail for Microsoft Exchange where, by default, attachment blocking under the virus-scan function had been enabled.

So while Postini was allowing the mail with attachments to go through after filtering it, Trend Micro was stopping it from going any further. This is why there was no trace of those e-mails even when Postini had forwarded them to our Exchange Server. We took the check mark off the box that enabled attachment blocking, put the restriction on port 25 back on the firewall and, Presto! Everything was back to normal.

About the Author

Syed Asif is the IS director for Queens Centers for Progress in Jamaica, N.Y.

comments powered by Disqus

Reader Comments:

Sat, Sep 22, 2007 Anonymous Anonymous

I enjoyed this story and I too work in a smaller shop. Those of us in smaller companies don't have the luxary of having exchange experts that do nothing but monitor a handful of servers all day. We wear many hats and work on many different unrelated things throughout the day. It's easier than you think to not know about or even forget how something like an Exchange server was originally setup a year or two ago or perhaps someone else set it up entirely.

Fri, Sep 14, 2007 Anonymous Anonymous

Should this have even been published as an article in redmondmag? Idiot doesnt even know what software is running on his exchange server!

Fri, Sep 14, 2007 Anonymous Anonymous

Yes, I do agree. Even Trend Micro's ScanMail has logs for all emails blocked, can even send email alerts to the Admins. One look to the logs will show where the problems where.

Thu, Sep 13, 2007 Anonymous Anonymous

I find it interesting that the exchange admin didn't know what software defenses were being run on the exchange server nor the settings being used.
Why wasn't this caught before (eg no attachments allowed) and if it changed, why & who changed it or installed the Trend Micro sw? That would be a bigger admin concern to me than that there was a double check going on...

Add Your Comment Now:

Your Name:(optional)
Your Email:(optional)
Your Location:(optional)
Comment:
Please type the letters/numbers you see above

Redmond Tech Watch

Sign up for our newsletter.

I agree to this site's Privacy Policy.