ISP Nightmare: Shaw Cable is stealing our mail

by admin on October 24, 2008

Okay, in all my fifteen years of IT experience, this has to be the one of the worst screw-ups that I have encountered in dealing with a vendor. And for those of you who know me, you know that is saying something.
Basically, about 5% of our business mail is being swiped by Shaw and sent to mailboxes controlled by them. This has been going on for about two months, Shaw knows about it, and hasn’t fixed the problem. This is the story:
At the beginning of last summer, we started to experience weirdness with our email delivery. Shaw, our ISP provider, was also handling our email which could be accessed through a web browser or download via POP3 to our Outlook client.
After about the third bout of email outage, we made the decision to switch over to google apps for the domain. Over the August long weekend here in Canada, we pushed the MX record for our URL over to the Google mail servers.
Over the following weeks, we started to notice that about four of our mailboxes were literally missing mails, especially mail from some of key clients. The email senders were not getting any bounce backs or “undeliverable mail” receipts. They were sending mail and we were just not receiving it.
As you may know, if you use the google apps for the domain mail services, you don’t get phone support (or any support at all) unless you upgrade to the paid service. So after a couple of weeks, we had a choice to be made: upgrade to google’s paid service or switch providers (again). We decided not to skimp anymore and chose a remote Exchange hosting provider called Radiant solutions.
We migrated to Radiant at the end of September and their customer service was excellent, especially with help migrating the Blackberries of our sales force so that they were synchronized with our email accounts.
But even after the migration, we noticed that some mail was still missing, and some mail was hitting the Blackberries but not hitting the Outlook inbox. This was a mystery to both us and to Radiant until one of my client’s sales staff reported that he found some missing email in the shaw web mail account!
With total shock, we opened up the Shaw mail accounts and found our missing mail from the last two months.
On Tuesday of this week we reported the problem and Shaw told us that one of their servers had an incorrect “zone record,” and they had deleted it: Problem solved. The next day, Wednesday, one of our salespeople and our operations manager were still not receiving mail from key customers (I was not working that day.)
Thursday morning we log into the old Shaw accounts to retrieve our mail and discover to our shock (we get shocked a lot in this story) that our mailboxes are deleted but we are still not getting mail from about 5% of our customers.
Okay, by this time I am super-upset. This problem has been a major irritation in the front office for about two months and Shaw has been worse than useless, they haven’t fixed the problem, but made it worse because we can’t even access the mail that is going to the wrong mailboxes.
So we get on the phone again to Shaw and after a lot of conversation we get them to restore the mailboxes. We phone a client who we know is getting their mail sent to the wrong box and ask them to send a test message. Yup, it ends up in the Shaw box, not at our Exchange server.
We phone Shaw, actually Sherri from Radiant phones them who so far has been great in helping us with this crisis. But after a long conversation with the Shaw, their brilliant solution is to delete Shaw mailboxes again!
And we respond that we did that Tuesday and it didn’t fix the problem why do you think it would work now? And oh, the answer is that the servers “take time to synch.”
By this time, our trust in Shaw is close to zero – actually it’s in negative territory – so we answer well let’s give it even some more time then. So we are downloading our mail both from our Exchange server and from the Shawcable mail server. And in a week’s time, week, we will carefully run a test to see if we can get 100% of our mail, not just 95%.

Update Thursday evening: And now we mail is not getting to the owner's mailbox at all. Is this Shaw's fault or something else? At this point in time, I would place blame for upcoming economic apocalypse on Shaw.

Do me a favour, please go to digg.com and dig up this story

{ 2 trackbacks }

A nice thing to say about Telus | Blue Butterfly Consulting
December 15, 2008 at 2:27 pm
Grandma and the Slow PC | Prepress Pilgrim
July 8, 2009 at 2:13 pm

{ 6 comments }

Dennis October 26, 2008 at 7:01 pm

“Launch the nuke into the kill zone, NOW!”

EJ November 3, 2008 at 5:33 pm

DJ,

Have you used a tool like Dig, http://networking.ringofsaturn.com/Tools/dig.php, to check your client’s zone records in the DNS? If they’re not what you expect, that leads to the next question…

Who hosts your name servers?

If the answer is Shaw, consider migrating to something good and cheap like easydns.com that you can manage yourself and not have to wait for people to do it while executing the expert yoga positions you allege they can pull off, which must take a lot of concentration.

Of course, this requires your client having control over their root domain registration.

If you’re lucky, your client registered their domain through something like webnames.ca and you can get the admin account and make the nameserver change yourself.

The process goes like this:

1) set up an easydns.com account (or whatever).
2) copy your zone files over to the new name servers
3) test them to make sure they’re answering correctly when you query them.
4) change your root domain record to point to the new name servers
5) forgive Shaw – it’s good for your soul

–E

RG November 14, 2008 at 8:07 pm

We have exactly the same problem as you. Has your problem been resolved? if so, how did you get it resolved. If not, I would like to contact you for further info and possible action.

admin November 15, 2008 at 7:04 am

We finally fixed it by telling Shaw to nuke the mailboxes. That is to say, even though the old Shaw boxes were disabled, they were still grabbing mail. So we insisted on having them deleted, and finally the mail got routed out of the shaw portion of the internet and to the proper mailboxes at radiant. The procedure went as follows:
1. Log onto Shaw mailservers one last time and scoop up the mail.
2. Phone 1st tier support and them to blow away the mailboxes.
3. Wait a few hours and test from a known problem email address.
4. You might have to phone Shaw again in a few hours and ask them to escalate the call to their DNS people (second tier) to fix the problem once and for all.

But you have to blow away your mailboxes, otherwise you are stuck at 1st tier forever, with Shaw support just shrugging their shoulders and passing you around like a hot potato.

rak August 22, 2010 at 12:00 pm

Shaw was “stealing” your mail? That a funny way of putting it… How was it even getting to them? Magic?

Dan October 4, 2010 at 7:42 am

I’ve seen this happen when the mail provider is also the DNS provider. Moving the mail and DNS somewhere else without properly terminating the mail accounts and DNS on the old provider will result in all mail originating from the old email provider to continue to get delivered to the old obsolete mailboxes. The only way to fix it is to go back to the old provider and get them to remove the mailboxes and the DNS entries.

Comments on this entry are closed.