If you have never worked a support desk before, it is quite an experience during a long outage. Comcast has had a routing issue for about 9 hours now where certain routes go nowhere after a certain hop:
Tracing route to iss.net [209.134.161.35] over a maximum of 30 hops:
1 1 ms 1 ms <1 ms 192.168.1.1
2 117 ms 11 ms 7 ms 73.8.64.1
3 9 ms 8 ms 8 ms GE-2-2-ur01.N4 atlanta.ga.atlanta.comcast.net [68 .86.107.189]
4 25 ms 9 ms 8 ms te-9-4-ur01.b3marietta.ga.atlanta.comcast.net [6 8.86.106.129]
5 9 ms 8 ms 10 ms te-9-4-ur02.b0atlanta.ga.atlanta.comcast.net [68 .86.106.125]
6 9 ms 8 ms 8 ms te-9-4-ur01.b0atlanta.ga.atlanta.comcast.net [68 .86.106.13]
7 9 ms 9 ms 8 ms te-9-2-ar01.b0atlanta.ga.atlanta.comcast.net [68 .86.106.9]
8 * * * Request timed out.
9 * * * Request timed out.
10 * * * Request timed out.
My first thought...oh well isn't this just great. These types of problems are usually hard to find and report. I have an idea what happened but figured I would torute myself and call their little support line and see which support locale is answering the phones today. I have my money on SE Asia.
During the popt(press one press two) motions, there was a message from what sounded like another non-english speaking genius who reported the outage as this:
Download rec_+14042662278_13_Sep_2007_23_36_41.mp3
If you had a hard time understanding that,you are not alone. Here is what he said:
"We are currently experiencing technical difficulties with internet access, preventing the loading of several websites; in all of the atlanta region. This is not a comcast related issue but we are closely monitoring the situation with our partners in the field."
Classic response. No responsbility, no real problems, underplay the extent of the issue, and give no ETR for getting this fixed. I love the world I left.
So if anyone from comcast happens to stumble upon this, here are a few things to consider:
- never give numbers to explain the extent of an outage. Serveral means 7-10. This outage is effecting access to 100's if not thousands of sites and services. I have counted over 15 things I can't reach right now including my office VPN. Thanks.
- Never ever say "it's not our fault". If it is effecting your customers you provide a service to, I don't care if the FBI came in and unplugged a bunch of boxes, you say the party line "We are investigating the issue and will have service restored as soon as possible. We apologize for the trouble." You don't say "not our issue" cause it is since it effects your customers. I know this is going to end up being come peering issues with Level3/GC/Sprint, someone deprovisioned a circuit, something got cut in atlanta (Do contractors in Atlanta get paid more if they cut a utility line?) I just cannot believe there is not some redundancy built into their routing somewhere.
- Provide the time of the update and ETA. One sure way to get the crazies calling in over and over is to make them think there is no progress being made or if they are hearing the same message. Give people hope. Internet access is not a novelty anymore and most folks depend on it for access to the office, travel, social stuff, and keeping in touch with the world.
In a way, I guess I miss the callcenter stuff cause I would be the person to manage this situation from customer communication, status updates, pushing and escalating with any vendors involved, and then getting kudos for taking care of it. Instead, most ISP's and ILEC's are made up of underpaid, underqualified people that either think they know more than they do since they got 10 certifications or they have managers that can't manage their way out of a car. The high level of service this industry was known for 10 years ago is gone and most of us have moved into managing the coporate IT stuff in one way or another. I just want comcast to use some com-mon sense and have REDUNDANCY plans so this kinda crap doesn't happen. This will cost them 10% of the subscribers who will be renewing or thinking about switching service over the next month...then it will all be forgotten.
Subscribe to:
Post Comments (Atom)
1 comment:
ah the good old days. my favorite part of an outage call with a vendor would be the moment where, after hours of throwing data back and forth (ususally it was just "forth", to them) and trying to explain what the problem was on their end while they deny any issue (and don't look into it), the vendor guy would get really quiet and go " . . . oh." ta da!
Post a Comment