MCI Network still down after 8 days...teams still searching for root cause

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Read this for yourself and SEE how "fragile" this world of interconnectedness really is.....and how complex and difficult to fix on failure....the expert teams are still scratching their collective heads...many ISP's are losing major portions of their customer bases.

See for yourself... http://www.news.com/News/Item/0,4,0-40499,00.html?macintouch.com

-- John Amphlett (vacajohn@jccomp.com), August 14, 1999

Answers

I have been wondering.... Just what company would make huge upgrades to computers/equipment at this time if it were not related to Y2K? From where I sit (system administrator), all work has been halted to everything else but Y2K. I am SURE that this is the case in most companies. New Financials going in (big trouble), HR upgrades to become compliant.... Forget testing... JUST DO IT! Deal with the problems as they come.

Adding more lines possibly? Could be, but that wouldn't be classified as an upgrade.

Ten years in telephony here, and something just doesn't smell right on this picture.

I wonder if Pony Express still works? :)

Seems like spending is down, unless it is Y2K related for most IS or IT (depending upon your organization) departments. All spending is upgrading or fixing what you have. No new "development" until this "blows over".

-- (cannot-say@this.time), August 14, 1999.


Cannot Say,

You are right on the money! The average Joe has no concept of the magnitude of activity behind the scenes right now. IT shops are in a "if it ain't broke, don't fix it" mode. Granted every publicized problem is not Y2K related, but you better believe that a lot of the implementations of new releases are Y2K based deployments.

-- MoVe Immediate (MVI@yepimhere.com), August 14, 1999.


"I wonder if Pony Express still works? :)"

It did, eventually, in "The Postman" :)

-- Andy (2000EOD@prodigy.net), August 14, 1999.


Nice to see you drop in MoVE Immediate!! Are you still in Dee Cee? Keep us posted on your activities!!!

-- jeanne (jeanne@hurry.now), August 14, 1999.

Let's see here, another refinery explosion yesterday, MCI down all week, another power plant explosion tonight....I think Y2K is really beginning to rear it's ugly head. On the power plant explosions, how many does that make so far this year? 5? 6? anybody keeping count? I can't ever remember a power plant blowing up before this year and suddenly there's about one a month popping off. Methinks mr. Beech may have had something on those embedded chips.

-- Nikoli Krushev (doomsday@y2000.com), August 14, 1999.


Nikoli,

Welcome back - we've missed your valuable input - anyone know where Arlin is these days? Is he all tuckered out with this forum? can't say I blame him... :)

As for all those pesky explosions, yup, you're just imagining things... :)

-- Andy (2000EOD@prodigy.net), August 14, 1999.


FWIW,

I received this email from my service provider this afternoon. No mention of Y2K. We've been disconnected at least 3 times in the last 7 hours. I've also posted this on another earlier thread

Subject: *IMPORTANT* Notice of Emergency Network Maintenance

Date: Sat, 14 Aug 1999 15:10:14 -0400 (EDT)

From: feedback@bellsouth.net

Valued Customers:

At 11:15 pm 8/13/99 WorldCom, our Global Service Provider, notified our Network Operations Center of the need to perform emergency maintenance on their Frame Relay network beginning at 12 Noon (EDT) Saturday 8-14-99 and finishing at approximately 12 Noon (EDT) Sunday 8-15-99.

During the course of this emergency maintenance, you may or may not experience the following: congestion over the network, latency and potentially, loss of connectivity. The work being performed by WorldCom necessitates the complete shutdown of all frame relay switches within the WorldCom network, and a controlled, one by one, reinstatement of each frame relay switch back onto the network.

We have been assured by WorldCom that every effort will be made to reduce the impact to our network and to resolve the issue necessitating the emergency maintenance as expediently as possible.

We will notify you once we have received confirmation from WorldCom that all work has been completed.

Thank you for your patience and continued business,

BellSouth.net

Consumer Communications & Relations

feedback@bellsouth.net

-- Spindoc' (spindoc_99_2000@yahoo.com), August 15, 1999.


MVI,

I just wish that I had a way to contact you. There is so much more I want to know from your shoes, and wish to tell you from mine.

Things aren't looking too good, as I am sure you know.

Nik, glad you are back.... don't be a stranger!

-- (cannot-say@this.time), August 15, 1999.


Guess Maria's embarrassed, but she told her friends this:

Debunking Y2K Webboard

Eddie's place has been asking me this one Sunday, 15-Aug-1999 00:25:01

152.163.204.72 writes:

People have been pretty quite about it. Most claim that it's Lucent's faulty equipment. We put in the system Friday night and didn't follow through and found the problems Monday morning. (That's what I hear). I don't believe it has anything to do with Y2K. We remediated and tested our business systems last year and I don't think it was a part of that. But as you could imagine Eddie's place has been pointing fingers at Y2K. Of course, they know and see all. Wish they would let MCIW know of their profound experience.

-- Troll- (watching@the.bridge), August 15, 1999.


Some big hints here...

The work being performed by WorldCom necessitates the complete shutdown of all frame relay switches within the WorldCom network, and a controlled, one by one, reinstatement of each frame relay switch back onto the network.

Looks like a hardware failure, disconecting all and putting them back in line one at a time. Too bad they don't have any diagnostic software or a cludge box to use to troubleshoot the problem. They probably have IT's working on the problem *grok*

We put in the system Friday night and didn't follow through and found the problems Monday morning.

Typical big business attitude. They evedently did not think anything could possibly go wrong, so they did not bother to check out the system after installing it. It does not sound like a Y2K problem, but just another case of inadiquate procedures.

-- Cherri (sams@brigadoon.com), August 15, 1999.



Could this problem be fixed if: The power was out

Damn Bill I can't see squat how the hell are supposed to fix this

There was rioting in the streets

Sorry boss those bastards are burning and looting and moving this way,I've got kids here The telepnones didn't work I donno John I can't get thru to the dispatcher

8 days and counting: sure am glad we got 3 days of supplies honey,john k. told us storm would only last 2 maybe 3 days.

..........................

Lets see 8 days and counting

-- Desertj98 (jturner@ptway.com), August 15, 1999.


This link is to a story about ISP's shutting in their doors and LOOSING THEIR BUSINESSES due to this problem. I personally switched ISP's yesterday for that very reason. I had been down last weekend, during this past week and it was starting again this weekend. So I dropped the hammer on my provider and switched to one with redundant backbone services.

http://www.news.com/News/Item/0,4,0-40499,00.html?st.ne.lh..ni

I wrote this earlier to Maria and she has not replied....

"Gordon, the problems MCIW has been experiencing have nothing to do with Y2K. " -And how the hell would you know?

The KOS was correct. I am trying to validate your credentials. I have no inside knowledge of MCI at all. NONE. I never claimed to. If you know what the problem is, genius, then why not enlighten us all. Dazzle me with your profound insight to frame relays. Better yet go help MCI fix their fucking problem.

My friends at "bigass consulting" reference their "tech hotline" each day and this week's message said "MCI is still experiencing problems with their frame relay network. They are uncertain as to when they will be able to fully restore service". He then told me that they are experiencing problems with their clients all over the globe.

I just got done switching ISP's. The reason that I could not reply to you is that your damn buddies at MCI hosed my ISP. So screw you and you're indignant little attitude. My service has been down for two weeks. I've spent several days at work watching 100's of thousands of dollars slip away due to your freaking friends at MCI. Piss off.

-- Gordon (g_gecko_69@hotmail.com), August 14, 1999.

-- Gordon (g_gecko_69@hotmail.com), August 15, 1999.


As of Sunday 9:00 PM ET any new updates on this situation with MCI?

-- thinkIcan (thinkIcan@make.it), August 15, 1999.

http://biz.yahoo.com/prnews/990815/ms_mci_res_1.html

Sunday August 15, 5:54 pm Eastern Time

Company Press Release

SOURCE: MCI WorldCom

MCI WorldCom Restores Service to Its Frame Relay Customers

CLINTON, Miss., Aug. 15 /PRNewswire/ -- MCI WorldCom (Nasdaq: WCOM - news) today successfully completed recovery of its domestic frame relay network platform, following a planned outage that began on Saturday, August 14. The company has restored service to its customers using this frame relay network platform.

``We very much apologize for the difficulties this frame relay network outage caused our customers,'' said Bernard J. Ebbers, MCI WorldCom president and CEO. ``We have resumed providing service to the customers who were impacted and will be working hard to regain their full confidence. MCI WorldCom is very committed to providing the highest levels of network reliability on all of its services to all of its customers.''

SOURCE: MCI WorldCom

-- c (lurking@home.now), August 16, 1999.


>CLINTON, Miss., Aug. 15 /PRNewswire/ -- MCI WorldCom (Nasdaq: WCOM - news) today successfully completed recovery of its domestic frame relay network platform, following a planned outage that began on Saturday, August 14. The company has restored service to its customers using this frame relay network platform.<

If the planned outage began on Saturday the 14th, just what do you suppose was wrong during the week before when they screwed over the CBOT so badly, and then kind of upset the CBOT dude by not getting it running again when they promised?

They were down for 8 days weren't they? Was that week just a "pre- planned outage" outage? Or an "unplanned outage" outage?

Ms. Maria; you really do need to talk to those folks about their spin control.

S.O.B.

-- sweetolebob (buffgun@hotmail.com), August 16, 1999.



Moderation questions? read the FAQ