[Fair Use: For Educational and Research Purposes Only]

Regional News - June 29, 2000

911 Glitch in Switch, Bell Says

By Michael Hasch

TRIBUNE-REVIEW

Bell Atlantic investigators believe an outdated switching system contributed to Sunday's failures in the 911 systems in the city of Pittsburgh as well as Beaver, Lawrence and Westmoreland counties, a spokeswoman said Wednesday.

Bell Atlantic spokeswoman Sharon Shaffer also said she has yet to find any documentation that supports the contention of 911 officials in all four areas who have said they were promised the backup system would be in operation within 10 minutes.

Callers could not make 911 calls Sunday from 5:25 p.m. until about 6:30 p.m. when the backup system went into operation in both the city and Beaver County.

It took Bell Atlantic technicians until about 7:15 to get the backup system working in Westmoreland County and until 7:48 p.m. in Lawrence County.

As far as can be determined, county officials have said, no emergencies went unanswered during the problem.

Shaffer said preliminary findings of an internal Bell Atlantic investigation show a switch failure led to a computer software failure.

The switch, made by Rockwell International and housed in Bell Atlantic's headquarters on Seventh Avenue, Downtown, routes all 911 calls to the emergency center in the area of the person who phoned for help.

"It is one of the oldest switches in the Bell Atlantic system," Shaffer said. "It was state of the art (when it was installed) in the early '80s. Rockwell is no longer manufacturing switching equipment and declared this one discontinued."

She said Bell Atlantic has recommended the communities install updated technology similar to that in use in both Mercer and Washington counties where two switches housed in separate buildings handle the 911 calls. If one switch fails, the other is not affected, Shaffer said.

Long before Sunday's problem, she added, Beaver County officials decided to upgrade to the new system. It should be in operation in late fall, she said.

Shaffer acknowledged that upgrading technology is costly and means replacing much of the equipment in use at the 911 centers.

She blamed some of the delay in implementing the backup system Sunday to the severity of the damage in the switch. Technicians had to bypass the system not only for Pittsburgh but also for the three counties, which took time.

She also said that at some point, the three affected counties and the city of Pittsburgh agreed to use the same Rockwell switch. On Sunday, technicians implemented four separate backup operations to restore service.

Shaffer agreed the delay was much longer than Bell Atlantic would have liked and that it remains the subject of the company's investigation.

She also said she is unaware what commitments may have been made in years past to the communities concerning the amount of time it would take for the backup system to activate.

She added that she expects that when the investigation is completed, changes will be made in an attempt to alleviate problems and delays in the future.

http://triblive.com/news/rbel0629.html

-- (Dee360Degree@aol.com), June 29, 2000

Answers

Bell Atlantic says 911 gear outdated

Thursday, June 29, 2000

By Lillian Thomas, Post-Gazette Staff Writer

Bell Atlantic said yesterday that the 911 systems in three counties and the city of Pittsburgh failed on Sunday because emergency officials had elected not to replace outdated equipment.

Emergency officials responded that Bell Atlantic had never informed them the equipment was dangerously outmoded.

"I am furious," said Frank Jannetti, Lawrence County's emergency services director. "If they had told us the equipment no longer works, we would have gotten off it. But no one ever said it was a problem."

The 911 system failure caused disruptions of up to 2 1/2 hours in 911 service in the city and Lawrence, Beaver and Westmoreland counties.

When the computerized system known as the "Rockwell switch" failed, calls had to be manually rerouted by Bell Atlantic workers, said Sharon Shaffer, Bell Atlantic spokeswoman.

The 911 switching equipment is 15 years old, is no longer manufactured and is the only one of its kind left in Pennsylvania, Shaffer said.

"Although we do make every effort to implement this procedure quickly it does take time when you have a severe failure like Sunday," said Shaffer. "It's for this reason that Bell Atlantic has told all four 911 centers that they'd all be better served if they moved their 911 call processing from Rockwell single-switch technology to more robust technology that's available today and currently operating in Mercer and Washington counties."

Wayne Harley, assistant director of Beaver County's 911 center, said he expected to get an explanation for the outage. Instead, he was shocked at what he regarded as having the blame put on his center.

"We're the customer," Harley said. "Did they forget that? We're buying a service from them just like a residential customer buys service from them. And we pay Bell Atlantic to the tune of $15,000 a month for that service."

Kathy Kraus, acting public safety director for Pittsburgh, said the city was aware that the equipment was old and had been looking at upgrading the system. "We know there's better equipment. Technology's always improving. But we were never put on any kind of notice that it was dangerously unreliable. And they continue to take taxpayers' money to operate it."

Shaffer said the internal investigation was continuing, but, she added, "Bell Atlantic believes that it was a hardware failure in the switch, which then led to a software failure that prevented the system from processing the calls. That's why when many people called the number they got silence."

Bell Atlantic had to simultaneously put into place backups for the four systems, she said. "It's not a question of someone flipping a switch and calls going to another route. You have to disconnect all the 911 lines that go to the Rockwell. Then you have to issue commands to reconnect those lines to another switch. They were actually running programs to do this.

"For someone to think that that should happen in a few minutes' time - - there was just no way."

Officials said they did indeed think that would happen in a few minutes and that they had never been told that the system could lead to disruptions like the one Sunday.

In fact, Kraus said, a system failure a couple of years ago was corrected much more quickly. She did not have the date or time it took to rectify the problem, but she said it was quick enough that most people were unaware of any interruption.

Shaffer said Bell Atlantic doesn't give time estimates for recovery from such system failures to the 911 systems that use its equipment. "Within the Bell Atlantic system we don't give any specific time that the system will be up because it depends on the particular situation."

Shaffer said it was common knowledge within the industry that a "dual switch environment" is what's needed for reliable service.

In such a system, twin switches are installed in two different locations. If one fails the other continues to operate. "This is recommended as best practice by the National Reliability Council," an industry group that issues guidelines and recommendations on such systems, Shaffer said.

"Bell Atlantic has made various suggestions over the years. The decision and choice of system is the responsibility of each public official," she said. "If they choose to go with status quo, we maintain that technology."

But that's what emergency officials feel that Bell Atlantic did not do.

"They own this equipment, not the city of Pittsburgh," said Kraus. "We pay them over $300,000 a year to provide us with service and part of that is to maintain the switch" and to provide a timely backup protocol, she said.

Jannetti said Lawrence County never would have used a system that wasn't thought to be reliable.

"It's Bell Atlantic's equipment. They are delivering 911 calls over it. If it's not adequate to deliver those 911 calls, they should take it out of service or replace it," Jannetti said.

The 911 officials were also irritated to hear about Bell Atlantic's take on the situation through reporters. Kraus said she had not been contacted by Bell Atlantic regarding the Rockwell switch.

Neither Jannetti nor Harley has heard from a Bell Atlantic official since Sunday's outage.

"I have never experienced this before, where I was notified by the media," Jannetti said. "It's outrageous."

Jannetti and Harley said they were already thinking of upgrading their equipment, but in light of Sunday's system failure, they said, they may reconsider doing business with Bell Atlantic.

"The county has a proposal to go with the dual tandem system with Bell Atlantic," said Beaver County's Harley. "Frankly, at this point in time, with this kind of attempt by Bell Atlantic to shirk responsibility, I really wonder if we should be going with them to replace the Rockwell switch."

"We are really going to rethink what we're doing, too," Jannetti said.

The switchover to a dual tandem system will cost Beaver County about $546,000, Harley said. The county commissioners still have to approve the expenditure, which is not included in the emergency center's $1.5 million annual operating budget.

Price is the main reason Beaver County has been waiting to make the switch, he said.

http://www.post-gazette.com/regionstate/20000629emergency3.asp

-- Martin Thompson (mthom1927@aol.com), July 01, 2000.