"National alert from pipeline accident"

http://www.seattle-pi.com/pi/local/pipe09.shtml

-- Linkmeister (link@librarian.edu), July 13, 1999

Answers

[Fair Use: For Educational/Research Purposes Only]

National alert from pipeline accident

Regulators urge review of computer systems

Friday, July 9, 1999

By SCOTT SUNDE

SEATTLE POST-INTELLIGENCER REPORTER

Breakdowns in the Olympic Pipe Line Co. computer system just before and during last month's deadly accident in Bellingham have so alarmed federal regulators that they have issued a nationwide warning.

The federal Office of Pipeline Safety issued the warning this week to the 2,000 operators of liquid and natural-gas pipelines in the United States. It urged them to make sure that computer systems used to operate and monitor pipelines are working properly.

The advisory details a series of computer failures on June 10 around the time Olympic's 16-inch line leaked up to 277,000 gallons of gasoline into Bellingham creeks. Gasoline vapor later exploded in flames, and two 10-year-old boys and a teenager were killed.

After the accident, Olympic acknowledged that its computer system crashed on the afternoon of the accident. The computer problems may have kept Olympic personnel from reacting quickly to the leak, regulators said.

The computer system is known as SCADA -- supervisory control and data acquisition. Such systems are common in the industry, though they may have been built at different times by different manufacturers.

All such systems go under the generic name of SCADA.

Some companies, including Olympic, add to their computer systems leak- detection equipment. Olympic's uses such information as temperature and pressure to detect leaks.

But investigators with the Office of Pipeline Safety have determined that Olympic's computer system broke down on the day of the accident.

"Immediately prior to and during the incident, the SCADA system exhibited poor performance that inhibited the pipeline controllers from seeing and reacting to the development of an abnormal pipeline operation," regulators said in their advisory.

The Office of Pipeline Safety is part of the U.S. Transportation Department.

Regulators did not name Olympic in the advisory. But Patricia Klinger, a spokeswoman for the Office of Pipeline Safety, acknowledged that the incident mentioned in the advisory and prompting the warning was Olympic's Bellingham accident.

The message to other pipeline operators, she said, is to "take extreme caution."

"We don't want to see this repeated."

Gerald Baron, an Olympic spokesman, said the company believes federal regulators are being prudent in sending out the advisory to pipeline operators.

Baron could not discuss the details of the computer problems and cautioned against focusing on computer difficulties or any other single factor as a cause of the accident.

Regulators believe Olympic's computer system typically operated at 65 percent to 70 percent of capacity.

But on June 10, the system had an internal database error. That error, plus the demands put on the computer by the leak, "hampered controller operations," the advisory said.

"The combination of the database error, the inadequate reserve capacity of the SCADA processor and the unusually dynamic changes that occurred during the upset condition appear to have combined and temporarily overburdened the SCADA computer system," regulators said.

"This may have prevented the pipeline controllers from reacting and controlling the upset condition on their pipeline as promptly as would have been expected."

Regulators also said that modifications made to the computer system after it was installed may have caused it to malfunction.

The Office of Pipeline Safety ordered Olympic on June 18 to find out what went wrong with its computer system and correct it. It also ordered the company to make a comprehensive review of its SCADA system.

Those demands came as part of a corrective order that closed the upper 37 miles of the 400-mile pipeline. Regulators also ordered the company to undertake several safety modifications and reviews.

The Office of Pipeline Safety may soon issue additional orders regarding Olympic's pipeline , Klinger said.

P-I reporter Scott Sunde can be reached at 206-448-8331 or scottsunde@seattle-pi.com

---------------------------------------------------------------------

-- Linkmeister (link@librarian.edu), July 13, 1999.

Very sad that 3 young people died,

There will never be a 100% perfect computer system. There is not one for NASA, MIR, or anywhere.

Interesting the authorities say "You must make sure it will not fail -.. find out what happened, .. make sure it will not happen again." The space shuttle saw a challenge with 5 computers on board, they were all to check each others performance and remain a hot standby that automatically switched into service when the results did not match. That system had so many problems it delayed mission schedules.

-- Living in (the@real.world), July 13, 1999.

The 6800 had bugs the 68hc11 has bugs (many SCADA and similar controls) The 386 had bugs the 486 had bugs the Pentium 1 had serious bugs all newer Pentiums and others also have bugs (not as visible as P1) most software has bugs we have just excepted the small number of halts or anomolies we restart, reboot or power down and try again.

The only high end unit that has a perfect operational record (so far)is the auto reduntant para system running LINUX.

-- Living in (the@real.world), July 13, 1999.

To me it indicates an over-reliance on an over-complex technology. The system that performed a fundamental safety operation, viz. shutting down the pipeline if the pressure dropped (meaning a big leak) was down.

Clearly, someone didn't regard this as a safety-critical function, and three innocents are dead as a result. A safety-critical function would, or should, have redundant back-ups, preferably very simple non- computerized ones. For example, a pressure sensor linked to a motorized valve that shuts if ever the pipeline pressure drops too low -- the inverse of the safety valve on a steam boiler. Would you be allowed to run a steam boiler if the safety valve was "down", ie known to be jammed shut? Answer: no, in most juristictions this would be criminal.

I may be being over-simplistic in my trigger condition, but the fundamental rule applies. If it's safety-critical there should be a completely separate, autonomous, simple-as-possible, emergency shutdown mechanism, and I'm amazed there wasn't.

Lawsuits inbound? For once I hope so.

-- Nigel Arnot (nra@maxwell.ph.kcl.ac.uk), July 13, 1999.

Thanks --Linkemeister for the informative post.......

-- kevin (innxxs@yahoo.com), July 13, 1999.

Explosion in Bellingham, Wash

xxxxxxxx xxxxxxxx xxxxxxxx

-- Ashton & Leska in Cascadia (allaha@earthlink.net), July 13, 1999.

ADB99-03 Potential Service Interruptions

PIPELINE SAFETY ADVISORY BULLETIN

ADVISORY BULLETIN: ADB-99-03 Date: July 7, 1999

 

To: Owners and Operators of Hazardous Liquid and Natural Gas Pipeline
Facilities

Subject: Potential Service Interruptions in Supervisory Control and Data
Acquisition Systems

Purpose: Inform pipeline system owners and operators of potential
operational limitations associated with Supervisory Control and Data
Acquisition (SCADA) systems and the possibility of those problems leading
to or aggravating pipeline releases.

Advisory: Each pipeline operator should review the capacity of its SCADA
system to ensure that the system has resources to accommodate normal and
abnormal operations on its pipeline system. In addition, SCADA configuration
and operating parameters should be periodically reviewed, and adjusted if
necessary, to assure that the SCADA computers are functioning as intended.
Further, operators should assure system modifications do not adversely affect
overall performance of the SCADA system. We recommend that the operator
consult with the original system designer.

Background: During an Office of Pipeline Safety (OPS) investigation of a
recent pipeline incident, OPS inspectors identified inadequate SCADA
performance as an operational safety concern. Immediately prior to and during
the incident, the SCADA system exhibited poor performance that inhibited the
pipeline controllers from seeing and reacting to the development of an abnormal
pipeline operation.

Preliminary review of the SCADA system indicates that the processor load (a
measure of computer performance utilization) was at 65 to 70 percent during
normal operations. Immediately prior to an upset condition occurring on the
pipeline, the SCADA encountered an internal database error. The system
attempted to reconcile the problem at the expense of other processing tasks.
The database error, coupled with the increased data processing burden of the
upset condition, hampered controller operations. In fact, key operator
command functions were unable to be processed immediately prior to and
during the abnormal operation. It is possible that post installation
modifications may have hampered the system's ability to function
appropriately.

The combination of the database error, the inadequate reserve capacity of the
SCADA processor, and the unusually dynamic changes that occurred during
the upset condition, appear to have combined and temporarily overburdened
the SCADA computer system. This may have prevented the pipeline
controllers from reacting and controlling the upset condition on their pipeline
as promptly as would have been expected. For further information, contact
Chris Hoidal, Director, OPS Western Region at 303-231-5701.

 

Return to the Advisory page.

-- Brian (imager@home.com), July 13, 1999.

Remember "follow the money"? How about "follow the software".

Who wrote it? Probably not Olympic, probably a small shop writing this highly specialized stuff. How many people are in this software shop, writing the new stuff under incredible pressure, trying to be done in time to install before 12/31/1999?

How many pipelines are running on this software, waiting to load the new modules for Y2k? How many different versions are out there, each not quite the vanilla version, each with its own wrinkles? How many Bellinghams are we going to see this summer?

-- bw (Home@Puget.Sound), July 13, 1999.

One bug + one program + one computer + one process control failure + one pipeline (failing at one after surges from low pressure, no less!) + one flaw (structurally) = one failure = 3 deaths.

Damn. Sorry it happened. But if the rest learn, whether as professionals learning from an expensive error in somebody's else's system, or as a bureacratic employee frightened into reacting to a requirements letter from the fed's, perhaps other lives can be saved next winter.

Perhaps many, many lives if the secondary effects of losing steam heat, water, gasoline, fuel oil, and nat. gas failures are included.

-- Robert A Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), July 13, 1999.

Baron could not discuss the details of the computer problems and cautioned against focusing on computer difficulties or any other single factor as a cause of the accident.

Interesting comment. Especially when the article begins with... Breakdowns in the Olympic Pipe Line Co. computer system...

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), July 13, 1999.

"It is possible that post installation modifications may have hampered the system's ability to function appropriately."

What dear reader do you think those modifications were exactly? What is the mysterious data base error?

I'd like to know, but unfortunately everyone up there has there own attorney and they're not talking. I have heard from several people who've been to that control center that it is quite advanced. State of the art. Goes to show you, even the best are subject to visit's from Mr. Murphy now and then. I hear Murphy's got one hell of a New Years party planned.

-- Gordon (g_gecko_69@hotmail.com), July 13, 1999.