Link

[snip]

...the Coast Guard has not been immune from early manifestations of the problem. In January 1997, it encountered a bug at the Coast Guard Institute in Oklahoma City. A system was supposed to generate a notice to people enrolled in a correspondence course that must be completed in three years. When the system first had to process dates in 2000, it malfunctioned and deleted hundreds of student records. "The staff of the institute required two weeks to correct the problem," Naccara told a House committee. "It was a sobering wake-up call."

[snip]

A single ship can have hundreds of microprocessors ("chips" to most of us) working unseen in systems that control functions such as ventilation, ballast, navigation, communications, detection of fires and other hazards, and so on. Operators of one cruise ship thought they had brought it into full Y2K compliance, Naccara says, but when they turned the ship's clocks forward to Jan. 1, 2000, in a test, the stateroom doors all locked automatically and stayed that way, because of an overlooked chip.

Moreover, the Coast Guard is responsible for U.S. port safety, so it must concern itself with items such as fuel depots and pumping systems, cranes and other cargo-handling equipment, and shoreside utilities. In today's automated ports, a systems failure could lead to a major environmental disaster or loss of precious energy resources. "According to the Energy Information Administration, more than 50 percent of the oil consumed in this country comes to us from foreign sources through our ports," Naccara told the House committee, "Any disruption of the cargo and especially oil flow, for even a few days, would have a discernable effect on our economy, particularly during the winter heating season."

[snip]

The external systems issues, often labeled "external interfaces," are complex because the working environment for businesses and government today is highly networked, and few organizations have kept track of all the ways they are connected to the outside world. Federal agencies reported to the General Accounting Office earlier this year that they have more than 180,000 data exchanges with outside parties.

The true number may be much greater. People who think they have tracked down all those with whom their agency exchanges data keep uncovering new and unsuspected links. Once they've inventoried the connections, they may need to gather information about the other party's technical approach to Y2K and the timing of repairs so that the two efforts can be synchronized. Where mission-critical federal systems exchange data with others, formal, written agreements between the parties are needed.

The multiplicity of networks may mean wireless, telephone, cable television and several kinds of data networks must be checked out. In short, Naccara says, the complexity of the external interfaces "in itself assures us of some failures."

The chips are proving troublesome for everyone, inside the Coast Guard and outside, Naccara says, because even though they are ubiquitous, information about them is difficult or impossible to obtain. A single chip may perform a few known functions in a machine, but it may have dozens of unused functions built in by the chip manufacturer, he says. If just one of those dormant functions recognizes date-related information that reaches it, the chip can perform unpredictably.

Manufacturers don't always know precisely what chips they included in a product, and sometimes the chip manufacturer cannot be located to answer inquiries about whether the chip has date-related functions. Some makers of chips and products that use chips aren't responding to queries about Y2K, on advice of their lawyers who are worried about liability. Although a recent federal law reduces this exposure, not everyone believes candor is advisable. Moreover, definitions of Y2K compliance are elastic, despite efforts to make them more precise.

Furthermore, seemingly identical pieces of factory-made equipment can have different versions of the same chip. That means that if a Coast Guard office has five of the same fax machines, for example, testing one of them isn't enough. All five must be tested. More complex machinery tends to have more chips. A single huge crane in a port could have 150 chips, federal Y2K czar John Koskinen told an audience last year. Some organizations reportedly have been pleasantly surprised to find that their embedded-chip problems were less serious than had been feared, but the chips still need to be checked.

[snip]

"Just changing the date and trying it [the system] isn't enough," says William Curtis, the Defense Department's Y2K coordinator. Operational testing of all the modules and systems that must work together, using real or nearly real data--testing that's sometimes labeled "end-to-end"--is the only way to ensure all the pieces of the puzzle are in place.

Well before other agencies began Y2K testing, the Coast Guard launched a program called "Operation Millennium Dawn" to ensure that it's ready to do whatever is needed on Jan. 1 and afterward. The scope of this program is daunting. Agency officials are looking at ways units could communicate if telephone systems or data lines are inoperable. For example, they may pre-position some cutters in port to serve as backup communications hubs.

The Coast Guard is preprinting forms to capture and pass along information in the absence of vital computer systems. Teams are considering where people can be stationed to observe vessel traffic that today is tracked by radar and similar systems. Agency executives are planning to have most operations workers on duty the weekend of Dec. 31, 1999, through Jan. 2, 2000, and they are considering how they could shelter and care for employees' families if conditions are difficult and the employees are reluctant to leave home and report for duty.

[snip]

"You have an infinite number of variables to contend with here," Naccara says as he discusses the embedded chip problem and the way chips can interact with software. He and others in similar positions are reluctant to predict precisely what will happen at year's end. They know there will be systems failures, but no one knows to what extent those failures will cascade and have unexpected consequences.

[snip]

-- regular (zzz@z.z), May 03, 1999

Answers

"The Coast Guard is preprinting forms to capture and pass along information in the absence of vital computer systems. Teams are considering where people can be stationed to observe vessel traffic that today is tracked by radar and similar systems. Agency executives are planning to have most operations workers on duty the weekend of Dec. 31, 1999, through Jan. 2, 2000, and they are considering how they could shelter and care for employees' families if conditions are difficult and the employees are reluctant to leave home and report for duty." Great work by the Coast Guard! Superb contingency planning. We can only hope others are taking the same approach, though evidence for such thoroughness is scant.

"You have an infinite number of variables to contend with here," Naccara says as he discusses the embedded chip problem and the way chips can interact with software. He and others in similar positions are reluctant to predict precisely what will happen at year's end. They know there will be systems failures, but no one knows to what extent those failures will cascade and have unexpected consequences."

Given that the Coast Guard seems KNOWLEDGEABLE and is testing, it appears they STILL find they have numerous potential exposures on the embedded front (they "know" there will be systems failures).

Then again, the embedded system problem has been declared "over" by Poole and Davis. CET, would you please alert the Coast Guard about this?

-- BigDog (BigDog@duffer.com), May 03, 1999.

[snip]

Operators of one cruise ship thought they had brought it into full Y2K compliance, Naccara says, but when they turned the ship's clocks forward to Jan. 1, 2000, in a test, the stateroom doors all locked automatically and stayed that way, because of an overlooked chip.

[snip]

-- Kevin (mixesmusic@worldnet.att.net), May 03, 1999.

To paraphrase what we hear often, don't expect the pollies to come rushing to this thread.....................

Kevin, would that count as a "chip" failure, ya suppose?

-- parrot (green@gold.macaw), May 03, 1999.

Another example of a "chip" failure:

http://www.greenspun.com/bboard/q-and-a-fetch-msg.tcl?msg_id=000Mce

-- Kevin (mixesmusic@worldnet.att.net), May 03, 1999.

I do remember (quite clearly, for a change) a report stating that tankers and freighters contain between 100 and 200 embedded chips.

-- Old Git (anon@spamproblems.com), May 03, 1999.

OK, Parrot, I'll volunteer as official Polly on this thread.

First, the Coast Guard has been addressing y2k issues for some time. They are well aware of how complex and difficult the problem is. This is good.

Second, they are engaged in a real, comprehensive search-and-destroy mission against these bugs. This is good.

Third, they are doing real (not just rubber-stamp) testing, and these tests are uncovering things they missed. Presumably they are fixing whatever their tests find. This is good.

Fourth, as Big Dog emphasizes, they 'know' they will have system failures. This is an excellent point and deserves some thought. How can they 'know' they'll have failures despite their efforts? Obviously, they can't. Instead, this is a policy statement. It's saying "Don't get complacent. These bugs are woven so inextricably into all we do that our search will never end."

To me, this last point is subtle and tends to escape many here. I believe this Coast Guard policy is excellent. I believe that *all* organizations should assume that there will be failures, that all the bugs cannot be found, and that remediation and testing should continue full blast at least right up to rollover. I think it's irresponsible for *anyone* to sit back, close down the project, declare compliance, and believe "We got them all." I just don't think anyone can ever say this honestly.

But this means nobody can ever declare compliance! It means that the pessimists can continue hollering about lack of declarations and late dates and bell curves. It means that the most any organization can properly say is that they have reached the point where they expect no substantive problems and are substantially (never fully) compliant. Not even comprehensive IV&V can ever get anyone past this point.

Y2K remediation is in many respects a maintenance task. When is maintenance 'complete'? By popular request, it's complete on some arbitrary 'complaince' date, or complete when public pressure to release a compliance declaration becomes strong enough, or complete when the decision gets made that further testing isn't worth the money (because no bugs have surfaced for X days).

But maintenance goes on forever, and even in theory the 'last' bug can never be found. Combine this fact of programming life with the threat of legal liability, and it means we are left pretty much in the dark. And in the dark, we all see whatever our imaginations conjure up.

I feel very confident about the Coast Guard, and I wish every organization were as diligent, honest and forthcoming.

-- Flint (flintc@mindspring.com), May 03, 1999.

"Agency executives are planning to have most operations workers on duty the weekend of Dec. 31, 1999, through Jan. 2, 2000, and they are considering how they could shelter and care for employees' families if conditions are difficult and the employees are reluctant to leave home and report for duty."

Why on earth should that be a consideration. After all, everything is assured to be alright last I heard. Why would the coast guard of all sources indicate that things might not be alright, to the degree that people would be afraid to leave home.

Fascinating isn't it.

PJ in TX

-- PJ Gaenir (fire@firedocs.com), May 04, 1999.