Cross post from Rick Cowles' EUY2K forum:

[snip]

9/9/1999 came and went. No major failures happened, at least that were reported in the public domain.

Are you surprised? I'm not.

The NERC drill is being characterized as a smashing success.

Are you surprised? I'm not.

I was there.

To set the stage, a little background is in order. On behalf of a client, I was invited by a major bulk power transmission provider to observe the latest NERC sponsored Y2k drill from their main control center during the evening and early morning of 9/8 and 9/9/1999. I was given free rein of the entire facility, and wasn't shadowed by a public affairs person, beyond an initial facility tour. I was allowed to speak with anyone I wished, and overhear all communications during the course of the drill. And I was also pleased to see that one staple of all electric industry drills in which Ive participated in the past is still alive and kicking  free food for the participants and observers.

From the outset, it was quite apparent that a lot of effort had been expended in preparing for this drill. The drill procedure listed six objectives:

1. Demonstrate the ability to effectively deploy resources related to the staffing allocations for the 12/31/1999 transition
2. Demonstrate the ability to implement operating and administrative procedures, as necessary to maintain the reliability of the Bulk Electric System
3. Demonstrate the ability to effectively use backup (alternate) communications systems.
4. Demonstrate the ability to effectively deploy elements of the Y2k Contingency Response Plans
5. Demonstrate the ability to observe and document drill activities and provide an objective analysis of the overall response by the participants.
6. Train drill participants in Y2k specific activities

In reading the above objectives, it should be clear to the most casual observer that the drill was not intended to actually exercise any equipment, or perform any actual rollovers of time / date sensitive components, either embedded controls or computer system based. This is not a criticism of the drill scope, but rather, a necessary and important clarification of drill scope that has been totally absent from the impressions being provided in the media over the past two days. Alternate communication systems were tested, and some personnel were dispatched to the field from remote locations, but no equipment was actually tested or exercised during this process.

The drill began with a coordinators teleconference at 8:30PM to confirm readiness for drill activities and status of remote facilities staffing, and the drill ended with a critique around 4AM the following day. In between, there was a lot of satellite telephone communications going on from the region, and NERC performed a roll call of all participating organizations as midnight approached. Drill activities were suspended between 11:30PM and 12:30AM so that all participants were in a standby mode for the 9/9/1999 rollover.

And honestly, the whole thing was about as exciting as watching paint dry.

The region that I was observing from did not have any simulated problems thrown at them by NERC, so it was impossible to gauge response to even a single potential problem, much less multiple problems arriving at basically the same time. Communications appeared to be adequate, but satellite based telecommunications systems are somewhat flaky on a good day, and this day was no exception. Using a sat telephone is just not the same as using your cell phone. Hopefully, the regular phone lines will continue to work during the actual rollover

The personnel in the facility were very confident of their ability to work through the drill. I dont know if Id characterize the attitude as cocky, but it was apparent from the outset that they didnt consider this to be a very big deal. Most of the system operators and supervisors continued with their regular responsibilities, sandwiched around an occasional drill related phone call. To sum up in a very short sentence, they were confident not only in their abilities, but also in the outcome of the drill prior to the drill.

Heck, they were thinking about the press releases even before the drill started. The entire Public Affairs group was manned, the fax machines were preprogrammed with media contact numbers, and the Business Wire internet connection was warmed up and ready to go.

This is, in my mind, a good thing and a bad thing. It has been noted in this forum before that hubris kills  and if the drill really accomplished anything, it was to reinforce the emerging attitude that Y2k is not going to be a big deal for the electric industry. I continue to be concerned that this will lead to significant complacency on the part of the electric industry in general, and individual participants in particular, because I dont believe that this attitude is confined to simply the facility or region that I observed.

I am concerned that the drill was not a full-blown exercise. If the purpose of the drill was to challenge the operating status quo (which it wasnt), drill scenarios would introduce multiple simultaneous failures to 1) gauge how field personnel would respond and 2) test the command and control coordination of recovery efforts. For example, when training pilots in simulators, and testing nuclear plant emergency response, multiple concurrent failures are always thrown at the participants. If such a thing occurred during this drill, it wasnt apparent to me as an observer.

So, were the regional objectives of the drill (listed above) met?

• Demonstrate the ability to effectively deploy resources related to the staffing allocations for the 12/31/1999 transition

  From the standpoint of being able to pre-position personnel in key strategic locations, this objective was met. From the standpoint of being able to dispatch multiple technical teams, in order to deal with simultaneous microprocessor based failures and perform manual workarounds in multiple geographic locations, the objective was not satisfied.

• Demonstrate the ability to implement operating and administrative procedures, as necessary to maintain the reliability of the Bulk Electric System

  To the extent that this objectives 1 and 2 overlapped, this objective was met. However, it was not apparent that any written Y2k specific contingency plans were utilized, from a command and control perspective, either in the control center or remote locations. Even if the contingency plans were utilized, again, no specific operating procedures could have been exercised because no equipment was exercised or disabled during the course of the drill.

• Demonstrate the ability to effectively use backup (alternate) communications systems

  Per the discussion above, this objective was met.

• Demonstrate the ability to effectively use backup (alternate) communications systems

  Per the discussion above, this objective was met.

• Demonstrate the ability to observe and document drill activities and provide an objective analysis of the overall response by the participants

  Im not totally sure what the purpose of this objective was meant to convey, other than the ability of the organization to have observers taking notes. A consultant who has worked extensively with the transmission providers Y2k program was the designated observer in the facility  he was there and taking notes. Does this satisfy the objective? I suppose. There were also counterparts to this individual stationed in the remote locations. However, Im still not sure that this objective meant anything to begin with, from a standpoint of operationally exercising equipment, personnel and procedures.

• Train drill participants in Y2k specific activities

  A lot of training took place prior to the drill, particularly in using satellite communication equipment, and this training paid off. Communications was not a major problem. However, communications is only one aspect of this objective  theres another part that wasnt exercised, and that is the ability of personnel to adequately respond to simulated multiple (and simultaneous) system and/or equipment failures.

---

Ill close my comments on this drill by relating an interesting conversation that transpired with one of the system operators. The operator had gone through the big northeast U.S. / Canadian ice storm a few years back, when power was lost for weeks in many locations. Fortunately, the operator was only without power for three days. Still, even during the three days, life without water (an electric well pump supplies water to the operators house) was difficult at best. Imagine, the operator said to me, flushing the toilets with snow. It wasnt pretty.

The operator then described some personal preps to me, including the purchase of a generator for the well. I dont know how bad Y2k is going to be, but Id rather be ready than not. I learned my lesson.

Others before me have said it best  listen to what the folks in the trenches say, and pay even closer attention to what they are doing.

-- Rick Cowles (rcowles@csamerica.com), September 10, 1999

-- Critt Jarvis (critt@critt.com), September 10, 1999

Answers

Thanks for the cross-post, Critt...I haven't gotten a password to Rick's site yet.

Heck, they were thinking about the press releases even before the drill started. The entire Public Affairs group was manned, the fax machines were preprogrammed with media contact numbers, and the Business Wire internet connection was warmed up and ready to go.

According to this thread CNN Reports A Day Too Soon on Y2K Test, some of those PR folks sent a press release about their success before the drill took place!

-- RUOK (RUOK@yesiam.com), September 11, 1999.

Thanks Critt (and Rick)

Out of all that this is what stuck out

"Still, even during the three days, life without water (an electric well pump supplies water to the operators house) was difficult at best. Imagine, the operator said to me, flushing the toilets with snow. It wasnt pretty.

Unbelievable how ingrained habits just continue on. That is just to funny that a person would use snow to flush toilets. A person just doesn't use a toilet in such a situation. Or just drain the tank and use a plastic bag. I would use a 5 gal. pail and put a plastic bag in it. That is what northerners do. That is the problem I see, lots of "contingency plans" for business and goverment (we hope) but even the simplest things escape folk when they don't know any better at home. Just hope this "3 day storm" doesn't happen.

-- Brian (imager@home.com), September 11, 1999.