The NERC Report: You asked for it, you got it! Y2k and Power

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

A few other threads already discussed the NERC report somewhat, but I finally got to reading the entire report, and I offer my view in the spirit of those old Toyota commercials...You asked for it, you got it.

You asked, "We want to see progress...are power companies going to be ready?" The clear answer is yes. Page ii of the report shows that the average utility is 75% complete with testing and remediation, way up from the 44% reported late last year. We are really closer to 90% complete because the inventory and assessment phases took a great deal of time, and are near 100% complete on average.

You asked, "Isn't NERC hiding something by allowing utilities to report exceptions for the equipment that won't be y2k ready by June 30?" The answer is no. The exceptions are listed in their entirety in Appendix A. On page iii, "NERC has been able to confirm that the remaining items on the exceptions list...are a very small subset of overall system capabilities and DO NOT POSE OPERATIONAL RISKS TO ELECTRIC SYSTEM OPERATIONS..." [emphasis mine]. Go look at Appendix A. The most commonly cited exception is CEMS, or continuous emissions MONITORS. These are merely monitoring devices. The other two exceptions listed are DCS and EMS, but in all cases the delay is merely in being able to install the upgraded systems. You asked for more information on the exceptions; the report provides them.

You asked "Do power companies really take y2k seriously? What are you hiding with your fluff statments?" Go to page ii, Key result 3: Y2K is real. Note that the specific types of problems are connected to specific types of equipment. As an example, you read that protective relays have time stamp issues ONLY. Relays are pretty much the only devices that can trip a large power circuit breaker without human intervention, so their proper operation is key to keeping power on. ZERO, I REPEAT, ZERO protective relays have been found to inadvertently trip during y2k tests. I've personally tested about 50 relays from a dozen different manufacturers. You wanted specifics in the report; you got them.

You ask, "Why should we trust you or NERC, since NERC is a self reporting agency?" Page 9 notes that 98.6% of power companies are reporting. NERC has a list on their web site of those companies reporting, and does have the authority to notify companies that vary significantly from the rest of the industry. Trust me, they have plenty of pull on this. Compare their 98.6% response rate to any voluntarty reporting function...when we sent out several thousand surveys to our suppliers, we got a paltry 25% response rate. When we sent out second letters with a stronger message hinting at no longer buying their products, the response rate shot up dramatically. Second, NERC is working with DOE on an INDEPENDENT AUDIT, so if there are any significant deficiencies, you'll know about it soon enough. Besides, more than half of us power companies are already being audited externally (p. iv).

You asked, "What about all those co-ops?" Remember, us larger utilities use the same equipment in our distribution systems as the co-ops, and we have found ZERO distribution device failures that would interrupt the flow of electricity. See page 34 of the report, which states that nearly a third of the co-ops don't even rely on digital systems AT ALL.

You ask, "What can I do?" Page 6 lists four things customers can do. Included in that is "Power supply decisions should be based on the risk exposure of a customer on a year-round bases, rather than the anticipation of a single event, such as y2k." Y2k day is not expected to be any different than any other day, as far as power is concerned.

You asked for it, the NERC report delivered it. Rick Cowles (or anyone who wants to), let's discuss/debate the report. Also, since you have plenty to say about NERC, why don't you come down to New Orleans next week to hear first hand what NERC and other power companies are doing about Y2K? The conference info is on the NERC web site.

Rick, at the conclusion of your analysis of the last GAO report, you said to everyone "The need for you to do a critical analysis of your own local situation has never been clearer." To that I would add "And now that you have the latest information, you can analyze your situation based upon a continued safe, reliable source of electricity from your power company."

If you don't believe me, believe your power company. If you don't believe your power company, believe NERC. If you don't believe NERC, believe the manufacturers of the devices that we use in the power industry. If you don't believe any of them, well, then, there isn't really much to talk about, because it must then be some kind of cosmic universal conspiracy like in "The Matrix".

The NERC report. Accept no substitutes, adders, or fillers.

-- Dan the Power Man (dgman19938@aol.com), May 05, 1999

Answers

Sir! In respondse to your question...I spent thirty years helping to build those self same power plants you have described. And no...I am sorry! I do not; unfortunately believe them. The embeded systems that I and my compadres installed and calibrated WILL act on real time only. And you WILL NOT know which ones are non-compliant untill something goes out, stops or explodes. Such is the nature of computer chips sir. There are indeed secondary clocks in the lattice code embeded in a lot on the chips. And nothing you can do or say will cancel that revelant fact. And to try and remeadate the faulty,or suspect chips with out taking them off line is not feaseable in the time remaining. What am I doing about it personally? Why an! The only rational thing left to me...I got the H..L out of Dodge before it all shuts down. In closing, I must say that I xpect to be ridiculed or else ignored. After all! I only build those steam machines ( or rather built them).And I loved them like a gifted child. Their complexity, the Swiss clock precision that they have to be built with. There are so many things that has been ignored on this forum, perhaps as programmers you do not know these things. Or perhaps you don't care. But they are real. And they will not go away,no matter how you wish that they would. In your world, computers are the sole exhistance to you and rightly so...But that is your world. There are links from your world to mine. And mine (those computer micro processors) talk to those main frames, PeeCees that are your world. In your world if the computers go down...that is a loss of information. In my world if a embeded system goes down while under power producton. things get destroyed physically. And it takes time,man power, money and more time to replace the destroyed systems. I keep hearing Bull about a coal-fired unit going off line foe a few hours, then being put back on line. When it is standard (automatic) for the safty valve of the boiler to open and dump the steam when there is an interuption in the generation process.And even if there is nothing wrong with the power generation. It still takes the unit 24 hours to first let the boiler cool a bit, then inject more distilled water and feeding fuel to it to bring the boiler up to making steam...before you can turn the turbine again. And other little tid bits, did you know that the armature of a generator has to be kept turning at all times...You see. the spindles are turned so small in diamature that they will not support the dead weight of the armature.But will support the turning weight. If the armature stops even for a minute, the spindles will bow, and there goes your armature..Call up Westinghouse or G.E. and tell them ou are sending a 5 mil or so check for a new one. And lots of luck in your efforts. The last time I was aware that a utility needed a new armature it was on a five year back order. Ladies and Gentlemen, the embeded systems are going to be the real killers of y2k.And you cannot re-program them, you can only find and replace them. And if you think that I sound scared...YOU ARE RIGHT. The Dutchman

-- Dutchman (Electric@Shock.owee), May 05, 1999.

Dutchman,

I'll let Dan take the rest of it. To satisfy my own curiosity, describe the date-sensitive problem that an embedded system could have from a "secondary clock" embedded in the "lattice" of the processor. Better yet, give a specific example from one of the machines that you installed while building power plants: make and model.

-- Stephen M. Poole, CET (smpoole7@bellsouth.net), May 05, 1999.


At the risk of endangering my soon to be applied for DB (tm) card, I have to side with Steven on this one. BUT, I would also tend to ask Dan to speak to the interfaces that the Dutchman mentions. After, of course, the Dutchman gives us something we can approach Dan with.

Dutchman, Help us out here, and give us specific pieces of equipment so we can continue the discussion. It's not that we distrust either of you, it's just that now, we need real specifics, from both of you.

I can't make even an educated guess, nor can I determine actions based on calculated risks, in a "I said BLAKC" "I said WHITE" situation. I need to see specific equipment discussed, and educated estimates of the prevalence of use of the equipment being discussed.

Chuck

-- chuck, a Night Driver (rienzoo@en.com), May 05, 1999.


Hey Dutchman,

Please answer. I would like to know too!

-- Moore Dinty moore (not@thistime.com), May 05, 1999.


You just GOTTA love it when threads like this are completely ignored by the regulars....

A HAH HAH HA ha ha ha hah .....

-- Mutha Nachu (---@yulelogfire.org), May 05, 1999.



Dutch,

Your post appears as a bald attempt to stir up panic until you reply to Dan's question. As for me, I am with Dan 100%. I do not test DCS in F&H plants, but I know the folks that did. They were thorough, professional, competent, AND they answer tough questions put to them. I have personally overseen the "embedded devices" known as protective relays in the F&H plants. There is not a single generator or transformer protective relay that will cause a unit trip due to Y2K. So the question remains in your court - what devices and specific applications did you install that will cause the mythical unit rejection your invented to scare the unknowing?

-- cl (cl@sky.com), May 05, 1999.


cl-

I readily admit your post is helpful and informative. And I agree we need a response from Dutch....

BUT, even realizing all of this your attitude and smartbutt remarks always makes me WANT to side with the other guys. And anytime the facts are even on both sides I'll always line up with the others. it's a psychological thing with me. You make me want to dislike you and hope you are wrong. Do you do this on purpose??? LOL!

-- LindaO (whysorude@hotmail.com), May 05, 1999.


Dutchman,

You got a lot of people waiting for an answer. Don't wait too long to post it or you're gonna look like jus'another "scaremonger" and of course discredit others who might have valid information of the same ilk or kind.

-- Mevin HB (cahartwe@umich.edu), May 05, 1999.


Linda,

My response to Dutch is a reflection of the irresponsible nature of the post. Stirring up irrational fear can make Y2K become a self- fulfilling prophesy with the trigger not being embedded chips, but irresponsible people like Dutch.

Do you realize that if every automobile in California buys ONE gal of gasoline more than they normally purchase - that will be enough to drain the refineries of that state. (this was put forth by a oil company rep at a conference - I cannot verify its accuracy). Remember the Johnny Carson toilet paper incident???

As for your emotional response to my posts, sorry I try to deal in facts not feelings. This is the result of engineers trying to relate an engineering problem an solution to non-enginers. It is two different world views and communications norms. Kinda like the difference in communications between men and women. (Engineers are from Mars...) Dogbert is really quite accurate in his observations regarding the social communications skills of engineers.

-- cl (cl@sky.com), May 05, 1999.


While waiting for additional information, could someone please tell me if this is correct.

And other little tid bits, did you know that the armature of a generator has to be kept turning at all times...You see. the spindles are turned so small in diamature that they will not support the dead weight of the armature.But will support the turning weight. If the armature stops even for a minute, the spindles will bow, and there goes your armature..

-- Mike Lang (webflier@erols.com), May 05, 1999.



Diamature?? Are you going Dutch on this one too?

And one supposes that all the propellures of your plains will similarly spin away if stopped??

Keep planting them, Big Ed.

It will look great in 2001

-- Flying Dutchman (Flying.Dutch@Powerco.com), May 05, 1999.


Hey, Muthatroll: Chuck, Mike and I are regulars, no?

Nothing would make all of us happier than to see the entire grid humming along next year. It wouldn't mean (duh) that Y2K was a hoax but it would mean the utilities came through responsibly. Awesome. I AM more optimistic about the grid than I was twelve months ago, though I won't declare Y2K a victory on utilities or anything else until we pass through the date barrier and spend several months there. Heck, I'd consider the utility side a victory if the grid stayed up and there WERE local outages. I'm not really that picky.

As a layperson on this, I hope NERC is being given honest data (ain't saying they're not, just saying I hope they are) as well as competent data, that the Dutchman's of the world prove to be kooks and, among other things, that Russia, China, Japan, Germany, France and a few other places have power too.

-- BigDog (BigDog@duffer.com), May 05, 1999.


Dan, Now I'm curious. What's the deal with those armatures? On the one hand, it would be hard to make up something like that. On the other hand, how would you ever manufacture and ship and install an armature? I guess you could get Charlie Chaplin to wind it while its spinning.

-- Puddintame (achillesg@hotmail.com), May 05, 1999.

The spindle argument is not so hard to accept, particularly since i happened to have bent the drive shaft on my ceiling fan by laying the box on its side for a couple weeks (well, maybe a month or 2). Took 10 years of continuouis running to get the D&*N thing to spin straight!

Chuck Who is still waiting for something specific.

Maybe I'll dial somone on the csy2k board and ask for a card this week. Always assuming they actually printed them. ;-}

-- Chuck , d n d (not@wo.rk), May 05, 1999.


Hey, Muthatroll: Quiet, or you'll blow my cover.

I'm just waiting for Mr. Cowles or Mr. Cook to reply, you know, other guys in the biz, before I put my $.02 in. Now, quiet Mutha... <:)=

-- Sysman (y2kboard@yahoo.com), May 05, 1999.



BigDog,

Several utilities have already rolled plants and entire T&D systems into 2000 and are operating with them there.

-- cl (cl@sky.com), May 05, 1999.


Is it too soon to conclude that Dutch is full of $#!t?

Regards,

-- Mr Deedah (used2B@unkeeD.B4civility), May 05, 1999.


What stands out for me about NERC is their definition of compliance for utilities. If a utility tests ten systems and nine of them need no corrections, NERC will allow them to report that they are 90% complete with their remediation though they have not yet begun to correct the one system. Utilities must be questioned in detail as to what is not yet complete.

-- John littmann (JTL9700@JUNO.COM), May 05, 1999.

Problems on the Rails

But the U.S. railroad system, oil and gas pipelines and segments of the power grid remain a top concern.

We are deeply concerned about the railroads," Koskinen says. "We have no indication that they are going to make it."

The rails maintain their own telecommunications system, which provides computerized operation for vital systems, suc as switches and warning signals, Ksokinen says.

If railroads are forced to slowdown so automated functions can be carried out manually *by whom? I wonder* and to provide a greater margin of safety, then a broad section of the economy, from heavy industry to fresh foods, could be affected.

http://www.usatoday.com/life/cyber/tech/cte068.htm

While a discussion of embedded systems and spindles bowing is interesting, lets remember about all the EXTERNAL suppliers the utilities require to continue operations. It takes energy to make energy. If the power generating plants can't get timely (JIT) delivery of coal..............then? Or replacement parts, these are complex machines, things wear out. If lead times on critical components becomes really bad post Y2K, how RELIABLE do you think electricity will? While the grid hopefully stays up, keeping it RELIABLE by todays standards might be difficult for some time post Y2K.

Remember, connect the dots.

MarktheFart

-- MarktheFart (quke@ix.netcom.com), May 06, 1999.


Glad to see some discussion on this issue. Sorry, folks, I'm not an expert on power plants themselves, but other guys on our y2k team are, and they have never talked about issues in this manner. I'm not sure I understand the Dutchman at all (what is diamature? diameter?).

This I do know about how this relates to y2k. As cl alluded to above, over 100 power plants have had their system clocks forwarded into 2000 with no problems. Several plants have left their clocks in 2000 and will leave them there until next year. Yes, it is true that the generator and turbine are a giant mass of metal with great momentum that could cause damage if not shut down properly. But remember, unplanned unit trips happen somewhere every month, and the existing safety systems make sure damage is minimized. Again, I don't understand what the Dutchman is talking about.

Mr. Littman, the 90% reporting you talk about is not how I've been reporting to NERC. We don't count progress on systems that have no date awareness. If we did, we would have been 99.9% complete from the get-go. If we have 10 systems with date issues, we must test and remediate 9 of those systems before we can say we are 90% complete.

-- Dan the Power Man (dgman19938@aol.com), May 06, 1999.


For more backup to Dan, here are NERC's explicit instructions on the subject of the percentages:

http://www.nerc.com/~y 2k/assessment.html

7.A standard method for determining per cent complete follows. It is preferred that your responses be geared to % work done compared to total amount of work to be done. This would account for various activities having different amounts of effort. For example: if your inventory shows 100 devices with possible Y2K problems, and your assessment shows that only 2 have Y2K problems and one device has been replaced with a Y2K ready device and the other still needs remediation, the per cent complete to report would be Inventory 100%, Assessment 100%, Remediation and Testing 50%.

Second, as for Rails, maybe check here. Won't repost the whole thing.

-- Hoffmeister (hoff_meister@my-dejanews.com), May 06, 1999.


is this a subjective issue,that we can talk away?or is it objective,and indifferent to our partisan bickering?

-- zoobie (zoobiezoob@yahoo.com), May 06, 1999.

Mark, thanks for the URL on the Koskinen quote on railroads. I hadn't seen that before. Nothing new, but it's coming from the horses mouth so to speak.

If you believe all the happy news from government and industry, there's nothing to worry about. If you believe all the bad news from official sources, then things will be very bad. Unfortunately, there's a lot of reason for people to lie about the good news; there no reason for any official to exaggerate the bad news. So it's not unreasonable to decide that the bad news by Koskinen, Bennett, Greenspan, Scott McNealy and others is credible.

-- Puddintame (achillesg@hotmail.com), May 06, 1999.


Just my $0.02 also...I attended the Embedded Systems Conference in Chicago this past March. The presenter on Y2K and Embedded Systems had done a lot of assessment and remediation with power companies on real and potential Y2K problems. My evaluation from this guy's professional credentials and his handling of the topic is that he was not blowing smoke.

His bottom line confirmed what we have heard in other venues -- Y2K vulnerabilities in power generation and distribution equipment are less than was feared. GOOD!!!! On the basis of this and other information I have personally downgraded my assessment of the possible/probable outcome of Y2K from a 8-9 to a 7 or so.

But note that we're still facing likely local power outages of unknown duration. Note too that local outages can and have caused larger outages. So the folks who state flatly that "the power won't go off" are out of line. Why the double standard when prognosticators of doom are castigated as irresponsible but prophets of "no problemo" aren't called on their unqualified absolutes, hmmmm?

Other countries, including some highly industrialized nations, are likely to face much worse power problems than us. The likelihood of civil unrest and rippling economic disruption from that is relatively high, IMO.

So I'm still looking for a "one year of disruptions, ten years of depression" scenario (no apologies to those naysayers who might conclude from this that I'm a Yourdon groupie. Bottom line: he makes the cogent case -- you don't). That's quite a bit more optimistic than I was six months ago.

IMO, the utility to watch now is water reclamation. How do you get emergency water to 30 million Americans? Yikes!

-- David Palm (djpalm64@yahoo.com), May 07, 1999.


Due to previous problems with the doomers my company has asked me to contribute less to these forums and to use a different name, which is what I am doing. However, sometimes you read something so full of crap you have to respond. I have 17 years generating plant experience as a mechanical engineer. When dutchman says you have to keep a generator rotating or you'll break the armature he is either lying just to stir up trouble or is a very low level technician who doesn't understand what he works on. The truth is that any large rotating piece of equipment, such as a generator rotor, rotates on and is supported by its bearings. Generators are routinely stopped for maintenance and we do not bend the armature. It is true that if the rotor rested on the armature it would bend and destroy it, but that doesn't happen. The rotor rests on its journal bearings. Even when the bearings fail the rotor does not collapse, it just rests on the ruined bearing. Dutchman is full of crap, of course this obvious since he hasn't defended his original statement. As usual, though, these doomers who claim to have expertise but a glimmer of truth in their statements so they will sound like they know what they are talking about. Again, those of us in the industry expose this. Of course there is that 15% of you who won't believe us and will believe people like dutchman, but that's okay, I'm responding to the 85% of those who listen and respond to reason. By the way, these numbers come from an EPRI poll conducted in March/April and just released at our EPRI Y2K conference in San Antonio. EPRI is the Electric Power Research Institute and is the vehicle the electric industry is using to accomplish a coordinated embedded systems effort for Y2K. I have participated, spoke at, and help lead this effort since its inception in September of 1998. The point is the electric industry will be ready. My company, where I am in charge of embedded systems and contingency planning, will be ready July 1, just like NERC is reporting. I file that report also so I know that the information presented by NERC is real and honest. Again, as to the glimmer of truth in this doomer, it is true that a rotor will bend when it stops rotating if sits for several hours, this is due to gravity and is fixed by rotating the rotor for several hours prior to returning it to service, again, a known issue that has no bearing to Y2K. Hope this helps everyone. thanks...murph

-- murph (murphjen@aol.com), May 08, 1999.

I should proof read before posting, I made a mistake in the date we started the EPRI effort, it was September 1997 NOT 1998. If you were worried that we didn't start till late 1998, we didn't, we started in late 1997. Also, I apologize for the few grammatical errors I made. thanks...murph

-- murph (murphjen@aol.com), May 10, 1999.

Moderation questions? read the FAQ