There have been numerous discussions about bits and pieces of the issue of whether power will be available com year 2000. Here is a summary of my views:

1) I am an electrical engineer working for a power company with about one million customers, doing Y2K project management and testing of generation, transmission and distribution devices. 2) I have witnessed or personally tested over 100 devices and systems used by power companies, including protective relays, remote transmitting units, transformer monitors and controllers, recloser and circuit breaker controls, and software programs used in communicating with these devices. The tests involve more than 15 dates, and include rollover, and reboot type tests. 3) Not a single device we tested so far has been found to lose its basic functionality during Y2k tests (yes, we have performed individual and system tests). The distributed control systems at power plants and the energy management system were in the process of being upgraded before Y2K issues came along, so we waited until these upgrades were made before testing. These systems also passed our tests. Except for a few one of a kind devices at select substations, we are complete with our testing efforts. 4) Our company is a member of the Electric Power Research Institute's Y2K program. Through this program, we have access to the test results of thousands of other devices. Those results are consistent with ours. I'm not saying there aren't devices needing some upgrades, but those few that do are widely known.

Based upon these facts (feel free to dispute them, but be sure to be specific about a device or system that you have proof that it fails Y2K tests for readiness), here are my opinions:

1) There is no need to make any special arrangements for Y2K as it relates to power. If you need a portable generator for general use or personal well-being, by all means get one. But don't get it for Y2K. 2) If someone tries to tell you that there will be numerous outages, ask them specifically how they know this. As an example, Robert Cook says outages aplenty will occur. I have asked him three times if he has Y2K tested any devices. He has not responded, so I assume that he has not tested anything (feel free to correct me, Robert). I recommend that you consider these things when reading posts on this or any other forum. 3) Power companies aren't scheduled to be complete with remediation and testing until June 30, so you won't see much on the web or otherwise until July. But at that time, you will see numerous utilities come forward and declare their system Y2K ready. I'm giving you this "inside information" so that you know what is coming. 4) Yes, it is true that we in the power biz have not done the best job of clearly and consistently communicating what we are up to with Y2K. But the news is GOOD, we just need to figure out the best way of communicating this. We are not hiding anything.

I hope this clears up any confusion about where I stand on this. My goal here is to inform, not to discredit nor flame others.

Dan.

-- Dan (dgman19938@aol.com), April 11, 1999

Answers

Dear Dan,

Thank you for this very important post. This is the kind of information we have be waiting for. It is consistant with what I have read by a test engineer who posted some of his observations at euy2k.com. I also know a senior engineer at a power company who has relayed similar information to me. The news is good and it is high time we got some. Thanks again.

-- Tomcat (tom@cat.com), April 11, 1999.

hi guys... curious about what your expectations are for Y2K overall, not just as it relates to utilities...

scott

-- Scott Johnson (scojo@yahoo.com), April 11, 1999.

Dan,

Thanks for your post. My question is this: what is your confidence level in distribution systems, as opposed to generation? Have you found any difference so far?

-- Drew Parkhill/CBN News (y2k@cbn.org), April 11, 1999.

Thanks, Dan. It really is very encouraging, but there are still a lot of things to worry about, not related to your tests and work.

-- dave (wootendave@hotmail.com), April 12, 1999.

Thanks Dan. I think 90% of the electric companies should be ready by 1/1/2000, but it is the other 10% I worry about. We have seen what somewhat small problems can do (Western grid failure due to tree branch, San Fran due to fail safe system failing, etc...) The question is will failures lead to a cascade effect on the grid? Also, will you have coal, gas, etc.. to power your plants? I hope you are right and the lights stay on. If the power is flowing, the rest can be delt with over time. If not, lock and load.

-- Bill (y2khippo@yahoo.com), April 12, 1999.

Well Dan, add me to the thank you list, for this post and your answers to my other questions. First-hand information is very valuable, and we all appreciate the time and effort that you have contributed here. I really hope that what you are finding is true industry wide. This is, as they say, the big one. I will be watching the news in July with great interest. I also look forward to Mr. Cook's reply here. <:)=

-- Sysman (y2kboard@yahoo.com), April 12, 1999.

Dan I would like to know how many lines of code your scada has and when did you start code remediation. How many systems rely on the gps or timing signals from the atomic clock. can your operation run with a loss of telecomunications or loss of these timing signals or supply from the grid. Have you tested your systems with a loss of these signals. If your critical embeded systems fail are your ready spares y2k compient and tested so. Are all the vendors that make the parts for all your systems still in business if not will that pose a problem. Is your company willing to put it's reputation on the line by having an indepedent outside inspector verify your y2k readiness. It can't be a pridefull thing many peoples lives are on the line if power goes down hard during the y2k. You have to test it hard to see what bugs might shake out of the system. Better to have it go down now and have the ability to fix it quick while the grid is still operational then to wait and see what shakes. Test it now online in actual operations not just on the parallel systems. You must test the systems in real time and in real field operations. If it breaks now it will be fixable and you and the industry will learn alot from it. Y2k wont be fixable unless you make the full commitment to be as through as possible. Have independent experts with no political ties verify your systems. In the military when we use to fix mission critical systems that peoples lives were depentant on we always used a system of having a second source verify we made no mistakes. This kind of commitment is needed in the power industry at this time. NO politics or pridefulness just 100% professionalism verified. Thanks Dan I know you will try to do your best. Dont let the corporate lawyers or politics stop you from doing the job you know needs to be done. Trust no vendors word for it, test and verify all critcal components. God bless and Godspeed. y2k aware mike

-- y2k aware mike (y2k aware mike @ conservation .com), April 12, 1999.

Dan, You realise you are putting the knife into the TEOTWAWKI heart. With no disruptions in power and still no evidence of the embedded chips/systems bogeyman in other critical sectors, all that is left is the possibility/probability of some failures in some places inside some unremediated business application systems. Something to lose sleep over? Only if you've invested a large amount of emotional energy and all your savings in preparing for the worst, as well as burning up a large chunk of professional and personal credibility amongst colleagues, friends and family by screaming the same.

-- Tickle (Tickle_yer_fancy@hotmail.com), April 12, 1999.

Dan, if your experience is true for the rest of the power companies, large AND small, then your news is a great relief. Electrical power is the key to overcoming any Y2K problems, of course.

Those of you who are taking heart from Dan's post--don't donate your supplies yet! Mother Nature has plenty of surprises in store. And you never know when there might be a transportation strike of some sort or problems caused by gas price hikes/shortages.

I haven't even seen the fat lady yet, never mind heard her tuning up.

-- Old Git (anon@spamproblems.com), April 12, 1999.

Thanks, Tickle troll, you moron. Appreciate the detail, Dan. Very hopeful and, you're so right, the industry needs to find a way to report progress credibly. Utilities staying up, even should there prove to be area outages and fuel shortages at times, would be wonderful for all. In our ice-storm neck of the woods, generators are wise preparation during normal times!

Do you have any sense of how the "business" remediation is going in your company (ya know, the software for payroll, billing, etc)? Obviously, utilities are so crucial the nation will find a way to keep them going even if business operations are compromised, but we haven't heard a whole lot over the past year on this score, given the concern over embeddeds ....

-- BigDog (BigDog@duffer.com), April 12, 1999.

This does indeed seem like good news for the electric situation in the United States. Can we be so confident about the rest of the world? If the lights stay on in the US and Canada, Western Europe, parts of Asia, Australia and New Zealand, but half of the rest of the world goes dark, how will this affect us?

I may be a little more comfortable knowing I'll be comfortable in January, but what will things be like on April 12. 2000?

-- pshannon (pshannon@inch.com), April 12, 1999.

Dan, It's not the power companies' technical compliancy that people are most concerned about. It's their ability to get fuel. No politician or utility manager wants to talk about that since nobody seems to know enough to even lay out a plausible set of facts indicating ready supplies from overseas.

Your advice to take no action on y2k does coincide precisely with Carolina Power & Light's latest advice which was in the April 12 News & Observer. Their spokesman said there is no need to prepare as though for a hurricane. I was surprised to see that advice. Now I've seen it twice in one hour.

I call the fuel issue the "pink elephant in the parlor" which is being blithely ignored. Your post is another manifestation of the pink elephant syndrome.

I am not predicting power outages. I am not predicting that the power will stay on. But how can you and CP&L advise people not to prepare when you have published an analysis of the easy question and left the hard question dangling perilously above our heads?

Some rhetoriticians might pass off the "pink elephant" as quibbling. There are numerous contingencies that might be rightfully called quibbling, but that argument would not persuade me with regard to fuel and I don't think it would persuade many who are thinking about this issue.

-- Puddintame (achillesg@hotmail.com), April 12, 1999.

Also, a rhetorician would tell you that I need to check my spelling before I post.

-- Puddintame (achillesg@hotmail.com), April 12, 1999.

Puddintame - you are exactly right on about the FUEL question. I believe many coal fired plants are going to stockpile at least 90 days of supply - but what then if the railroads are not working as they should. This is a prime example of WHY this issue is so scary to me: Once I start thinking about a certain problem in a certain industry, it is like dominos falling in my mind....I just can't find a stopping place logically that would "end" the problem, unless all connecting dependencies are "fixed".

-- jeanne (jeanne@hurry.now), April 12, 1999.

There is no reason to not prepare for problems even if Y2K didn't exist. I am a contractor working at a power organization as a programmer and have already witnessed software problems, many of them have been fixed (I am told). Programmers don't always rely on "basic functionality." We need good information. If hardware engineers give us poor information, our programs might make the wrong decisions. What is "basic functionality" to one may not be to another who depends on the device. I am glad to hear that the problems are not any worse than this so far. This is truly good news, and I am glad to hear quanitified information.

I am still interested in knowing more numbers. How many devices were tested at the chip level? How many microprocessors per device on average? When will the "one-of-a-kind" devices be tested? Are any of these critical to customers? How many devices were originally found to be compliant and then found to not be compliant later? Have all of the replacements been completed as of today at your company? Would anything that remains to be replaced cause a problem for customers if for some reason the replacements never came? Are you having any trouble getting devices that must be replaced?

-- Reporter (reporter_atlarge@hotmail.com), April 12, 1999.

Truly hope your assessment is correct Dan.

If the grids mostly stay up, then we can muddle through. But muddle is the operative word, especially internationally. (Other factors overlap ... terrorism, war, natural disasters, etc.)

Just a tad concerned about power industry cover-up implications though. Weve certainly seen that about the San Francisco Blackout with PG&E and the latest PUC report alleging a cover-up.

Diane

Also the information here, concerns me. ...

Power utilities prepare wide-scale drill Y2K compliance not expected on June 30 deadline

By David M. Bresnahan

The nation's power companies have Y2K problems and they admit they are purposely keeping the facts from the public and from the government. ...

http://www.worldnetdaily.com/ bluesky_bresnahan/19990408_xex_power_utilit.shtml

-- Diane J. Squire (sacredspaces@yahoo.com), April 12, 1999.

Robert Cook responds to Dan here

-- Link (it@ll.together), April 12, 1999.

From the Bresnahan article quoted by Diane, "NERC learned that many utilities would not be Y2K ready by the June 30 deadline, but that they would become Y2K ready at a later date. A decision was made to permit power companies to report that they are Y2K ready by June 30 if they list separately any exceptions to being ready."

I was raised to call that "lying."

-- BigDog (BigDog@duffer.com), April 12, 1999.

Big Dog,

If what Dan says is true, then it would seem to me that few utilities should not be Y2K ready by June 30 (as per the NERC report). Am I missing something? If there are so few problems, and the ones that do exist are in the public domain, then they should be relatively easily remedied.

Diane,

Even if the power grid functions perfectly, all that does is ease one key Y2K worry. But the business problems- lack of proper remediation and testing in SMEs globally- certain remain. And remember, all previous recessions have occurred with the infrastructure working perfectly.

-- Drew Parkhill/CBN News (y2k@cbn.org), April 12, 1999.

Dan: I have asked you this before and you did not answer. Can you tell us the name of the power company that you work for? (You can attach the standard disclaimer that your opinions are not necessarily those of your employer).

I can understand the reasoning of those posters that have negative things to say not wanting to disclose where they work (such as myself). But those that are saying "no problem" should have nothing to hide.

-- a (a@a.a), April 12, 1999.

Drew --- As I said in an early post on this thread, I do view Dan's report as good news. I was being pedantic but not altogether so about the Bresnahan article. You're either ready or not by June 30. Telling the PUBLIC in press articles on July 1 that, say, 99% are ready when, in fact, 60% are ready and 39% are awaiting shipments of 'x' and/or still need to do this little pesky 'y' thing isn't counting "ready", it's "lying".

This is not necessarily all that pedantic, either, since Rick Cowles has long stated that getting the needed upgrade parts to everyone in 1999 may itself over-stress the utilities' own supply chain.

It's bad enough that "OK" and "ready" don't necessarily translate to "compliant" but let's at least use the terms-du-jour accurately when they're used/reported.

"Oh, you didn't mean REALLY ready, just kind-a ready."

-- BigDog (BigDog@duffer.com), April 12, 1999.

Drew,

... "all previous recessions have occurred with the infrastructure working perfectly."

Yes, and the same can be said for depressions, as well.

Not only is it Y2K, but basic global economics.

BigDog,

Lying is lying, not white lying in my book. Unacceptable corporate behavior with life-threatening consequences ... especially IF wrong.

And cover-ups, are just that. And it doesnt mean covered slightly.

Exposed is probably the more appropriate term, especially IF things go wrong.

Diane

-- Diane J. Squire (sacredspaces@yahoo.com), April 12, 1999.

diane,

you're right.

big dog,

i meant in my previous post- well, literally what i said (i wasn't questioning anything you'd said before). if things are so relatively easily taken care of, then all utilities should actually *be* ready by june 30 (although i don't want to stretch what dan said, or take him out of context).

-- Drew Parkhill/CBN News (y2k@cbn.org), April 12, 1999.

Dan: A man named Bruce Beach would like very much to hear from you, if you are confident that what the utilities are doing represents an adequate test of the innards of their embedded systems. He is sincerely asking for criticism of a paper he posted at http://www.Webpal.org/Gas.htm which deals with what he calls secondary clocks hidden within many microprocessors. Since he owns patents on microprocessors in the US and Canada, I take his concerns more seriously than most. Thanks for your work and for the good news about the testable parts of our electrical infrastructure.

-- Bill Byars (billbyars@softwaresmith.com), April 12, 1999.

Well, I'm glad to see such spirited discussion taking place. I will attempt here to add my comments to your posts, but forgive me if I inadvertently skip any. I'll try to do this chronologically.

Scott Johnson and wootendave: Yes, I realize that there are many Y2K issues other than power to discuss that are very important; it's just that I have little knowledge outside of power, so I'll leave that for others to discuss.

Drew P.: I have the same level of confidence in distribution systems because even the larger companies have distribution devices they have in common with co-operatives, and those devices are testing fine as well. The one exception I noted above with a few meter types. And yes, it appears that some smaller power companies are behind the larger ones, but my impression is that they will be Y2K ready before the rollover. If you are served by a smaller utility, by all means contact them for their Y2K readiness.

Bill: You referred to the Western system problem of August, 1996. Remember that the system was highly stressed at the time before the event, and our system will be much less so in January. Also, numerous improvements have been made to the system to help prevent a re-occurrence. However, the Northeast will be in a higher loaded state, so they will have to be on the alert. The advantage here is that we all know WHEN Y2k comes, so we will be watching.

y2k aware mike: You ask numerous good questions that I cannot fully answer due to space limitations. Our scada system doesn't have hardly any date-dependent code in it, and our energy management system is being completely replaced this year, so I don't see that as a big problem. We are looking at it, though. We have no critical devices that rely on gps for their basic functionality...it only serves as a good time stamp. Regarding telecomm, yes most utilities are highly dependent on it. For our control/comm to our substations we have our own fiber optic network, so the only issue there is the controllers for that network, which has passed y2k testing. Vendors no longer being in business was an issue for a few devices, but those devices were tested and passed. Yes, we do have an outside auditor (by the way, NERC just asked all of us this question, so expect to see some results in their next result about what % of utilities have outside auditors). Yes, over 50 power plants have been tested on line, while generating electricity, with their system clocks forwarded, and passed tests fine. You want us in the power biz to avoid politics and lawyering, well, that can be quite difficult, but I've found that our legal folks have been quite good about allowing meaningful information to go out. Sure, there is still several utilities that refuse to share anything "on lawyers advice", but the fact is that some legislation has limited our liability, so there is no good reason not to share information. If your utility refuses to share anything, keep bugging them... We do not just take vendor's words for anything...we test, and verify that other utility's test results are consistent with ours.

Old git, I agree that there are innumerable other problems that could significantly impact us. I just don't think power will be one of them.

Big Dog: Good point, the business side can be just as critical. We hired dozens of outside consultants to check more than 5 million lines of code. I stick with embedded systems discussions only because that's "my bag".

Puddintame: Yes, you have hit on a good point...the fuel issue. Keep in mind that some generation types (nuclear, hydro) don't have much of a fuel supply issue, and that many plants that run on natural gas also run on fuel oil, so there is a built-in contingency. And Reporter is correct, for coal, most plants have more than a 10 day supply on hand, so there is no immediate danger even if coal plants had problems. Many plants are located at the mine site, so this is even less of an issue, and remember, there is ALWAYS the possibility of a coal mine shutting down, so plants do have contingencies for that as well. Having said that, fuel is one of our greatest concerns.

Diane: Terrorism is a mitigating factor (as is the weather) with Y2K, and it could be a big concern. Yes, I'm sure some power companies have "covered up" information, but since the news on Y2k is so good, I don't see why they have to on this issue.

Big Dog: No, NERC isn't lying! Here's an example. There is a certain company that manufactures continuous emission monitoring software used by many power companies across the country. The y2k compliant version won't be ready for installation until August, 1999. So we HAVE to report this as an exception, because it is completely out of our control to fix it before July 1. But we will still get the fixes in before 1/1/2000. Also, many base loaded power plants do not go down for planned maintenance but every 2 years, so there are some that haven't been down since September 1997, and we have to wait until September 1999 to make the final fixes. NERC just wants to get a feel for the magnitude of these exceptions.

Well, I've said plenty for now....I'll catch my wind, then respond to Robert Cook's comments.

Dan.

-- Dan (dgman19938@aol.com), April 13, 1999.

BigDog - my apologies. I declare a unilateral truce. Y2K the technical problem and the psychological drama has been/still is a horribly traumatic experience. My anger at my earlier alarmism last year has been displaced into futile 'chicken little' teasing ('moronic', I agree BD). Honest admission: I still am myself, working it through intellectually and emotionally, researching TEOTWAWKI type scenarios out of my system. I'm still only nibbling at my stockpile, where I'd rather send it to help Kosovo Albanian refugees. I am on the side of truth and open-mindedness, as I think you are BigDog. Just that I believe sometimes that means asking uncomfortable questions of ourselves and how we have/still are handling this whole thing. I think you know these questions will have to be confronted by all of us who self-reflect - I can see it in the anger of your posts.

The relatively small number of people tuned into the online discussions are still the avant garde of Y2K thought. What they/we think and conclude might matter in the general perception of things. Whether we agree or disagree, we set the terms of the debate. So far I've been disappointed. Ed, Infomagic, North seem so compelling because they have had a free run for so long, and because of the technological ignorance and/or disdain for the online rough and tumble of so many in the intellectual and political 'elite' classes.

Sorry not relevant to this thread, I know, but thought BD might ignore me if I started another post with my own handle. I'll try and keep my word - no response necessary, actions will speak louder than words. Reserve the right to personal digs a la online culture.

-- Tickle (Tickle_yer_fancy@hotmail.com), April 13, 1999.

Dan,

Thanks for your answer. And, another point, as you noted to Mike, there really shouldn't be any reason for utilities to hide behind lawyers any more.

-- Drew Parkhill/CBN News (y2k@cbn.org), April 13, 1999.

Tickle -- Spoken like a man! (or woman ...). Thanks for showing real integrity. I quite empathize with you otherwise. Humor and digs are fine but, as you know, we've got enough real trolls floating around without making it worse ourselves. I don't know HOW alarmist you were, but Y2K is still plenty alarming, IMO. I reserve my anger for the gov-biz community that haven't done what they should have, including IT (my profession) which has shown all the integrity of prostitutes over the years .....

Dan -- Have to disagree with your example and NERC. You're not ready till the part is received, installed and tested. "Almost ready" isn't "ready".

Drew -- Yes, I see what you meant and especially agree with your final point on this thread.

-- BigDog (BigDog@duffer.com), April 13, 1999.

Complacency about the power supply may be a little premature.

Most of the elecricity in this country is produced by coal-burning power plants. We have ample reserves of coal. Most coal is transported on railroad trains and river barges. Both trains and barges are diesel-powered. Diesel fuel is produced from crude oil.

About 50% of crude oil used in the U.S. is imported. Many elements in the supply chain for imported oil are potentially at risk from the Y2K rollover: well head farms, pumping stations, foreign port facilities, and shipping. Domestic port facilities, pumping stations, refineries, and storage facilities are also vulnerable. Many reports indicate that Y2K remediation in the rest of the world lags behind the U.S.

For risks to shipping, see Lloyd's Register Articles.

-- Tom Carey (tomcarey@mindspring.com), April 13, 1999.

Dan: I have asked you this before (FIFTH request) and you did not answer. Can you tell us the name of the power company that you work for? (You can attach the standard disclaimer that your opinions are not necessarily those of your employer).

I can understand the reasoning of those posters that have negative things to say not wanting to disclose where they work (such as myself). But those that are saying "no problem" should have nothing to hide.

-- a (a@a.a), April 13, 1999.

Did I miss Dan's reply to Mr. Cook? <:)=

-- Sysman (y2kboard@yahoo.com), April 14, 1999.

He dissed me to Sysman. I think the dog ate his answer.

-- a (a@a.a), April 15, 1999.