Electrical Glitches and How They Happen

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

A question was asked today on this forum if gas regulators have embedded chips, and what the possibilities of power glitches are.

1. Gas regulators do not have embedded chips. They work off a spring in conjunction with gas pressure. As the appliance burns the gas, the pressure drops downstream of the regulator and the upstream pressure causes the diaphragm to depress which opens the valve seat allowing more gas to flow in. This is all controlled by the spring inside the regulator. There could be embedded chips in the compressors or the control panels that regulate the pressure inside the transmission lines which could be subject to Y2K problems. However, I have never seen a regulator that had embedded chips.

2. Earlier today, I spoke with a person who works in the electric industry. The information passed to me is as follows: Transmission and distribution electrical systems have what they call automatic reclosers that sense or feel a fault in the system that may be caused by a wire falling down or a tree limb falling on the wire, or by rodents. The recloser opens and closes within seconds to see if the fault has corrected itself, such as burning in the clear. After the third time the recloser stays open a human being has to find the problem and repair it. This could be one of the reasons why glitches may occur. I am sure there could be more reasons for glitches to occur. Voltages are controlled by substations and that's controlled by engineering.

-- Rasty (Rasty@Bulldogg.com), September 03, 1999

Answers

* * * 19990903 Friday

Rasty:

Two electrical utility engineers--software types!--attended one of my Y2K public awareness presentations.

Paraphrasing what they told me, as they tried to describe it in "non- geek speak":

There are 10's-of-thousands of ( the little ) distribution switching "substations"(?)--"one-of-a-kind" embedded configurations dependent upon the intended load(!) distribution point--in the "field," feed back their _status_ data to other ( adjacent little proximate units and ) then, to the major ( larger collection ) up line "substations."

These "little" units contain "safety" and "preventive maintenance" software that "self-checks" for predetermined, against recorded intervals of ( actual ) maintenance performed--as the field technician records ( inputs ) this "action."

If maintenance has not been done in the "specified" elapsed period-- by comparing DATEs ( i.e., pseudocode here:

IF PM-LAST-DATE [ e.g., 990903 ] > CURRENT-DATE [ e.g., 000101 ] THEN notify_surrounding_units_that_this_unit_to_shut_down_for_PM shut_down_until_PM_is_completed .

In other words, if the PM hasn't been done in the specified elapsed time, these configurations will automatically shut themselves down. Further, the units need to be physically replaced with compliant configurations. Something that the vendors--if they're still in business--cannot produce before 01/01/00.

According to these gentlemen, this is the state-of-the-art today that protects switching hardware/components in the field from "toasting" themselves during normal wear and tear performance. The specifications are based on "Mean Time Between Failure" ( MTBF ) spec recommendations from the hardware configuration component vendors.

Each of these "defective" field configurations

( I hope the above was clear! I parsed and re-read it a couple of times. It's probably the best I can do based on my recollections. )

Confirmation?

Rasty, if your "electric industry" contact knows--or knows someone who is an electric utility "embedded" software "geek"--could this scenario be confirmed?

For months I've been trying to confirm this ( very plausible ) "smart" distribution scenario to no avail.

I've e-mailed:

Rick Cowles - can't confirm Harlan Smith - didn't know Roleigh Martin - didn't know Jim Lord - didn't know Leon Kappleman - didn't know Peter de Jager - didn't know ... ad nausea...

The fact that 2 electric utility software programmers attended _my_ presentation, betrayed volumes about _their_ personal "concerns" about the integrity of the post-Y2K power grid. ( BTW: They expressed high praise for _my_ understanding and dissemination--in layman terms--re the big picture of potential Y2K events and ramifications.

Regards and seeking clarification, Bob Mangus

* * *

-- Robert Mangus (rmangus1@yahoo.com), September 03, 1999.


Please give verifiable detail.

I don't believe that anyone would deliberately program a device to disconnect all power to tens of thousands of consumers, simply because a service had been overlooked. That's akin to programming a jet engine to shut down if it's overdue for service, even if the plane is at 30,000ft at the time!

No: you just flag a warning and continue, because anything else could kill someone (in both cases). If someone persistently ignores the warning and the equipment eventually breaks down, you (the equipment manufacturer) did all you could, and the lawyers will sue the operator whose fault it is.

-- Nigel Arnot (nra@maxwell.ph.kcl.ac.uk), September 03, 1999.


* * * 19990903 Friday

Nigel Arnot wrote:

"I don't believe that anyone would deliberately program a device to disconnect all power to tens of thousands of consumers, simply because a service had been overlooked."

Nigel, you are amusing! ;-)

Substitute:

"I don't believe that anyone would deliberately ... write programs and operating systems using only two digits to represent year data spanning more than 2 centuries."

"They" did!

See how silly it sounds?

Y2K is a classical case of UNINTENDED CONSEQUENCES! You know, where the results turn out to be really bad.

Our illustrious federal government rejected a 4-digit year format standard in the mid-1960's; didn't change this _standard_ until 1998! And some would promote the idea that this same lot of "Keystone" bureaucrats establish a national computer software coding standard. ( ROFL! )

Y2K is caused by "deliberate" human actions causing "unintended consequences!!!"

Get it?!?

The only malice entailed in the entire fiasco, is the malfeasance by those in management that _chose_ to ignore real warnings by the thechies of the disasterous consequences if the problem fixes were not started a LONG TIME AGO!!

Regards, Bob Mangus

* * *

-- Robert Mangus (rmangus1@yahoo.com), September 03, 1999.


Robert, Working on Y2K in the electric industry, I have never seen such devices programmed as you describe - indeed, digital switching and protective relaying y2k bugs that I have seen are minor. I would like a manufacturer/model number for verification...can you contact those guys and get this? If such devices exists, lets get this out now.

Thanks,

-- FactFinder (FactFinder@bzn.com), September 03, 1999.


PEPCO Services in Washington, DC, performed an audit of all the electric distribution panels in "my" office building, searching for and identifying processor based circuit protectors.

I have not seen the report, do not anticipate any of my stuff to be affected except for the main switchgear.

However, according to the information I read on the contract agreement, there are processor based circuit protectors that have y2k problems.

If you want to learn more contact them.

I do accept the fact they are in use, and the scope of the problem goes beyond the issue of maintenance history logs.

My interpretation of the operation of these devices is they encorporate a time referenced algorithm to monitor stability and in event of anomoly prestage the circuit protector to trip rather than have it fail on temperature which is a destructive mechanical process.

-- Tom Beckner (tbeckner@xout.erols.com), September 03, 1999.



"My interpretation of the operation of these devices is they encorporate a time referenced algorithm to monitor stability and in event of anomoly prestage the circuit protector to trip rather than have it fail on temperature which is a destructive mechanical process."

-- Tom Beckner (tbeckner@xout.erols.com), September 03, 1999.

Tom, do you have an example of what anomaly might cause the prestage, and by prestage, do you mean the protector will actually trip, or is an alarm sent for a lineman to investigate, and one last question, what are those ~2 feet long black insulation-looking wraps that often seen halfway between poles on the lines?

Thanks in advance.

-- lisa (lisa@work.now), September 03, 1999.


I am a Serviceman/trouble shooter for a major utility on the west coast. I will tell you this, for service reliablity requirements, there are no relays installed and designed to disconnect the power due to no maintenance on that relay, switch or componant. This utility has a terrible record of routine maintenance. This system would be tripping out all the time if they were set to trip off the line due to a lack of maintenance.

-- R.J.R. (Y2KEYES@open.com), September 03, 1999.

Lisa:

The vendor is:

PEPCO Services 2000 K Street, N.W. Washington D.C. 20006

202-739-0800 fax 202-739-0801

The description was included in the pitch for having the audit performed, and it was not kept in our file. I have our audit, we have no problem I can see in our distribution panels or switchgear.

From memory, the processor monitors line condition, out of range variable (load, volts) prestages the circuit protector and then if the condition exists for x amount of time the breaker trips.

This is my recollection and conjecture and should be verified.

I must point out that just because we have been told that this type of protector is used on our end of the power supply does not mean that the utility uses them on their end.

I do know, however, they exist and they are used based on the vendor's material that was presented to us.

Re the two foot long black insulator, I don't know. If the question were multiple choice for a million dollars I would guess D) Cable Splice.

Regards,

-- Tom Beckner (tbeckner@xout.erols.com), September 03, 1999.


hmmm....."From memory, the processor monitors line condition, out of range variable (load, volts) prestages the circuit protector and then if the condition exists for x amount of time the breaker trips."

This sorta implies there must be a clock function, because one would assume that condition "x" must = true within a specific time frame. That is, five faults over six months wouldn't warrant a trip, but 5 in 45 minutes would.

Thanks for the homework assignment, sir!

Possibly this is what Tava/Beck was trying to I V&V when the questions regarding T&D surfaced a few months ago. Next question, does this processor interface with DCS or SCADA? What's the driver?

-- lisa (lisa@work.now), September 03, 1999.


Thanks for the Pepco info: they bought MET, which performs Y2K compliance audits for utilities. link

"By having MET perform a Year 2000 Compliance Audit on your electrical sytem, you can demonstrate that your electrical systems should not interrupt your business upon reaching the new millenium. You will also be able to document that you have taken prudent steps and exercised "due diligence" to expose and correct potential Y2K problems."

Off I go. Maybe they'd be willing to name what components could suffer rollover problems.

-- lisa (lisa@thanks.sir), September 03, 1999.



Boy oh boy, is there ever a bunch of misinformation posted under this question.

The information that Robert Mangus posted is absolutely wrong. What FactFinder and RJR said is correct. There is no equipment (especially switches) out there that would automatically take itself out of service if the maintenance hasnt been done by a certain date.

Actually your (Mangus) understanding of what a recloser is or does is also faulty but thats neither here nor there.

-- The Engineer (The Engineer@tech.com), September 03, 1999.


For that matter, Engineer is one of our most arrogant trolls, but I suppose that is neither here nor there, either.

-- Had it (with@this.troll), September 03, 1999.

For Robert Magnus: re-read your post and mine. Especially:

In other words, if the PM hasn't been done in the specified elapsed time, these configurations will automatically shut themselves down.

There's a world of difference between a normal Y2K bug, viz. something failing because of unanticipated consequences of a decision to use YY rather than YYYY, and something being designed to shut down if it calculates that its next service is overdue. That the calculation of when next service is due could be screwed up by a Y2K bug I don't doubt; that the deliberately designed consequence would be a shutdown I doubt very much, and other posts here confirm that I was right to doubt.

Enough.

-- Nigel Arnot (nra@maxwell.ph.kcl.ac.uk), September 03, 1999.


A question for those in the utility industries, either gas or electric.

Is it reasonable to assume that any failures related to monitoring/safety equipment, would lead to those systems being bypassed/removed, in order to restore the grid or gas service, as a band-aid measure?

-- Bokonon (bok0non@my-Deja.com), September 03, 1999.


Lisa:

You are getting it wrong.

We're talking about circuit breakers like the ones in your house except they are for heavier loads and they are processor enhanced to filter out nusiance trips and fast enough to protect themselves from thermal failure.

They would have a time reference for decision making.

That raises the question of whether or not the time reference is date capable despite the fact it does not have to be date dependent.

They could be monitored for status as part of a control system, but they would not be an input for the control system.

We are talking about stand alone devices.

I don't work on utility stuff and can't be much help, but I remember the PEPCO Services information was eye opening for me.

-- Tom Beckner (tbeckner@xout.erols.com), September 03, 1999.



Thanks again, Tom. I mulled it over on the errand (lunch) break and the date-dependency thing keeps coming back to logging, it seems. We're talking about circuit breakers like the ones in your house except they are for heavier loads and they are processor enhanced to filter out nusiance trips and fast enough to protect themselves from thermal failure. They would have a time reference for decision making. I guess they'd need a date for logging purposes? That raises the question of whether or not the time reference is date capable despite the fact it does not have to be date dependent. If it was...assuming 5 minute intervals, would the code look something like this? Check for anomalies.
While anomaly exists=true
log condition/time, increment array + 5 minutes,wait 5 min.
If total-time => 20 minutes
Trip breaker
Else clear time-array
Endif
Endwhile
Or something along those lines (I'm sure I messed up the indentation). If we're talking about a standalone device that doesn't poll scada or dcs for time (plus date), then the "bios" clock for this machine could be easily determined/replaced? Just curious, Tom, what month/year did you become aware of this equipment? Also, how many would you estimate are in service across the country? (Thanks very much, again.)

-- lisa (lisa@work.now), September 03, 1999.

Don't you just love it when some know-it-all states that someone else's perception is wrong, but refuses to correct the misconception? Hey, Engineer, for the benefit of those interested laymen here, what's the problem with correcting Mr. Mangus' explanation of a recloser? IMHO, his description was "close enough" for the non-tech folks. Pointing out such trivial technical inaccuracies hardly seems worthy of mention by someone with your big brain.

Bokonon,

Most of the posts of this thread address the question you've asked. Generally, monitoring equipment can be bypassed if it fails. Safety equipment, on the other hand, should never be bypassed. However, in some cases, the line between monitoring and safety equipment is not so clear-cut.

-- Elbow Grease (LBO Grise@aol.com), September 03, 1999.


I'd just like to know the specific reasons why we might lose power for up to but no longer than 72 hours, and why de Jager is still hedging on power. I live out in the country, in the Texas hills.... my NERC Y2K guy kinda recommended alternative heating/cooking gear and now I'm curious.

-- lisa (lisa@dolt.here), September 03, 1999.

Gee Had It is that just because I know what Im talking about?

OK, I'll try and explain all of this.

First it depends on what KV you are talking about.

What Rusty was talking about in his original post is:

For Main and sub grid (lets say 750 KV to 115 KV) the recloser is a separate device ( in the substation, not in the breaker) that closes the breaker after it trips due to a fault. Most of these are still electromechanical devices. Some of the newer microprocessor based relays have the reclose function built into them. Sometimes they are set up to give only one or two recloses. Sometimes they are set so that they will reclose for a single line to ground fault but not a multi-phase fault. It depends on each company's operating procedures. There is no industry wide standard. The reason for this has to do with probability. You have a high probability (over 90%) for a successful reclose for the first fault but it gets less each time the reclose is unsuccessful. As for single line VS multi- phase faults the reason is that its more of a hit to the system if the fault is multi phase and you dont want to hit it too many times. Wear and tear on transformers and generators.

These reclosers dont take into account dates, times, etc. All they do is time from when the breaker opens and then issues a close signal. Sometimes the circuit is just a simple RC timing circuit.

For lower (distribution KV): The breakers and relays (including the recloser) are one package. They have been around for quite a while and predate computers and chips by a few decades. Some of the newer ones do use microprocessors, of course. These are usually set up to do multiple recloses, sometimes 4 or more. Again it varies from company to company.

Some of these devices are in substation yards and some are pole mounted. None of these things worry about when the last maintenance interval was, etc. I think some of the confusion here is because of the newer devices measure how much amperage they interrupted. Some companies are trying to go from a regular time interval of maintenance to doing it on the devices that have been used the most or interrupted the faults with the most amperage. Thats what tends to cause the most wear and tear. However none of these shut themselves down. That just doesnt happen. I dont know where Mr. Mangus and company are getting their information but that is not the way it works..anywhere. You could imagine the problems for a company (regardless of Y2k) if some cold winter it automatically shut down power to a neighborhood. Or even in the middle of a summer heat wave. The systems are not built or designed to operate that way.

Then there is the recloser that Tom and Lisa seem to be talking about. These are not reclosers like above but rather a more sophisticated type of circuit breaker like you have in your house. The breakers in your house look at current and then trip when enough current produces enough heat. The new ones are able to look at frequency and a few other things and therefore trip faster. There isnt really a clock function one that keeps track of date and time except for possibly logging purposes. Normally a company wouldn't have SCADA poleing devices at this low a level.

Does that answer everyones questions about reclosers?

-- The Engineer (The Engineer@tech.com), September 03, 1999.


Thanks, Engineer, and no, it probably doesn't, but can you imagine what failures might cut power for up to 72 hours? Thanks in advance....

-- lisa (lisa@work.now), September 03, 1999.

Lisa,

Outside of physical damage such as caused by a major storm the answer is no. If you look at whats caused outages that long in the past its always been ice storms or hurricanes, etc. And thats not for everyone. The way utilites work is they try and repair the damage that will bring the most people back on line first. So if you are in the boonies and you are out of power, even if its a simple and easy repair, youll probably be the last one brought back on line. Places with more people will be brought back on first.

The 72 hours is just CYA.

-- The Engineer (The Engineer@tech.com), September 03, 1999.


Then why are so many cities (mine included) spending millions on generators? Are we expecting a worse-than-average winter? Also, check this out, from ERCOT: link

Preparing for the year 2000 is no small task considering ERCOT has over 35,000 miles of transmission line, 300 generating units and a peak demand of approximately 53,600 MW. The illustration in Figure 3 above is a very simplistic view of an electrical system. Transmission lines are interconnected throughout the state of Texas among the ERCOT member utilities. At each point of interconnection is a substation or switching station. The concerns for the millennium bug are in microprocessor based devices at these substations. Protective Devices (Relays) determine if power flows are normal along a transmission line or through the substation and will send a signal to interrupt the flow in the event of an abnormal circumstance such as a short circuit. This is why relays are considered as mission critical systems in the electricity delivery business. Other components being checked include voltage regulation devices, battery chargers and communication devices that report the operating status to the utility company.

Power generation plants are typically located in a member utilitys service area near large load centers. Since the NRC (Nuclear Regulatory Commission) is facilitating Y2K readiness with the nuclear power plants, ERCOT is focusing on the fossil fuels plants. Power plants generate onto the transmission system for transportation to the point of electrical load. Generation of electricity has several stages of operation that are critical to deliver energy. ERCOT member companies have projects in place to inventory, assess and remediate components or systems in each stage of generation in an effort to eliminate any Y2K bugs. These projects are on schedule to be complete by the NERC target date of June 30th, 1999. ERCOT expects to have sufficient "Y2K Ready Generation" available for all Y2K sensitive dates.

Finally, the delivery of energy to homes and businesses contains some mission critical devices as well. Similar to the transmission substations, distribution substations have microprocessor based systems. These are generally devices that would not cause power outage events on the transmission systems, however the inconvenience to our customers is still of the same magnitude. Utility companies are identifying, testing and fixing problems with relays, battery chargers, voltage regulators and communication devices to provide as seamless of a transition into the year 2000 as possible.

-- lisa (lisa@work.now), September 03, 1999.


Lisa,

As for the generators, I dont know. Better be safe than sorry? Some relative of a politician owns a generator supply house? I dont mean to be flippant but it does make sense for places like hospitals, police stations, etc to have their own separate power supply, Y2K or no Y2K. As I said above about living in the boonies. I dont have a generator, I live in the city. My friend has one because when they get an ice storm where he is they can be out of power for two, or three days, and once up to a week. Im not going to go out and buy one and hes not buying a bigger or newer one.

The problem is I dont know when that quote from ERCOT was written. A few years ago (about 97 or so) I became concerned with Y2K in the microprocessor based relays. Most relays are still electromechanical by the way. But the more I looked into it the less problems I found. Actually most things didnt have any problems and the few that do have nothing to do with the control of the power system. The dates in the relays dont do anything except give the date.

I could see someone initial being concerned (I was) and writing that but its so out of date as to not be even relevant anymore. I suspect its still up there as a CYA general statement. Ditto with the part about distribution. The only problems we found so far are with logging devices. And even then in a lot of cases you get the correct information, just the wrong date and time.

The problem is we live in such a litigious society that you really cant come out and say there wont be a problem .

-- The Engineer (The Engineer@tech.com), September 03, 1999.


My apologies to Bob Mangus, and to a lesser extent to The Engineer. Mr Mangus never used the word recloser. The recloser description was by Rasty. If The Engineer was truly referring to Mr. Mangus' post, I must admit that I am unaware of any processor-based reclosers.

lisa,

The "two-foot long insulator" sounds like a *telephone* cable splice to me.

-- Elbow Grease (LBO Grise@aol.com), September 03, 1999.


Engineer, the ERCOT data was from May 2, 1999 and the advice was given to me a few weeks ago.

You know, you really make it sound as if the whole Y2K thing cf electricity was a big scam or hoax or something.

-- lisa (lisa@go.home), September 03, 1999.


Well maybe close to it.

The problem is that so many people think they know how it works and they dont. I use to post on another list with my real name and email address and I got some questions asking me: Is this true? Does it really work like this? kind of thing. When I would write back saying not only was it not true and no it didnt work like that I would get replies saying I was lying and I should come clean and tell the truth and let the people know how much trouble we would really have after Y2K. I eventually dropped off the list.

One of the problems is a lot of IT people (and they are almost the worst) think that because they understand computers and networks they understand everything because everything is computerized. No they dont and no it isnt.

One common myth is that everything is on SCADA and everything was computerized back in the 50s and 60s. All the people who use to know how it could be done manually were fired or laid off or died. Its just so bogus. A lot of people who post on lists like this think (I mean they really, really believe) they understand how the grid works, etc. and they dont have a clue. But they believe what they believe.

As for what your NERC Y2K guy recommend. Im not sure what you mean. Does he work for NERC (or ERCOT) and how is he their Y2K guy? Look at it this way. If you buy the stuff (Sterno? A camping stove?) and dont need it, will you be mad at him? Probably not. Its not that big of an expense. So hes covered. And lets say there is a problem. Maybe its Y2K and maybe its not. One problem I do worry about is some nut case going out and causing damage to try and panic people. I believe someone was arrested in Canada a week or so ago for just that. You get to use your stuff and feel good that youre a GI.

-- The Engineer (The Engineer@tech.com), September 03, 1999.


Lisa wrote:

"They would have a time reference for decision making. I guess they'd need a date for logging purposes?"

No, forget logging or history. Stand alone circuit protectors as in "honey, we just blew a fuse."

Logging, history, graphs, optimum start, phase monitoring, all that would be the SCADA control system in the way I have this pictured.

We are talking about processor enhanced stand alone mechanical safety devices, if they are in fact even being used.

I am real close to seeing it the way Engineer lays it out.

The deal is, thirty gagzillion miles of distribution, what is really out there?

-- Tom Beckner (tbeckner@xout.erols.com), September 03, 1999.


There seems to be a lot of reasoning of the form, "Well, yeah, they COULD have designed a gizmo to do this if that happened, but surely nobody actually WOULD have." I thought that part of the potential problem was the use of "general purpose" embedded chips, which might have an internal clock that operates, independent of the application for which it is used -- and that the internal clock may have a problem with Y2K. Thus, a large part of the embedded chip issue is checking for this, not assuming that if the function does not appear to be Y2K sensitive then the chip is Y2K compliant.

-- King of Spain (madrid@aol.com), September 03, 1999.

Mr. Engineer, you are right. I dont need to tell you this but the others that view this page should. I will be the guy to go out and switch your power back on if it goes out. Should a piece of equipment fail as a co incidence of Y2K or by an owl in the night, I will be able to by-pass the equipment and switch the lines back on. Now if a drunk smackes a pole, ice tears the cross arm off of the pole, some of you will be with out power for a short time, untill repairs are made. I've been in the power electrical business since 1973. Iam not the only one who remembers how to switch a line with out SCADA. The newer switching equipment that engineer talked about on the lower distrobution lines is what is called a DPU, or distribution protection unit. It logs all hit's on the line. Currently all of our DPU's are set for September of 2000. Our DPU clocks were moved ahead in March of 1999. When a bump or hit or outage on the system monitered by a DPU occurres, the date would show todays day, time, and the year 2000. Lisa, stop looking at this as a conspiracy. We need to worry about the fruit cakes and tin foil heads out there that want to see SHTF on Y2K.

-- R.J.R. (Y2KEYES@open.com), September 04, 1999.

rjr: you the man! thanks for being there when the stuff hits the fan ON those miserable stormy nights.

-- RASTY (RASTY@BULLDOG.COM), September 10, 1999.

Moderation questions? read the FAQ