Serious Y2K Risks In Embedded Systems - Report

April 23, 1999: 1:16 p.m. ET

LONDON, ENGLAND (NB) -- By Steve Gold, Newsbytes. The green light on Year 2000 issues relating to embedded systems may have been given too early by many experts, according to a report just published by BSC Consulting. The London-based IT consultancy firm says that it carried out its survey as part of its ongoing business continuity and risk management service. The survey revealed that the problem of embedded chips could have a serious impact on vital services and buildings, says Patrick Moore, the director of the firm. BSC says that its survey was in-depth, and covered a total of 38,361 systems involving telecommunications, medical equipment, air conditioning, lifts, heating, lighting and other utilities. According to Moore, the dangers of the Y2K issue on IT systems generally have been well documented and, as a result, many organizations are spending millions of pounds to try and resolve the problem in time for the next century. Concerns have also been raised about the potential risks to building systems and equipment, brought about by embedded chips and logic processors that could be affected by the Y2K problem. In its report, BSC refers to these items as assets. Its report found that 6.8 percent of these assets are not Y2K-compliant. While this percentage may seem low, the difficulty is that these systems may be the ones that are going to be vital in terms of running a building, according to Moore. "It's very important that companies and organizations take steps to ensure that their embedded systems are millennium compliant, otherwise they could be faced with the prospect of not being able to operate their business at the beginning of 2000 because vital machinery or building systems are not operational," he said.

home | digitaljam | contents | search | stock quotes | help

Copyright ) 1999 CNN America, Inc. ALL RIGHTS RESERVED.

comments?

-- (arnie@query.com), April 24, 1999

Answers

From the Presidents Council on Year 2000 Conversion which of course all of you have read :o) is this interesting bit. http://www.y2k.gov/new/FINAL3.htm

PHARMACEUTICALS

(Lead Agency -- Department of Veterans' Affairs)

 

7 A February 1999 survey of 24 drug manufacturers representing 90 percent of the industry found that they are 67 percent complete with remediation of critical systems and 50 percent complete with testing.

7 Drug companies report that 90 percent of embedded systems are Y2K compliant, while 10 percent require adjustments/replacement.

-- Brian (imager@ampsc.com), April 24, 1999.

Now why would this high a precentage be reported and there is supposed to be no reason for this problem? Now why would anyone want to think there is going to be a problem if 90% of the generic meds are imported? Would John K. be an alarmist or is there really a problem.

-- Brian (imager@ampsc.com), April 24, 1999.

So basically, this BSC tested 31,368 embedded systems, and found a failure rate of 6.8 percent. Doesn't sound too bad, but lets figure exactly how many failed:

.068 multiplied by 31,368=2,133 system failures

Now, lets dive into this a little more carefully. Basically, anyone can take this whole article at face value. Of 31,368 embedded systems were tested. Of those, 2,133 failed. That is so small of a number that honestly any person can't sit here and fret over a number so miniscual.

Gartner released a survery and Y2K problems will be spread well over a year, and more likey 23 to 36 months. With only 8% of all problems occuring at the stroke of 12AM on January 1st, 6.8% not even really worth worrying about.

Also, these embedded systems, you have to ask several questions, what kind? What type? What problems were detected? Did the problems shut down all responsibilities of the microchip, or did it still function?

Pat "Don't trust the gov't. Trust me. I can sell you family safety for only the low price of $10,000."

-- Pat (BAMECW@aol.com), April 24, 1999.

So a Risk Management Consulting Company finds risks to manage? Will wonders never cease. Such objectivity!

-- Flint (flintc@mindspring.com), April 24, 1999.

In terms of the health care industry they have to have accurate dates even if it is not critical to the basic operation of the equipment cause if there is a death or otherwise and the lawyers get involved then the hospital has a problem. The cost of legally fixing the problem is going to amount to more than fixing the equipment. Not to mention of course the patient but people don't count as it "will never effect them". The health industry is finding 2 - 5 percent failure in hospital equipment because of date problems. Fix on failure or work arounds are not an option in such a case.

If the rate of failure is as high as 10% are you really going to say that the happy faces in Clintons command are just throwing numbers out for the sake of the consultants to make money? Or could the problem be a bit bigger than we would hope?

Back to the chemical industry report

>>>>>>>>>>>>>>>>>>>>>>>>> SMEs managing high hazard chemicals can pose large risks to works and the surrounding community. While some exceptional SMEs are highly resourced, more generally, SMEs lack awareness regarding the Y2K impact, resources, and the technical know-how for fixing the problems. Given the time constraints, there is very little chance of changing that reality. >>>>>>>>>>>>>>>>>>>>>>>>>

Come July maybe we will hear if these plants are going to shut down during the rollover. Then we will see how big a problem we could be having. Pardon me if I kind of try to figure out this situation but after reading about embedded systems for 14 months and hundreds of opinions you figure there would be some kind of agrement if there is even a problem. Now I realize that failure in these cases are often not critical to the equipment involved but if there is potential for failure why do people think it won't be serious in all cases. Why bother even looking at the problem. It would seem to be a waste of money. They can give it to me instead :o)

-- Brian (imager@ampsc.com), April 24, 1999.

Flint..In answer to your comment about embedded chips not causing a problem. In most hospitals, they use eiher Hewlett-Packard or Ention monitoring systems for their CCU/ICU/Open Heart units. AS OF 3/20/99, the 'older systems' of HP (more than 4 years old) were not compliant and did not have a fix for either the programming or the chips in the sending units. This per HP website. They might have this under control but I for one do not wish to have chances taken with me or mine. The monitoring system is extremely date/time sensitive. The ventilator systems for non-breathing patients (Bird and one or two others) are compliant---if they are the old systme. The new ventilators are very sensitive. Spoke to the head Resp. Therapist at one of our large local hospitals Thursday. They tried to roll the date on (no brand please) machine. Locked up tight and refused to reset.

Last week at one of our MAJOR hospitals (800+ beds) they went to test the emergency power (not a y2k as far as I know) under complete shutdown....guess what? The new Heart Floor emergency plugs were not included in the circuit. Had a bunch of RNs and RT bagging people for almost 30 minutes until power could be restored. I understand it took them almost 20 minutes to get down 5 floors and find the maintence people to tell them. Phones out all over the hospitals etc. One hell of a mess. No one died but plenty of administrative staff shaking in their boots.

What's my point? It only takes one embedded chip not functioning to kill you...if you happen to be on that machine.

Stay healthy.

-- Lobo (atthelair@yahoo.com), April 25, 1999.

Lobo:

I expect quite a bit of this, to be honest. Saying embedded problems are rarer than expected is quite different from saying they don't exist. There will be fatal failures due to y2k, and fatal failures due to Murphy as well. Telling these apart will probably require a lot of analysis. Those who automatically assume that any mechanical failure they hear about must be y2k related are muddying the waters at best.

-- Flint (flintc@mindspring.com), April 25, 1999.

Flint, the number of failures I see in medical systems every day even without y2k scares the bejeesus out of me. They can't keep the systems intact NOW. When you figure in an additional 1 - 3% because of the date change (even if the equipment is insignificant) I don't think the maintenance people can keep up. I know that when significant machines go down, the other 'stuff' gets put on the back burner. Then that machine breaks because of lack of care, etc. Then two machines, ad infinitum.

Like I said, stay healthy.

-- Lobo (athelair@yahoo.com), April 25, 1999.

Pat,

>Of those, 2,133 failed. That is so small of a number that honestly any person can't sit here and fret over a number so miniscual.

Think of it like this; if your car is 99% in working order but the 1% that fails is your dead battery, your car ain't goin' nowhere. If it's your A/C you might be uncomfortable, but you can still drive.

Where the problems occur is much more important than the overall failure rate.

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), April 25, 1999.

Flint,

So a Risk Management Consulting Company finds risks to manage? Will wonders never cease. Such objectivity!

I'm beginning to like you.

You have been warned. :)

-- Stephen M. Poole, CET (smpoole7@bellsouth.net), April 25, 1999.

TECH32,

There's the rub. These "non-compliant" (in quotes because the definition seems to change at need nowdays) embedded systems could range from hand-held test equipment to entire complex process controllers. These surveys tend to lump them all together. How to tell?

You should be aware that most of us who install and use these things don't really trust them, and tend to leave the old (manually-operated) stuff in place as a backup. So ... even if my shiny new Mark III Widgerwockie goes south on 01/01/00, I've probably got a backup that can be switched in rather quickly. Worst case, I'll have something in place to smoothly and safely shut the process down until I can get there to scope it out.

You know why? Because these things fail now, all the time, and usually at the worst possible time. Have you ever seen what a bad power supply can do to an embedded processor? Not to mention operator stupidity ...

("Duh ... hey, Stephen? Whaddazit mean when the little meter pegs in the red an' th' thing starts smokin'?")

That's why I've never believed the fantasy scenarios about embedded widgets going insane and dumping 10,000 times the lethal level of flourine into a water supply. Does anyone honestly believe that these systems don't fail NOW, and that, as a result, there aren't safety mechanisms in place?

("Garsh, we just sent some highly-corrosive stuff into Aunt Maude's commode, hilk, hilk! Bet she'll getta kick outta that!")?

These numbers are meaningless as far as predicting disaster. I would suggest you look at Flint's angle -- this is very possibly a consultant (or group of same) ensuring job security for their humble selves. :)

http://www.wwjd.net/smpoole

-- Stephen M. Poole, CET (smpoole7@bellsouth.net), April 25, 1999.

Flint (or should that be "Flip" as in "flipant")said

"So a Risk Management Consulting Company finds risks to manage? Will wonders never cease. Such objectivity!"

If there do happen to be "risks to manage", who do you expect to find them, the janitors? It may be the case that these consultants are exaggerating the problem, it may be the case that most of these problems are merely cosmetic, it may be the case that I am a brain in a vat, and all you other "people" are just figments of my programmers' imaginations. But, you know, sometimes you've just got to go with whatever information you can get, even if that information is not perfect.

A 6.8% failure rate in embedded systems worldwide, even assuming that most failures would be minor, results in one almighty Jupiter-sized turd hitting the fan over the next year. The only range for reasonable debate would be from small-t tEOTWAWKI, to capital-T TEOTWAWKI. I hope these consultants are full of it, but from my limited vantage point I so far see no reason to suspect this.

And on the topic of the 6.8% figure, Pat said "That is so small of a number that honestly any person can't sit here and fret over a number so miniscual." Pat, I've got an answer for you, and it's a one word answer..."EXTRAPOLATION". I'm worried that a 5 sylable word might be a bit much for you, but give it some thought.

-- humptydumpty (no.6@thevillage.com), April 25, 1999.

Tech32 wrote: >Think of it like this; if your car is 99% in working order but the 1% that fails is your dead battery, your car ain't goin' nowhere. If it's your A/C you might be uncomfortable, but you can still drive.

Where the problems occur is much more important than the overall failure rate.>

But that is the whole point I made. The number is small. In the grand scheme of things, 6.8% of all embedded systems having some sort of failing, and remember that the "stroke of midnight" is not the witching hour of doom. The failure rate can be almost non existant and fixable by technicians as problems are going to be spread out over 30 months.

The car batter analogy works well here, you care battery goes dead, that is just 1% of a problem. So jump start the car, everyone should have a pair of jumper cables for any reason, jump the car and drive off. It is fixable.

-- Pat (BAMECW@aol.com), April 25, 1999.

Humpty:

Just turn your observation around and see how it sounds. If bankers say banks are OK, just who would you expect to know this better, the janitor?

I'm not saying this outfit fabricated their results at all. I'm sure they found noncompliances. But I also notice two things about this blurb that raises red flags for me:

1) There isn't any good definition of the scope of a system. Most embedded systems (NOT your VCR) are layered into larger systems, which are in turn layered into larger systems. By selecting the largest scope of 'system' you can maximize your 'rate' of noncompliance. Did they do this? They don't say.

2) There isn't any summary of the nature of the noncompliances. What percentage of those noncompliances would cause significant loss of operational functionality? Any? They don't say.

I was at a Harley fest once, 10,000 motorcycles and one rep from a tire company (Dunlop, I think). He went around examining everyone's tires, and found over 80% of them were 'noncompliant' - incorrect pressure, too much wear, suspicious wear patterns, etc. Yet every motorcycle worked perfectly adequately, and there were NO flat tires experienced by anyone (and the average attendee had ridden over 100 miles to get there). But he WAS selling tires. And his 'noncompliances' were real.

Think of that tire rep as a risk management consultant. If you're selling something, you don't fabricate the need for your product or service, but you sure as hell do everything you can to *emphasize* the need for it. And if your description makes the risks look much larger than they actually are, you done good (provided no particular thing you say is inarguably false).

-- Flint (flintc@mindspring.com), April 25, 1999.

Yes, points 1 and 2 that you mention occurred to me too. I tried to find out more at their web-page

http://www.bscconsulting.com/index.html

but it appears that their server is down. Maybe it won't be soon.

Your tyre analogy is good. I'm sure this describes the situation to some degree, in that many of the non-compliances are trivial, but technically they are y2k-glitches. But this has always been a known factor in the percentages of the problem. Let's guesstimate that 60% of y2k chip problems are trivial, 20% are annoying, 10% are trouble, and 10% are killer. (Choose your own numbers if you wish.) Plug these numbers into a base rate 0f 6.8% failures and you have a gynormous problem, especially compared to the "only 0.1% of chips have any problem" numbers that have done the rounds at times over the last year. A survey of 38,361 systems from various industries is not a bad survey sample, IMHO. Oh, and another thing. Flint, if we were discussing any old very complex issue in science or history or whatever, then I would commend your scepticism and reserve. But seeing as how this is such a high-stakes issue, where we have to make important decisions soon if not earlier, I sometimes find your approach to be exasperating.

-- humptydumpty (no.6@thevillage.com), April 25, 1999.

Humpty:

Your point is well taken. I don't mean to exasperate you. I understand that when you're faced with a clear and present danger, you shoot first and ask questions later, because if you don't, there might not be a later for you.

When later comes, though, you must satisfy the legal system that the danger was indeed clear and present. In our system, we have a standard of "the ordinary reasonable and prudent man", and what that man might reasonably have done under the circumstances. This standard is unfortunately applied using hindsight, by people with all the time they need to think about what you *might* have done differently.

I see the same situation with y2k. There is clearly considerable debate as to the danger, and some very well-informed and influential people are saying the danger is neither clear nor present. Your position is that it is "reasonable and prudent" to assume that those people are wrong, and that any information supporting their position (and there's a lot of it) is also wrong, and that those whose business it is to sell you protection against dangers they themselves have identified are being straight with you.

Since the stakes are not at all clear and obvious, you recommend preparing against *assumed* stakes. I think there's some limit beyond which prudence blends into paranoia, but I don't know where that line is, and I don't think you do either. I believe in preparing against much worse than I expect, but that shouldn't force my expectations to match my preparations. I should still be able to evaluate the danger, rather than just assume it.

-- Flint (flintc@mindspring.com), April 25, 1999.

The high-stakes aren't just an assumption if these 6.8% people are correct. (Therefore I should assume they're correct because the stakes are high?? - that would be circular.) Researching all the other aspects of y2k tells me the stakes are very high, and the probabilities of rosy outcomes quite low.

I think it is prudent to err on the side of caution when assessing evidence. In normal, inconsequential intellectual debates erring on the side of caution involves witholding judgement until the weight of evidence forces it. In this intellectual debate, the considerable potential for personal harm (and you know of what I speak, this potential is not just an assumption!) plus the limited time available for meaningful decisions, changes my operating definition of "erring on the side of caution." In y2k debate, erring on the side of caution ought to mean paying special heed to each and every plausible warning of potential harm to you and those you're responsible for. Certainly don't close your ears to explanations as to why those problems might have been overstated, but basically, epistemology needs to take a backseat to ...something or other..I dunno, I'm tired... Is it just instinct? This is a bias, but I think it's justified. Be nice to be proven wrong in the end. I'm sure Flint could critise what I've tried imperfectly to say, and maybe when it's not 4 a.m. i could clarify in response to him. It kinda comes down to this...some emb.systems reports say it's no big deal, some indicate, (though they don't spell it out) that it's TEOTWAWKI time. What are YOU gonna do?

-- humptydumpty (no.6@thevillage.com), April 25, 1999.

Humptydumpty,

First of all, if you'll look at my Web site, you'll see that I don't tell people not to prepare. In fact, I recommend that they keep extra on hand all the time anyway -- forget Y2K -- because life has a tendency to throw curve balls when you least expect it.

(My wife went blind in February from a condition I'd never heard of before and still don't understand. Since I lived in the coastal plains of NC most of my life, I've also been without power more times than I can count. I know all about curves. [g])

I have been called a Polly because I won't tell people to prepare for, say, several months (or even years). Why? Because the fact is, if it's _that_ bad, having extra noodles and wheat in the closet won't make that much difference.

And anyone who thinks that the shiny H&K or Remington in the closet will pull them through a complete collapse of society is living in a Hollywood fantasy, complete with Tina Turner in studded leather.

If it does become clear that Y2K is going to be bad -- and I would remind you that the Doom Prophets ain't exactly batting 1.000 in the prediction department thus far -- what I'll do is simple: I'll ask what I can do to help. I'll pitch in. I'll do my part to see that Y2K passes as smoothly as possible.

The solution is NEVER to panic and run and hide. It's to face the problem squarely and fix it -- as a society, not just on the individual basis.

-- Stephen M. Poole, CET (smpoole7@bellsouth.net), April 25, 1999.

Stephen if there is a total collapse of the civilisation, which some (not me) expect, then presumeably there will be SOME survivors, barring serious nuclear f**k-ups. Someone with lots of food and weaponry and a country home stands a much better fighting chance than someone with no preparations, although there's obviously no guarantees.

Although I disagree, I sort of know what you are getting at by saying that prepartions wouldn't make any difference. I was talking about y2k to a guy I met at a festival over Easter. He was really blase about the problem, but his preparation plan was to come to the new years festival, and have six months of food stashed a few km's away in the bush. While commending his preparation plans, I tried to get him to understand just how fubar the situation would be if he actually needed all of that food, because most everyone else would have nowhere near that level of prep's, and things would get ugly pretty quickly. I think it went over his head.

You said "The solution is NEVER to panic and run and hide. It's to face the problem squarely and fix it -- as a society, not just on the individual basis." If one were convinced that the civilisation is going down, then abandoning ship makes the most sense. The most effective altruistic thing to do would be to convince other individuals to do likewise, there'd be no hope in trying to save everyone. Even if we're really facing a 7 and not a 10, the emphasis placed on personal preparations is not misplaced. Every little bit will help. But emphasising the need for personal preparations does not preclude people acting to help others, by informing them of the problem, and trying to generate pre-emptive community planning and action.

-- humptydumpty (no.6@thevillage.com), April 26, 1999.