This critique done by Harlan Smith. Credentials at bottom. This is also good solid information for those who are unsure of what an embedded system really is.

*********************************************

* I am critiquing Mark Frautschi's report right now at his specific request and I am very displeased with it. He has gathered every urban legend in sight, scrambled them some more and given us an omelet of urban legends that is very counterproductive.

* The "embedded systems" problem is very serious but bears no resemblance to what Mark Frautschi portrays in his report. His references are in many cases poor and in many cases do not support what he says.

* He clearly does not understand "embedded systems" even though he apparently has a Ph.D. in Physics.

* Please take everything you read in there as possibly completely incorrect.

* I don't know how many times I have to say this but THERE IS NO EMBEDDED CHIP PROBLEM. There is an "embedded system" problem. "Embedded chip problem" is the jargon of the technically illiterate. I will clarify:

a. All Y2K problems are software problems. There are no Y2K hardware problems, none, zero.

b. Even PC Clock problems, where the century is not properly accounted for, are software problems as the BIOS software has not properly accounted for the fact that the RTC (Real Time Clock) does not incorporate a century counter. Almost no RTCs do, with the exception of one little used device from Dallas Semiconductor.

c. The non-compliant software that will cause Y2K grief is not designed by the semiconductor manufacturer but the "embedded systems" designer. This is true regardless of whether that software is supplied to the semiconductor manufacturer to deliver mask programmed ROMs or the "embedded systems" house programs a blank EPROM, EEPROM or other blank field programmable ROM.

d. When one looks for "embedded systems" problems, one is _not looking for "non-compliant chips". THERE IS NO SUCH THING AS A "NON-COMPLIANT CHIP". Instead one is looking for a cluster of chips that forms an "embedded system". A small computer, if you will. If that "embedded system" is found to be non-compliant, the cause will be that there is non-compliant software resident on a ROM (Read only memory). If the system can be repaired, the fix will be to find the source code, re-compile, re-program a ROM (by whatever means is consistent with the ROM technology) and repair the circuit board by replacing the ROM. This may not be possible for numerous reasons, in which case the "embedded system" may have to be replaced. (Don't get confused, the ROM could be incorporated into the micrcontroller chip.)

e. Can an "embedded system" be a single chip? Yes. In simple systems, a microprocessor based system is not used. Instead a "microcontroller" is used that can have on-chip ROM and RAM to form a complete system. However, those remediating systems important to the infrastructure, or safety critical, generally find that they are almost always dealing with more complex, microprocessor based, multi-chip systems, on the order of complexity of a PLC (Programmable Logic Controller) or greater. I will explain shortly how this cuts down the number of systems we have to be concerned about immensely.

f. Can a microprocessor-based system be placed on a single chip? Again yes. There is a technology called ASIC (Application Specific Integrated Circuit) that permits this. However, it is expensive to have such designed and we will likely find very few of these to concern ourselves with. When we are doing our detective work, we will be in general looking for multi-chip devices, as ASIC technology is relatively new and expensive and most PLCs etc. deployed to the field are multi-chip devices. However, this only impacts the numbers of "embedded systems" we must deal with as there are myriad exceptions to worry about. An expedition to find non-compliant "embedded systems" must try to comprehend the occasional oddities or they will miss a few important problems. This is one reason why the problem is so damn mind bending.

g. How many "embedded systems" do we have to worry about? I have done some homework here. Others, including, for sure, Mark Frautschi have not. Look at this:

World Semiconductor Trade Statistics (WSTS) from Integrated Circuit Engineering Corp. (ICE) Status 1998

Units (millions) 1991 1992 1993 1994 1995 1996 1997(est) Total Microcontroller 1722 1902 2221 2659 3067 3450 4167 19188 Microprocessor 136 143 167 170 212 249 286 1363 20551

-- Tom Scully (2scully@concentric.net), July 17, 1998

Answers

Here's the rest of it:

***************************** Every "embedded system" must incorporate either a rmicrocontroller or a microprocessor. But I've already said that threats to the infrastructure are likely microprocessors, not microcontrollers. That's important, when one looks at the numbers.

>From this, we can perhaps estimate that 30 or 40 billion microprocessors and microcontrollers have been deployed to the field but less than 3 billion microprocessors. So we've already reduced the potential threats to the infrastructure by a factor of **10**.

Now further, we know that microprocessors are used in computers and there are huge numbers of PCs and other computers built that will chew into that 3 billion.

We again reduce the number, by those not incorporated into date sensitive systems.

And again by those that are compliant.

Rick Cowles uses a figure of 25 million of concern. I'll go with that. If we assume that is worldwide, that is 25*10^6/40*10^9 = .000625 or .0625% of the problem that would be represented by 40 billion. Worldwide, we can probably handle 25 million.

25 million is doable and fixable. 40 billion is not.

g. So now that we've got "techie", non-compliant chips, and the numbers under a little control let's go on and respond specifically to Mark Frautschi.

>-----Original Message----- >From: Tom Scully [SMTP:2scully@concentric.net] Sent: Thursday, July 16, 1998 11:34 AM To: year2000@efn.org Subject: Embedded Systems Question

>Harlan and other techies- >Could you check out the following url and let us know if this is accurate or not. Namely this:

"Thus, even when only relative time is required by the OEMs, this may often be derived from chips that keep absolute time internally. Those chips that represent absolute time using two digit dates are subject to Year-2000 failures just as with computers and software as has been>more widely reported."

This could happen. Think about your PC. If you replace the battery for the RTC and don't reset the date, it would start ticking away from the default date. So could an "embedded system". So, it would have 20 years before it thought it reached 2000 and that would be 2018. We have 20 years to worry about that prolem. I don't care if the problem occurs in 2001. Our present concern is about what occurs in years 1999 and 2000.

>The logic "It does not NEED to keep dates, therefore it does not keep >dates." is not based on what is actually happening within the chip.

Not the chip, the "embedded system" let's just forget this ignorant "chip" jargon.

"This has resulted in a number of systems being declared Year-2000-compliant when in fact their chips have not been tested. Examples of systems containing unassessed chips include remote control load management switches installed at consumer sites by electric utilities, automobile power train transmission control modules and major household appliances."

Pretty much a garbage statement. "include remote control load management switches installed at consumer sites by electric utilities"? What does that mean? If he could talk in English, there may be date-sensitive "embedded systems" in electric utility substations or yard switching and we should worry about such. The electric utilities have been close mouthed about that, saying that problems are largely restricted to generating stations of all types.

Non-compliant "automobile power train transmission control modules and major household appliances", is an unsupported "urban legend". If I find any support for that, you will be among the first to know.

"Please see reference [4]. In the case where no date is set by an external agent, the chip defaults to its "epoch" date. This could be the design date, the date of manufacture, or some other, arbitrary, date. Non->compliant systems are subject to failure when the internal date reaches >1/1/2000, which in general will not be in step with actual time (since there >is no means, or need, to input actual time at the turn-on point). In >general, such a chip will reach 1/1/2000 internally AFTER 1/1/2000 >actually occurs."

Technically, this is possible, IF the "embedded system" has a built in lithium battery or equivalent. But why would a designer go to such lengths to design a system that way. This is really an unsupported claim.

"This is due to natural delays introduced by the production life cycle, shelf >life and possibly the duty cycle (the fraction of the time the chip is >"powered up"). For non-Year-2000-compliant architectures, these delays increase the likelihood that most of their failures will occur AFTER 1 January 2000. One manufacturer has released documentation to its customers that some of their systems will not fail until 2006. [5]"

So, possibly, there are a very few weirdly designed devices out there that will fail in out years. This is ;not a major concern. We really don't care. They will probably be lost in the noise level of random failure that we expect anyway.

http://www.tmn.com/~frautsch/y2k2.html editorializing on a long list of references (some good) is mostly a bunch of inarticulate baloney. I have been trying to read it and comment on it and am getting very frustrated about the gap with reality. Gary North of course loves it, as it supports his thesis of world calamity. Finally.

DON'T SAY "TECHIE"

DON'T SAY "EMBEDDED CHIP PROBLEM"

DON'T SAY "NON-COMPLIANT CHIPS"

DON'T SAY "A 40 BILLION CHIP PROBLEM"

DON'T SAY "WE MUST TEST 40 BILLION, OR ANY NUMBER OF CHIPS,IN THE FIELD"

-----

DO SAY "Embedded systems" are a big problem.

DO SAY "We may have to test, or otherwise investigate, as many as 25 million 'embedded systems'"

DO SAY "Investigating these is a very tedious and difficult task"

DO SAY "When a non-compliant 'embedded system' is found, it may be very difficult to repair or replace"

DO SAY "Most people completely misunderstand the 'embedded system' problem"

DO SAY "The 'embedded system' problem can only be solved by a lot of management understanding and hard work."

DO SAY "Let's clear away the fear mongering baloney and put our shoulders to the task of getting this done."

DO SAY "Mark Frautschi, you have made a very definite negative contribution to the effort. Spreading urban legends and stating that things are hopeless and we must go back to pencil and paper is asinine in the extreme."

DO SAY "This is a damn big problem and, if we don't approach it diligently and intelligently, disaster is likely to occur."

How do I follow up, "strong letter follows"? Harlan

Synergistic Mitigation & Contingency Preparation -- "Austere Infrastructure" http://2000.jbaworld.com/harlan/smcp.htm http://www.scotsystems.com/harlany2k.html (for printout) Quick Small Business Guide to Y2K http://www.angelfire.com/mn/inforest/harbiz.html Embedded Systems Remediation http://2000.jbaworld.com/embed/remediation.htm http://www.y2knews.com/harlansmith.htm YOU CAN HELP http://www.angelfire.com/mn/inforest/smpc1.html

-- Tom Scully (2scully@concentric.net), July 17, 1998.

So then, what is being done to remedy the "25 million", or so embedded systems?? What % is in the US?? Is there any way to know how the fix on these systems is coming? If 70% are in the US, that's 18.5 million.... that's 4,743 a day (embedded systems found, tested, replaced or fixed) based on a 5 day work week. I'd be surprised if half that number was being completed. The later it gets, the more we have to find and fix each day. Where is the greatest % of these systems focused? Infrastructure? I see no reason to shout for joy over your assessment. We need more info.

RA

-- R.A.Mann (ramann@hotmail.com), July 17, 1998.

Thanks for your critique of Mr. Shakespeare & Tao. This issue is a technical problem that has been embraced by many people for ideological reasons. There is little objectivity between those who have been infected with St. John's virus and those worshippers of normalcy who never believe it can happen here. It's too soon to say how bad it's going to be.....this forum has few souls who seem to be hoping for the best. These are crazy times we're living in....

-- Joseph Danison (JDanison@aol.com), July 19, 1998.

We are ALL hoping for the best, Mr. Danison. We are just not stupid enough to sit there and do nothing else. You are wrong about it being too early too tell , it is just that you are standing too close to your idol of normalcy to see what is horribly obvious.

I can see you now, full lotus postion, incense gently wafting through the temple, chanting,"itstooearlytotell, itstooearlytotell..."

-- Will Huett (Willhuett@usa.net), July 20, 1998.

This was a very credible explanation of the exageration that surrounds the embedded systems problem. Thanks Tom. Will, your remarks were predictably redundant and non-evidential - you are going to hate all us "Polyannas" even more when modern civilization does not melt down in 2000, but we will continue to love you and pray that you can some day find a little happiness.

-- Joe (Logic@sanity.com), July 20, 1998.

A quick not just to clarify that I don't necessarily agree with everything Harlan Smith says, but I believe he does have the low-down as to the technical aspects re y2k. I wanted his evaluation to be heard to help people understand embedded systems. 25 million is still a big number that will take a massive effort to fix, which presently just isn't there, and time is slipping away. This post was not meant to say we don't have a huge problem, only to bring another side of the embedded systems issue to light.

-- Tom Scully (2scully@concentric.net), July 20, 1998.

> but we will continue to love you and pray that you can some day find a little happiness. <

Oh, golly, Joe, ya *love* me? I get chills.

Just so we can burn this straw man once and for all, those of us who recognize a real threat to our society aren't unhappy misanthropes wishing Ted Kazinski was still licking stamps. Your generalizations regarding our personalities is representative of your lazy thinking. Just so you know, I am a very happy man. I make all kinds of money in a profession I love and have a wonderful wife and three loving daughters. I really LOVE things just like they are. It pisses me off that y2k had to come shit on my parade, but, such is life.

Not that you can grasp this, but the systems are broken. What took 40 years to build cannot be all fixed in 17 months. As Sen. Bennett said at the Washington Press Club last week, we can't go back 25 years, 25 years ago is gone. The infrastucture has been dismantled. He also said that if 1-1-00 were today, western civilization would collapse. Oh, but what does he know, right? You should call him and straighten the Senator out Joe, really.

We live our little lives floating on a silicon cloud and in a few short months it will

stop working...

I hesitate to add anymore, tossing pearls to swine and all, but the rest of the world is in really, really bad shape, Joe. And we can't continue on our little majical mystery tour without them. That's why remediation is futile. We will BE FORCED to pick up the pieces, regardless of what may get kinda fixed between now and then.

But hey, who am I to spoil your fantasy? And Joe, if you refuse to prepare for the disruptions that are going to occur, then you are part of the problem, not part of the solution. This isn't high school and we don't need cheerleaders. We need people that can think, Joe. We need people that have food to eat and share, and fresh water to drink and share, and seeds to plant and share, and-

Joe? Joe?

Is that incense I smell?

-- Will Huett (willhuett@usa.net), July 20, 1998.

Rocky, I am glad you have such a happy existence. Too bad that all those evil people who plotted this Y2k calamity pooh-poohed on your parade. You say we don't need cheerleaders - that speaks volumes about your attitude so I won't add to it. I say we don't need hecklers - we need people working on solutions. Personal and family preparation is definitely necessary but we don't have to stop there. What value are you adding to the community (World, country, local) problem solving efforts when you attempt with unsubstantiated rhetoric to put a negative spin on any progress that is reported? Have you ever been involved in a huge, complex project that was successsful? Was that "successful" project led by a cheerleader or a "heckler?" The serious work of solving Y2k problems is moving forward without your help Rocky but we still luv ya, man and we won't need any of your stash because we'll have our own just in case. The difference is that we on solutions side of the equation will have done more than just cover our tails - we will have reduced the extent of the damage in spite of distractions from hecklers. Now Rocky, go in peace to love and serve yourself! Y2k team you'll have to win this one for the Gipper, 'cause coach Rocky left the stadium at halftime.

-- Joe (logic@sanity.com), July 21, 1998.

Will, I am glad you have such a happy existence. Too bad that all those evil people who plotted this Y2k calamity pooh-poohed on your parade. You say we don't need cheerleaders - that speaks volumes about your attitude so I won't add to it. I say we don't need hecklers - we need people working on solutions. Personal and family preparation is definitely necessary but we don't have to stop there. What value are you adding to the community (World, country, local) problem solving efforts when you attempt with unsubstantiated rhetoric to put a negative spin on any progress that is reported? Have you ever been involved in a huge, complex project that was successsful? Was that "successful" project led by a cheerleader or a "heckler?" The serious work of solving Y2k problems is moving forward without your help Will but we still luv ya man, and we won't need any of your stash because we'll have our own just in case. The difference is that we on solutions side of the equation will have done more than just cover our tails - we will have reduced the extent of the damage in spite of distractions from hecklers. Now Will, go in peace to love and serve yourself! Y2k team you'll have to win this one for the Gipper, 'cause coach Will left the stadium at halftime.

-- Joe (logic@sanity.com), July 21, 1998.

Glad you finally got my name right, gee, I'm a sensitive guy...

Joe>What value are you adding to the community (World, country, local) problem solving efforts when you attempt with unsubstantiated rhetoric to put a negative spin on any progress that is reported? <

Thought you'd never ask. There are only two ways to add value to the (World, country, local) community, Joe. Remediating code and preparation for the inevitable failures. Since I am not a code cranker, that leaves the second.

You really should get out more, Joe. My "rhetoric" is not unsubstantiated and soporific tales from the Little Engine That Could are specifically detrimental.

-- Will Huett (willhuet@usa.net), July 21, 1998.

Will, Sorry about the name screw-up but it's really hard to tell you and Rocky apart. I think you must be a tag team. Which of you has the assignment of trashing the good news reported from FAA. Some of your friends have already weighed in with the usual gibberish: conspiracy, incompetence on the part of the testers, etc. Maybe you can come up with something more creative.

-- Joe (logic@sanity.com), July 22, 1998.

Now further, we know that microprocessors are used in computers and there are huge numbers of PCs and other computers built that will chew into that 3 billion.

Someone wrote --

>We again reduce the number, by those not incorporated into date >sensitive systems. > >And again by those that are compliant. > >Rick Cowles uses a figure of 25 million of concern. I'll go with that. If >we assume that is worldwide, that is 25*10^6/40*10^9 = .000625 or >.0625% of the problem that would be represented by 40 billion. >Worldwide, we can probably handle 25 million.

I'm confused on a point -- how do we reduce the number of problem embedded systems "by those that are compliant"? We know which ones those are, and where they are? That's hopeful. I'd thought we'd have to test most of them in order to find out which ones *are* the 25 million problem ones.

Fixing 25 million does seem doable. I'm daunted by the idea of finding them among all the good ones -- like lines of code: they don't have to fix all 3 billion lines of code at GM; they just have to look at all 3 billion to find out which are the few that need fixed.

But please, I'd be delighted to be set straight . . . .

-- L Hunter Cassells (mellylrn@nist.gov), November 03, 1998.

L Hunter,

You stuck your finger in the right place. They don't count as "problem systems", but we will still have to find them.

Even if we find them (questionable, IMO) I have a hard time believing that we can/will make them all usable in the time remaining, whether we do them at a rate of thousands per day (minimum) or all at once (impossible).

Excepting the above, everything else in Harlan Smith's post was dead on the money.

In the end, I don't think it will matter whether we drown in the Mariana Trench or in a "kiddy pool" although if it turns out to be the "kiddy pool", we can stand around with wet feet until we get things going.

How long can Western Civilization tread water?

-- Hardliner (searcher@internet.com), November 03, 1998.

>How long can Western Civilization tread water?

Not very long without life jackets. Y'all got your life jackets?

-- Tom Scully (2scully@concentric.net), November 05, 1998.

Be aware that Frautschi has altered his paper based on new evidence. I believe the new one is dated 11/2/1998.

-- Buddy Y. (DC) (buddy@bellatlantic.net), November 05, 1998.

Buddy,

Do you have a link or URL to the new version?

-- Hardliner (searcher@internet.com), November 05, 1998.

Hardliner, per your request:

http://www.tmn.com/~frautsch/y2k2.html

I think it's the same URL as the old one, but the content changed some. It's dated Nov. 3, 1998. He hasn't changed the general discussion much, but did make changes to his facts. Notably, footnote #4 was changed to:

>>http://www.gartner.com In a December 1997 report, GartnerGroup gave a world wide failure rate of between 1 and 3 percent. This was substantially updated in October 1998 where the figure was reduced to 0.01 percent: http://gartner11.gartnerweb.com/public/static/aboutgg/pressrel/testimo ny1098.html.

It should be noted that within a given sector, the incidence might be much higher. In electric utility remediation, failure rates of approximately 15 % are not uncommon (with about half of these being of the "nuisance" variety), while in automobile manufacturing and in other industrial environments higher rates of failure are being found. See http://www.tmn.com/~frautsch/cargill.txt. <<

-- Buddy Y. (DC) (buddy@bellatlantic.net), November 05, 1998.

Buddy,

Thanks for the link. I agree not much has changed. Even the reduction in number of "bad" systems to 5 million doesn't seem like it makes much difference in the end. At the 400 day mark, that would be 12,500 systems per day.

If we could be sure that all 5 million were in places where failure wouldn't have significantly bad effects, I'd feel a lot better about them.

Oh well, every little bit helps and 20 million systems is a lot, not a little. I only hope that Gartner isn't "blowing smoke". I used to pay Gartner (with my employer's money, of course) and they deliver what you need to make your sales numbers, not necessarily what is total objective reality. They perform a lot of the same functions in commerce that lobbyists do in government. In my experience, if you pay Gartner, you can trust Gartner, if your vendor's sales/marketing rep is paying Gartner, Caveat Emptor.

-- Hardliner (searcher@internet.com), November 05, 1998.

Ok, mind you I do NOT work in a technical field, but I was thinking about what chips need to be checked out first and doesn't it come down to mission critical first? Let's say you have two systems, system A and system B. Now in assesment and inventory you decided B was mission critical, but A was not. Won't you concentrate on B first? Yes A will have to be fixed someday, but not before New Years Evil. Yes there is a lot of system fixing to be done, but I think that just like code, it is being set in the perspective of priority. (and before anyone says it, no I am not trying to downplay the problem:)

Rick

-- Rick Tansun (ricktansun@hotmail.com), November 06, 1998.

Rick,

You're right, of course, about the prioritization; it has to be done that way to make any sense.

Just exactly like the other facets of the technical problem, this goes to scale. Even after we use our best efforts and techniques to find the problem (20th century code, embedded system, etc.), and then reduce the number to those that we need to address as "mission critical", we still are faced with numbers that look to overwhelm our capacity to deal with them in the time remaining.

The outlook is grim, but the alternative of doing nothing and for Mankind to go out with a whimper is not my style.

Judging by the tone and content of your posts, I suspect that it is not yours either.

-- Hardliner (searcher@internet.com), November 06, 1998.