Why we get it and embedded chips...?

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

Im looking for some credible discussion about the Embedded chips problems we've all been discussing.

I think it's safe to say that Embedded chips are a problem. My question is how many chips or what percentage of them can fail and still have continuously functioning critical systems.

It's a question of whats the margin of error.

I'm anxiously awaiting your comments.

Father

-- Thomas G. Hale (hale.t@att.net), April 14, 1999

Answers

Father ... Have you read the post/story by Beach?? Ran under "The Embedded Processor SECONDARY Clock Problem" on April 9th. URL is http//www.webpal.org/Gas.htm . Get out the rosery beads , Father, BEFORE you read this one !! And say a prayer for all of us . Eagle

-- Hal Walker (e999eagle@freewwweb.com), April 14, 1999.

Thomas,

I don't think anyone knows what the fault tolerance is when it comes to embedded systems, and you would think it varies from system to system. Sometimes just one error in a system can cause big problems, like the non-Y2K date problem at a New Zealand aluminum smelting plant on December 31, 1996. The system didn't know 1996 was a leap year, was confused on December 31 when the 366th day arrived, and major damage happened as a result.

A good all-around article on embeddeds is:

http://www.jsonline.com/bym/tech/0214chips.asp

Problems lurk in more than just computers

By Douglas Armstrong

of the Journal Sentinel staff

February 14, 1999

Embedded chips are the wild cards of Y2K.

Only a tiny percentage of them are expected to fail when the calendar rolls over into the next century after 11:59:59 Dec. 31. But there are literally tens of billions of these dedicated processors out there in everything from microwave ovens to airliner cockpit controls (a Boeing 777 has 1,000).

Some, obviously, perform critical duties. And, according to many experts, there isn't time to check them all and tell which are bad and which are not by the time the new millennium ticks ominously in.

One reason is that the programming in embedded chips is not always readily accessible for inspection. And there are hundreds of different varieties. It's like looking for burned out light bulbs in Las Vegas -- with the power switch turned off.

"Most of the failures will be nuisance issues," says Bill Thompson, senior analyst with Automation Research Corp., a consulting firm in Dedham, Mass.

Not everyone is so sanguine.

"The embedded systems problem is still a black hole," says Harlan Smith, a Y2K analyst who moderates an online forum on the issue at y2knews.com.

"Identifying the devices that are not compliant and assessing the effect of them on the environment in which they operate is complicated."

Corporations spent a lot of time and money bug-checking the front office software code on their mainframe computers for Y2K compliance before realizing an even bigger problem existed on the plant floor in automation controls and other systems running on embedded chips.

A massive catch-up effort is under way, at least in the United States. How big is the job? Experts can only estimate.

Tava Technologies, a Colorado software and consulting firm that specializes in assessment and repair of plant Y2K problems, says that in its experience at more than 400 sites, it has "yet to find a single site that did not require some degree of remediation (repairs)."

At a pharmaceutical firm with operations in 39 countries, for example, Tava found 4,457 embedded processors in the laboratory equipment and manufacturing facilities of one location.

Based on an inventory it conducted, 18% of the items were not Y2K compliant and 17% could cause a plant shutdown or affect production.

"The chance of these systems failing was 70% for the lab and 80% for manufacturing and facilities," says Bill Heerman of Tava's Denver office.

Tava estimated that it would take 39 weeks to inventory and analyze the firm's 125 plants at a cost of $11.5 million. The fix would take another 31 weeks and cost $54.8 million.

Is there time to fix it all?

"There is little reasonable prospect of timely correction of all Y2K exposures that exist," says a report from Manufacturers Alliance/MAPI Inc. "The effort to achieve compliance is one of damage mitigation."

The effects of a maverick embedded processor are unpredictable. It depends on where it exists in the chain and what is connected to it. Typically, these chips gather a lot of information to make limited decisions.

If a single temperature sensor tied to an embedded chip in a complex chain of measuring instruments used in manufacturing were to go haywire because of a Y2K problem, for example, the manufacturer could end up with a product with different ingredients -- if the product came out at all.

The stakes involved in locating and repairing these chips are huge, given the dependence of our systems on them. The size of the chore is every bit as large, given the proliferation of embedded chips in number and design.

"They are everywhere," says Steve Barnicki, an associate professor of electrical engineering and computer science at Milwaukee School of Engineering.

Why?

"They are cheaper and more trouble-free than mechanical systems," says Barnicki. As a result, they have played a pivotal role in powering productivity improvements everywhere since first introduced in the 1970s.

Fortunately, many (like the one in your portable CD player) couldn't care less about dates.

"There are embedded systems that don't have the faintest idea what year it is," Barnicki says.

So why not hunt down those that compute dates and fool them by turning back the year to play it safe, you ask?

The answer lies in the sheer number of chips and the independent way many have been programmed. These processors also work in tandem with chips and systems that would experience their own set of problems if a false date turned up.

The issue is made more difficult by ubiquitous quirks, such as chips that have the ability to disguise that they have date capabilities and escape detection until they fail. Or those that can have a delayed reaction.

"We encountered a controller on a process line recently that rolled over to Jan. 1, 2000, just fine," says Kurt Schmidt of Tava Technologies' Denver office.

"And it kept working just fine until it went to Jan. 32, then Jan. 33, Jan. 34 and so on all the way up to Jan. 54. Some of these systems won't show the date problems immediately."

Embedded chips come in a number of varieties from a host of manufacturers.

On the low end are ROM (read only memory) chips that contain basic instructions that cannot be changed. If these have a Y2K problem, they cannot be saved. The machine they are attached to may have to go as well, if a compatible substitute chip cannot be found.

Next are PROM (programmable read only memory) chips, which typically can be reprogrammed only once, according to Barnicki.

EPROM chips (erasable programmable) can be reprogrammed thousands of times after they are exposed to ultraviolet light. Finally, EEPROM (electrically erasable programmable) chips and similar Flash ROM chips have the potential to be reprogrammed tens of thousands of times.

Rockwell Automation, based in Milwaukee, is a leading maker of programmable logic controllers (which use embedded chips) to run factory automation configurations. The brand name is Allen-Bradley.

The company lists 17 different known year 2000 issues with its controllers on its Y2K Web site.

In addition, it outlines a procedure to test its controllers for other potential problem dates, such as Feb. 29, 2000 (leap year), Jan. 10, 2000 (1/10/2000 -- first seven character date) and Sept. 9, 1999 (the "9999" date field matches an end-of-data "9999" input signal in some computer programming codes).

Rockwell/Allen-Bradley's programmable logic controller issues are a microcosm of the complexity of the problem. They have:

Processors that won't roll over on their own and must manually be set to 2000.

Processors that roll over to a new century only if the power is on at the time of century change. (Jan. 1, 2000, falls on a Saturday in a holiday weekend when many plants would ordinarily be dark.)

Processors that won't roll over without new software or bug fixes.

Processors that are dependent on the compliance of the system they are connected to.

Processors that are totally dependent on systems that are not prepared for 2000 at all, such as 286 and 386 computers.

Many programmable logic controllers don't have clocks.

"You don't put a date in there unless you need it because it wastes power," Barnicki says. "Embedded processors are stripped down to fit the application."

Although a vast database of embedded chip compliance has been assembled by Tava Technologies and others, manufacturer assessments of the chips can only help so much.

"They can test all they want," says Automation Research Corp.'s Thompson, "but it's really up to the end user with the local application to test out the system. (The processor) might work in a vacuum.

"Once it's installed with custom add-ons and special report functions that have been locally written, there is no way for suppliers to help the users predict what will happen."

Says Tava's Schmidt, "There are going to be hiccups."

And some hiccups may occur in places that cause more than just a nuisance or harm to a negligent company.

The American Chemical Society has warned that chips automating control pumps and valves to prevent spills and other hazards may have problems that have not been addressed by small to medium-size firms.

"Even chemical companies that have actively addressed the Y2K problem may have underestimated its depth," says an article in the society's Chemical and Engineering News.

"Consultants hired by Occidental Chemical found 10 times more systems with potential Y2K problems than the company's own engineers found."

The new assessment of Y2K progress by larger American companies from Manufacturers Alliance/MAPI Inc., on the other hand, found cause for "cautious optimism" among big companies, given the level of awareness and the amount of effort.

Larger companies surveyed said they were on track to be compliant by 2000, while smaller firms were having trouble finding technical help that was affordable and competent.

"In the final analysis, the Y2K issue is an annoying, resource- intensive exercise in triage and damage mitigation," the report concludes. "Time is short and the stakes are high.

"The century rollover could be a nuisance or a calamity depending on the diligence with which Y2K correction is pursued."

[snip]



-- Kevin (mixesmusic@worldnet.att.net), April 14, 1999.


"At a pharmaceutical firm with operations in 39 countries, for example, Tava found 4,457 embedded processors in the laboratory equipment and manufacturing facilities of one location.

"Based on an inventory it conducted, 18% of the items were not Y2K compliant and 17% could cause a plant shutdown or affect production.

""The chance of these systems failing was 70% for the lab and 80% for manufacturing and facilities," says Bill Heerman of Tava's Denver office"

Great Article.

So of the 18% found to be noncompliant 17% of them would cause serious disruptions. In a system with 1000 chips like the boeing 747 mentioned 170 chips could potentially be suspect of causing serious safety hazards.

Now, if the analogy isn't true and the number of chips that could cause disruption were less in any particular scenario that still means there is an alarming number of embedded chips that are tied into the critical systems. If only 5% of the chips have problems out of a thousand thats 50 chips, but 80% opperate critical systems. (It still doesn't look better.) Thats 30 chips out of a thousand. And how many are difficult to test or replace (or can;t replace? Conclusion is you ground the plane). Even if only 5% of the 5% of chips ran Critical systems that would leave 2-3 chips they must find. And what if they are not aware they need to look for them or that they figure the chance is too small that they will effect the critical systems.

It only takes one chip to effect a critical system. So what does the FAA deem as precautionary when examining this challenge to safety?

This picture is ugly no matter which way you look at it.

Is there still more good info out there?

Father

-- Thomas G. Hale (hale.t@att.net), April 14, 1999.


Eagle,

I'm a happily expecting parent. *smile* Though not a priest I'd say we should all get out a rosary.

Yes, I read the article.

I'm gonna have to look for a new job come the new year. I work at a Car Dealers.

Father

-- Thomas G. Hale (hale.t@att.net), April 14, 1999.


Our website has several good links to the embedded systems issue. The problem has surfaced in critical farming applications, (such as sophisticated harvesting/cultivation equipment and irrigation systems.) http://www.snowcrest.net/siskfarm/millenm.html

-- marsh (armstrng@sisqtel.net), April 14, 1999.


Moderation questions? read the FAQ