Mid-year Review

greenspun.com : LUSENET : TimeBomb 2000 (Y2000) : One Thread

As the more senior members of this forum know, I have been an occasional respondent for about a year. It has been very interesting to watch various personalities come and go. My credentials include over 25 years of "big iron" software developement and trouble shooting. I currently have less and less time to devote to online Y2K matters so I will probably be even less involved here in the next 6 months. Therefore, I thought this would be a good time to post a personal review of where we have been, where we are and my own guesstimate of the road ahead.

Y2K did not become a generalized issue for major corps and the govt until early 1997. Oh, there were a few exceptions (like SSA), but the level of consciousness (LOC) was quite low. And the resources devoted to remediation were, to be kind, modest. The rest of the world was basically still asleep. By late 97, the alarm was beginning to sound from various independent sources and Y2K awareness reached a certain critical mass within government and business. People like Sen Moynihan and Congressman Horn were alerted by some of the managerial IS techies that this was going to be serious and expensive. Interestingly, anti-Y2K forces were organized very quickly. The whole concept threatened too many powerful people, especially amoung the financial interests. Anything that threatened short term profits in the current Wall Street Boom had to be minimized. Oh, money was allocated for the computer techies, but the PR people went into overdrive spin. I think they even convinced themselves that this was all so much Nerdic VooDoo nonsense. This wasn't some homogeneous conspiracy of the financial Orcs. It was simply a reflex of greed and incredibly narrow vision. The fundamental problem of Y2K literacy is that the underlying story is soooo mundane, drab, non-threatening and technically incomprehensible to the general public. It has zero media sex appeal - even in the literal sense. And, lets not forget the dominant news story of the time. Zippergate evoked a semen colored news fog which obscured the vision of most. And then in early 1998, within the major players, the unthinkable happened. The internal reports from senior IS managers were grim. Requests were made for major budget increases ranging from 100 - 400% greater than original (96-97) estimates. (If you don't believe me, go back and look at news reports/10Q's/annual reports for companies like AT&T, Chevron, GM etc). Decisions were made to scrap some things and convert to ERP software like SAP as quickly as possible. Old retired programmers were brought back as "consultants" and deals were were cut with firms like EDS (re:GM). A lot of Y2K remediation work was teleported to overseas programming brothels (Russia, India, Southeast Asia). The financial houses (ML, Pru, etc) pushed hard and made a lot of headway because their inhouse systems were relatively new. Other companies buried their heads and said "don't worry" they'd FOF (fix on failure) in 3 days on Rollover.(The amazing thing is that a lot of people believe they can fix in 3 days what they couldn't fix in 3 years!). Then, the whole issue of "embedded" technologies raised its head. This was of serious concern to the utility industry. So much of the infrastructure (electric substations, oil/gas pipeline control, satellite/telephone comm) is controlled by "smart" devices that even small failures could cascade into large ones. Chevron basically threw in the towel and said to expect "failures". Again, the issue was trivialized in the media into VCR, microwave, elevator failures. Some private tests of roboticized plants (Kraft, Chrysler) showed significant failures. The spin docs did their work. Some bandaids were applied and the story went away. Meanwhile the Federal government realized that panic was a real possibility. The decision was made at the highest level to manage the news. A bank run and economic ruin were greatly feared. The logic was as follows. Give the private and public sector as much "calm" time as possible to fix the problem. Press hard behind the scenes to get the tech problems solved, but present a public Happy Face. In the long run, that would mean more would get solved and so any Y2K disruption would be (hopefully) minmal. The gamble, of course, was that there better be minmal disruptions because very few people would be prepared. As 1998 progressed, the reports flowing back up the chain of command were mixed. SSA looked pretty good while DoD couldn't even effectively assess what was critical and what wasn't. Certain embarASSments occurred. The DoD was caught lying TWICE about nuclear weapons status testing so an embargo on Y2K reports was established. The FAA kept issuing contradictory statements re:deadlines (remember the done by 9/30/98 statement?). And then we slid into 1999....

At the beginnining of 1998/end of 1997 there were many statements by the mega-corps and the Federales that contained the mantra, "Done by Dec 1998. All of 1999 for testing." So what happened? NO ONE was done! Not one of the major players were finished and just polishing the roough edges. And don't hand me that legal excuse of "done but not saying". I have connections within the Fortune 100/Fedgovt that assure me that isn't true.

Meanwhile, Jo Anne Slavin became famous because of a donut eating, AR-15 packing super-geek. The Jo Anne Effect is one of the most misunderstood parts of Y2K. She (and Cory H) never predicted anything dramatic. Its a cummulative error that may be significant (ie:serious money problem) at some point. Its an accounting problem which will affect corporate accounting first and might affect production later. In general, the serious geeks expected some behind the scenes problems with 99 rollover and fiscal year rollover(and there have been some). These problems may grow through 99. BUT, the major problems are expected at 1/1/2000. Its called Y2K for a reason!

So thats where we have been. As of today, the news front is relatively quiet and that is perfectly understandable. JAE is present but is mostly a nuisance. That could change. Fiscal year rollover for the States just occurred. I wouldn't expect news of any problems until August or September. Five states have patched their welfare systems to work until 12/31/99. Many others will probably need to do the same. My assessment of where we are right now:

Federal Government--------------------------------------------
Overall, maybe 50% of "critical" programming remediated and tested. It varys widely by department. SSA looks good. HCFA, FAA and DoD are in serious trouble. Everybody else is somewhere in between. (Have you noticed that we have had different regional FAA problems every week for the last month?) The latest Horn report (PDF format) is frightening.
State Government(s)--------------------------------------------
Very difficult to get a good overview. However, they are probably in better shape than the FedGovt. Still, major state based entitlement programs (including State part of SSA) remain unfinished. Wildly different spending patterns. Pennsylvania says its finished and spent $40 million. New York, California, Florida, and Ohio have all spent two to five times as much.
Local Government(s)-------------------------------------------
Largely unprepared and clueless. Real threats to things like local water plants, sewage treatment facilities and 911 services. Depends where you live what to expect. Parts of the Northwest have done a great job. Large parts of the East Coast have done nothing beyond putting Y2K-OK stickers on their 386 based computers.
Major Corps (Fortune 100)-------------------------------------
Probably in the range of 85% remediation completed. Substantial amount of testing already done. The financial houses and major banks are even further along. Still, no one is actually done and not much integrated (end to end) testing accomplished. The oil and telecomm companies face special challenges with the "embedded" issue. There is a sense of confusion within these companies about what to do next about it. Major contingency planning is starting to get rolling. Alternate means of communication is a priority. Special dedicated satellite phones are being bought by the truck load. Some plants and operations will be suspended for Rollover. Vacations cancelled for all geeks & geekettes. I know of one major bank which has reserved hundreds of hotel rooms near its headquarters for Rollover weekend. I assure you its not for swinging ( Yeah, Baby!).
Utilities
Wow! This is the key quesion! Lose just 10% of electric power generation/transmission and you have a debacle. Despite the happyface reports, insiders are seriously worried about what they don't know. The Grids are dependent on communications. Lose telecomm services and (eventually) the Grids collapse. PECO (Philadelphia area) reported that they had to replace roughly half of the software/embedded devices that they could find. People are nervous about what might still be out there. Huge unknowns. Electric transmission is the Achilles heel. Ironically, the nukes are probably the safest and most reliable of the electric generation facilities. The actual operation is almost exclusively analog - not digital. However, the safety monitoring equipment IS digital and has had serious Y2K problems (re:Peach Bottom incident). I don't expect them to be shut down. (Shutting down the nukes would guarantee an East Coast Grid collapse.) I am more concerned about the gas fired plants. The pipeline SCADA systems are very vulnerable. Nationally, this supplies 10% of electric capacity. I have heard rumors that Washington D.C. will be "islanded" and guaranteed power which virtually insures a power outage in Virginia/Maryland. (Well, Milne will feel vindicated!)
Transportation--------------------------------------------------
The airlines are really in a quandry. There will be some domestic disruptions. Could be minor at first and then grow as intermittent failures cascade. The overseas part is looking like a nightmare. Stay out of the air! The FAA could use Moe, Shemp et. al. The railroads depend on very complex scheduling software. I do not have reliable status information. Again, this could be a cascading problem. Recent dismemberment of Conrail has been messy and caused serious JIT delays. The railroads also depend on good communications for switching trains remotely. Even small screwups could cause major JIT problems.
Foreign Overview
Europe - 12 to 18 months behind the US. Italy, Greece, Spain are hopeless. Germany gets 25% of its natural gas from Russia - nuff said. England roughly 6 months behind us with the exception of their banks which may be slightly ahead of ours. Their military is in somewhat better Y2K shape.
South America/Africa - Hopelessly behind
Russia- Very dangerous situation. They still believe this is some capitalist plot. We have DoD troops fixing their command and control systems which point weapons at us! Why isn't this a mjor news story? A lot of their computer technology is early 80's versions of stuff they stole from us!! In many ways, Russia is far more dangerous now than in the depths of the Cold War. America is oblivious.
China- Ditto Russia situation but add ethnic/racial bias. They truly believe we are the unworthy White barbarians.
Asia - Japan has really been working hard the last 6 months. Very hard to get reliable status info due to the insular culture. The utility infrastructure is vulnerable. Southeast asia is two years behind and wont catch up.

My best rough guesstimate of events:

Answers

Thanks for the update, RD.

-- dave (wootendave@hotmail.com), July 04, 1999.

RD, excellant review! This one should be put in a time capsule to explain what happened to us---to future people.

-- River (riverwn@aol.com), July 04, 1999.

RD: At some point in the months ahead Wall Street investors will suddenly realize the precarious position of their investments and decide to sell ASAP. Presently they are in a feel-good position with record highs. I won't be surprised if the DOW shoots up to 12,000 later this summer. But the balloon can't keep expanding forever.

What are your predictions of Y2K fears on Wall Street and other world stock markets?

-- Randolph (dinosaur@williams-net.com), July 04, 1999.


Excellent review.

-- Greybear (greybear@home.com), July 04, 1999.

Great post RD ! I too feel we're looking at a global depression, because of market crash/bank runs and oil export/shipping problems.

peace, Dan

-- Dan G (thepcguru@hotmail.com), July 04, 1999.



RD, Took a lot of courage to provide a global view of y2k. I really appreciate this short consensus of the whole.

Two things: Do you have a reference to PECO's having to replace half of the embedded chips they found. (They just announced 100% compliance).

Secondly, based on your report, your rough guesstimate seems on the low side. If the embedded chip problem is as bad as you have reported, and Russia, supplier of gas and oil, is not close to ready, how can you give a global infratructure collapse only a 5%-10% chance of happening?

-- BB (peace2u@bellatlantic.net), July 04, 1999.


RD: It was nice to read something written by someone who is a better position to judge than I am and is more optimistic than I am. 31% chance it will be less than a depression here in the states? Great!

The 5% - 10% chance of global infrastructure collapse is less than encouraging though. Makes me want to order more seeds.

-- Gus (y2kk@usa.net), July 04, 1999.


Did I miss something here? Reread that, Gus. He said 30% chance of Recession and 45% chance of global/US DEPRESSION !

-- sue (deco100@aol.com), July 04, 1999.

Thanks RD. I think this is close to my position also. I do not like the tremendous variables, and sincerely doubt if there will be too many surprises on the upside.

-- Mike Lang (webflier@erols.com), July 04, 1999.

RD,

That was an excellent synopsis of the situation. One of the best I have ever read presented as such with really a minimum of words. Hat's off to *you.*

-- Gordon (gpconnolly@aol.com), July 05, 1999.



Sue: 1% chance of minor bump, plus 30% chance of recession = 31% chance of something less than depression. Is that not how you read it?

-- Gus (y2kk@usa.net), July 05, 1999.

Thanks RD,

reinforces my contention that we all must think of this as a GLOBAL SYSTEMIC PROBLEM -

not looking too peachy is it???

-- Andy (2000EOD@prodigy.net), July 05, 1999.


RD -

You seem pretty gloomy on the Embedded front (Chevron "throwing in the towel" and all that). Your qualifications are in the Big Iron area - where are you getting the info about Embedded?

Regards

Johnny

-- Johnny Canuck (j_canuck@hotmail.com), July 05, 1999.


Here's my call. In Dec.1999 The banks world wide are getting run. The Fed is busy notifying them now to "batten down the hatches." We'll be seeing a HUGE confidence game being played out this fall as the banks and the markets spend a fortune in direct mail and Tv etc. to reassure the public. IMHO they will over do it and a significant portion of the middle class; say 15% will hit the banks early in Dec. Everyone else will follow. From Xmas to rollover could be a real nightmare as most rush to make last minute preparations. First the banks then the food stores. Then rollover and not much happens here in the US. On Mon. Big time celebrations as most y2k problems are still being held close by most orgs. If no major power failures y2k will be heralded as a NON- EVENT by the media. The market will surge for a week. Then the "real deal" will start. See it's really a Data problem and alot of these newly compliant systems haven't been really tested let alone linked to other networks and tested. The "Tower of Babel effect" begins and cascades creating a nightmare for networking engineers and programmers & system designers. Seems nobody ever decided on any standard protocols for remediation OOOps. As the month of Jan rolls into Feb. more and more systems go down and up and down and up and down again!. Increasing numbers of serious data foul ups bring many companies to a stand still or greatly slow them down. Internal problems start to consume many management teams.( Hey I thought these PC's were checked?) Services and the products they service are slowing down. Production lines start having supply problems in Feb. and March as stored supplies began to vanish. The bank runs and market dislocations of Dec. are now being felt as credit lines tighten and % rates rise to try and re-liquidify the banks. The Feds early 90's stargety of low % and treasury financed re-liquidification can not be done this time. (the Fed is Stretched to the limit from Dec's cash calls) The economy already suffering a classic "inventory recession" now begins to suffer both supply and accounting problems at the same moment. Lay offs begin as companies desperately try to stay a float. By summer 2000 unemployment just 4.5% a year earlier is edging up to 10%+ and the bottom is nowhere in site as a recessionary spirul accelerates. The automation that drove a huge boom just 1 year earlier has now turned into a cruel joke. The Blame is being shifted too and fro. In fact the presidental campaign of 2000 is nothing but a Y2k mud throwing match. By Fall 2000 Pres. elect Jesse Ventura is trying to give the public "the Real story." But, no matter who wins the 2000 presidental sweepstakes u can bet we the taxpayers will be forced to pick up the bill for Y2k. The congressional Y2k Liability bill passed in June 1999 and largely ignored by the public and the press now comes back to haunt many small businesses and individuals. Bill Clintoon is now being called the "Hoover of our times ". Welcome to the New Millennium! Oh yea have a nice day :)

-- Glenn L. Klotz (lee00@earthlink.net), July 08, 1999.

Scary if true. Is it hype? Is it part of a self-fulfilling prophecy made of a collective over active imagination?

At any rate, I will take mild precautions.

If the shit really hits the fan, where is the best place to be geographically, for the average family? ie- Manhattan? Honolulu? a suburb in the mid'west? a small coastal town in California? My guess is a small rural coastal community either in California or the South-east Atlantic. b/c low risk of crime, and temperate climate easier to live if no heat or electricity. Also b/c on the ocean may have greater chance to international trade w/ maritime commerce for scarce goods.

-- JTO (jtonealiii@aol.com), July 19, 1999.



Thanks,

Good note. Will print off and give to some "non-aware" friends. I don't really think of them as DGI's. They tend to listen and express concern when the subject is brought up but their lives are so busy with kids etc they really haven't found the time to sit down and think abt it. Hopefully this brief synopsis will help them along with their assessments.

-- Moore Dinty moore (not@thistime.com), July 19, 1999.


RD, I do not share your degree of pessimism--yet. And I may not. But I am compelled to congratulate and thank you most sincerely for promulgating your views without talking down to or belittling anyone. Your post held my interest and gave me much food for thought. Please keep posting on a regular basis as, heaven knows, we need a balanced view.

-- Old Git (anon@spamproblems.com), July 19, 1999.

Thanks, RD,

Yer a rock.

That was a terrific going away present you've given us. Stay in touch as possible.

Hey Andy. Welcome back

Hallyx

"The conveniences and comforts of humanity in general will be linked up by one mechanism, which will produce comforts and conveniences beyond human imagination. But the smallest mistake will bring the whole mechanism to a certain collapse. In this way the end of the world will be brought about." ---Pir-o-Murshid Inayat Khan, 1922 (Sufi Prophet)

-- (Hallyx@aol.com), July 19, 1999.


Notice the date of RD's post. This is a ripe post. Andy is not back.

Glenn, good to see you're still adjusting to the latest as we all must. Your post paints a picture of how this may go down. It is as good as any, better than most.

What should compound what you see is cyberterrorism (130 known viruses waiting for the rollover), terrorism using NBC weapons, possible Russian attack, and a stock market correction/collapse sometime before November. And then there is the mystery of GPS.

With this in mind I hope your analysis comes to pass.

-- BB (peace2u@bellatlantic.net), July 19, 1999.


I have been reading and watching the y2k issue for some time now. I have listened to people all across the board like most everyone else who has educated themselves to this problem. It is a very, very interesting problem to say the least! I agree and appreciate your synopsis of where the world currently stands regarding this issue. It seems to me that you have left out a scenario in your percentage of events that, while maybe it is a small percentage, needs to be vocalized given the catastrophic nature of this event. what percentage would or do you give to annihilation or near annihilation of the world thru nuclear holocaust?

Yours in Truth,

Damon Devine

-- Damon Devine (ddevine@impulse.net), July 24, 1999.


To paraphrase Rodgers Rangers "Keep your powder dry and your hatchet sharpened at all times" Never more have I believed that this could be the "Endgame" An excellent analysis with truly frightening consequences. I find it rather amusing how at this 'late date' the reality of the possiblity of the things that are to come are starting to grab hold of Mr. and Mrs. America. At this point....I'm keepin' mah head down, and mah kin close at hand....

-- Bill Coughlin (rakkasn@yahoo.com), July 31, 1999.

Nice job.

Check out IEEE's letter to McCain. AT BEST their uncertainty should chill even the warmest fuzzies!

http://www.ieeeusa.org/FORUM/POLICY/99may17.htm

Will their be a problem? That's for sure! How bad will it be? They don't know!

Don't get too euphoric over all the good news from Washington about Y2K readiness. Remember, the reason President Clinton signed an executive order to absorb a first strike in the event of a nuclear attack, is so not to cause panic, I mean where is everybody going to go? Maybe there is something prophetic in the recent rash of roller-coaster ride deaths, certainty not funny, but possibly prophetic. Get ready for the ride!

Joe

-- Joseph L. LaPlaca (jlaplaca@ra.rockwell.com), August 31, 1999.


Moderation questions? read the FAQ