Long essay -- Why didn't Italy melt down?

greenspun.com : LUSENET : TB2K spinoff uncensored : One Thread

This is a subject that's been in the background, while various people make unquestioned assumptions about it. (And yes, any informed answer requires more IT experience than is required to estimate the size of Laura's powdered butt).

I know from my own profession why embedded systems didn't cause trouble -- they almost never require dates for the performance of their primary function. But IT systems as I understand it *rely* on dates for nearly every business purpose, including the business of government. So the issue is, what went RIGHT and how?

In a roundabout way, we've seen the notion that y2k was some kind of giant hoax, that remediation was nearly unnecessary, and that the large sums spent were wasted. This proposition makes little sense to me. Businesses are never disposed to waste large sums.

More from imagination than any hard data, I've dreamed up the following list of possible contributing factors to explain the LACK of problems experienced both in areas of the world that apparently performed little or no remediation, and by SME's who did (or rather didn't do) the same. NOTE that these are *guesses*. This is NOT my field.

1) As a class, date bugs aren't very virulent. When they happen, they don't tend to crash the system or corrupt data in subtle ways. Accordingly, they are easy to deal with (find and fix, or work around). Whether we could have known this in advance (assuming it's true) is another question.

2) Date bugs have an extremely steep curve of diminishing returns. In other words, only a small percentage of them are significantly awful. And these show up quickly, and are fixed quickly. Full buckwheat remediation might find thousands of date bugs in a system, costing a bundle to fix and test. But let's assume NO remediation was done on such a system, and FOF was adopted instead. Run, crash/screwup, fix. Get a bit further, repeat. Perhaps in only hours, this process will clean out the serious bugs.

Of course, in that case the entire system still has lots of date bugs, which will show up in annoying ways over an extended period. So the general maintenance cost ratchets up incrementally during that period. Live with it.

3) Most countries (and SME's) rely much more heavily on packaged software than the big US companies/agencies, who tended to roll their own. Much of that packaged software comes from a relative handful of vendors (IBM, Microsoft, a few dozen major application vendors), so the remediation was much more efficient -- it was done in one place one time, and just shipped to all customers as an upgrade.

4) Other countries are (with a few exceptions) significantly less automated than the US. Various reasons for this are possible. The politically correct reason (I guess) is that these countries are less developed, lacking the financial or educational base to support heavy computerization. A less politically correct possibility is that delays and breakdowns are more the norm than the exception in many countries, so the contribution made by date bugs is taken in stride and does not disrupt significantly more than usual. And the least politically correct suggestion is that what is regarded as "corruption" in the US (special favors for some, special roadblocks for others) is standard practice in much of the world. Automation makes such practices more difficult, and has been resisted for cultural reasons.

5) It's a conceptual error to regard "Italy" (or wherever) as monolithic. Most references I've seen to Italy are really references to the Italian government, and not to private enterprises operating wholly in Italy or headquartered there. And (face it) most activities of any government aren't particularly necessary, except to specific (and often small, albeit rich) special interests. And those interests don't *need* those activities, however much they *like* them.

6) Much of the US "remediaton" expense was not sharply focused on just fixing date bugs. (I discuss this in more detail at the end of my "Reply to George" post). Beyond the clearly unnecessary bunkers-n-generators contingency expenses, a great deal of money was spent on what we can consider general housecleaning -- long-overdue upgrades of hardware and software, along with needed documentation, assessment, inventories, and analysis of how systems worked together and which systems were "in there" but never used. Italy (etc.) could easily afford to skip all of this and business continues as usual.

7) We really didn't know much about what was going on in much of the world, and the TB2K doctrine has always been that things will be bad by default unless we know otherwise (and, of course, no sources that SAY otherwise are reliable). Maybe a sufficiency of effort really *was* being made (see point #2 above), but without fanfare.

I'd certainly appreciate if someone with applicable knowledge can suggest other explanations, and/or estimate to what degree each of the above may have contributed to the lack of observed problems.

-- Flint (flintc@mindspring.com), March 08, 2000


While the MSNBC Year 2000 Issues forum was still running, a poster remarked about remediator friends just returning to the U.S. from Italy. She said they made good money performing remediation there. While that post constitutes only anecdotal "evidence", I would consider it a mistake to believe that Italy did NOTHING.

While mainframe work was typically done prior to 1999 overall, SME's WERE working on the problem in 1999. Fox-Pro clients DID need remediation. A friend of mine in Chicago was remediating 6 clients at a time. These 6 might take a month or two to complete and then she'd move on to another 6 or so. She works with about 5 others at a small consulting firm, and Y2k remediation was their most popular task in 1999. So, again, I would consider it a mistake to believe that SME's did nothing or even fixed on failure. Other small firms that used OTS software had every reason to procrastinate until fixes stabilized, which you may recall was quite late in the year.

-- Anita (notgiving@anymore.thingee), March 08, 2000.

Having been there a couple of times, I'd say that Italy is in and has always been in a constant state of meltdown. If every computer in the country shut down for good, it would hardly make a ripple.

Regarding date bugs, most of them are/were not particularly virulent, though unpedictable results due to an incorrect date comparison can be fairly subtle.

But I tend to think it's another application of the 90/10 rule - 90% of your input runs through 10% of your code, and that 10% is the best known and understood part of your system. The rest is there to handle unusual conditions.

That's why you could possibly run glitch-free for weeks or months on systems that were not completely remediated. That's why 100% complete remediation was never necessary. I'd say your #2 factor sounds prime to me.

-- RC (randyxpher@aol.com), March 08, 2000.


Good guesses. Sound reasoning. One possibility is that errors are now occuring that have'nt been caught yet. You know. On going curruption, undetected until such and such, yadda yadda. This is very possible. Where I am located now they had a Y2K program, and now consider it a "Done Deal". Meaning they are not on the look out for currupt data. They assume everything is hunky dory. My current clients rely on their databases. Yet they are not concerned.

Watch six and keep your...

-- eyes_open (best@wishes.2all), March 08, 2000.

Almost forgot. Your dead wrong on one thing.

This was not one of your LONG essays. ;-)

Keep your...

-- eyes_open (best@wishes.2all), March 08, 2000.

Good post Flint. As usual, always valuable, whichever way anyone may want to see it. The title itself is major chord. The subject matter at hand does need in-depth research and thought and your input certainly opens the debate constructively. Your guesses are not only reasonable: they are the only ones. That's a helluva good start. But it also shows the magnitude of Y2K lack of analysis. Hundreds of individuals in academia and think-tanks should be doing what you are doing Flint, and they are not. Many a Nobel Prize has been awarded for far less Flint. So please keep at it. Don't leave us either.

I tend to agree with "eyes_open", which doesn't mean I'm any any way invalidating your excellent guesses. The problem is that you should be followed by a crowd which doesn't exist yet. We are talking about justifying many hundreds of billions of dollars worth of Y2K expenditure in a world were most of its people are literally hungry.

Congratulations Flint. Good start. Go for more.

Take care

-- George (jvilches@sminter.com.ar), March 08, 2000.

P.S.: the key is, as you say Flint, what went RIGHT and how. IF everything did go 'right', right? In other words, a unifying theory which would explain the Y2K non-event.

Point 1: Is it agreed and definetly concluded that Y2K is/was a non- event? That's a question which requires a thoughtfull, low-key answer.

Point 2: What about government, and government agencies, worldwide, which involve literally hundreds of millions of custom-made code? Think Customs, social security, salaries, medicare, etc., think India, Brazil, China, Russia, Indonesia, Ecuador, Argentina, Mexico...

Take care

-- George (jvilches@sminter.com.ar), March 08, 2000.


So long as it's understood that this is mostly outside my experience and I'm guessing, I don't mind taking guesses. But I'd prefer if people who've been there and done it chipped in a bit.

[Point 1: Is it agreed and definetly concluded that Y2K is/was a non- event? That's a question which requires a thoughtfull, low-key answer.]

A matter of definition. Though I believe you overstate the final cost by a factor of 4 or 5, it was STILL an expensive proposition, and in some celebrity cases caused major new implementations to be rushed. That's a small handful, but still feeling the fallout. So in the sense of money spent, I think "non-event" is an exaggeration. Not a huge event (estimated less than 20% of *IT* budget for a 3-year period) but not a non-event either.

Secondly, maintenance programmers have been dealing with date bug *bites* for a couple of years, and continue to do so. These were never "critical" in a TB2K sense, nobody went out of business because of them that I know of, but I get the sense that for over a year errors due to date bugs have been at or near the top of the maintenance effort in many places, and still are in some. It might be some months yet before the occasional date bugs are settled back into the general "noise" of normal daily problems.

So I think we can safely say that y2k problems weren't non-events inside the glass rooms, but fit that description fairly well outside those rooms. Certainly those who allege that just any random inconvenience is a "latent" date bug have yet to come up with a single smoking gun. It's far from rational to point to any local wind blowing YOU the wrong way (and someone else the right way) and start hollering about y2k, without any evidence at all.

[Point 2: What about government, and government agencies, worldwide, which involve literally hundreds of millions of custom-made code? Think Customs, social security, salaries, medicare, etc., think India, Brazil, China, Russia, Indonesia, Ecuador, Argentina, Mexico...]

I cannot even guess. I've never seen a trace of evidence that any of this code was either custom-written or was not, in most of the world. We know social security underwent *heavy* remediation for a decade, we know people are being paid. Maybe every single agency has their own tale to tell, no two alike?

-- Flint (flintc@mindspring.com), March 08, 2000.

Hey Flint, right or wrong, half-right or half-wrong, whatever your guesses/my guesses/anybody's guesses are, have you noticed how few posts this thread has deserved?

Is the matter not important enough?

Is it too hard to dig out?

Can Y2K-type phenomena and reactions happen again then?

Is everybody on Prozac or something??

Take care

-- George (jvilches@sminter.com.ar), March 09, 2000.

Moderation questions? read the FAQ