First off, THANKS to all who contributed to our Chapter 1 thread. Second, JIM COOKE has agreed to join us with his insider's perspectives, so now you get to hear from *four* of us. (Everyone gives Jim a golf clap and welcome...) For chapter 3, another insider has offered his insights, bringing our total to five insider perspectives. :)

The first chapter summarized our first encounters with what was then called The Year 2000 Problem.

Chapter 2: Hypothesizing on how Y2k might affect power

==================================================

Dans comments:

A cursory look at how power might be affected started with a high level analysis of those parts of the power system that use date functions, or rely on computer software that uses date functions. Our first look revealed the following:

Power Generation: Very little date functionality except in newer plants, or those that are retrofitted. Nuclear plant controls and safety systems had very little (if any) date-dependent functions. The systems with some date functionality include programmable logic controllers (PLCs) in Distributed Control Systems (DCSs), and software in continuous emission monitoring systems (CEMS), along with some newer protective relays and data logging and test equipment.

Power Transmission: Again, very little date functionality overall. Newer protective Relays, transformer controls and remote transmitting units (RTUs) had date functionality, but mostly for date and time stamping functions. The energy management systems (EMSs) ALL had date functionality that was considered important (remember, the term mission-critical was not yet widely used).

Power Distribution: Same as for transmission, but there were also line recloser controls and transfer switch controls to contend with.

In summary, when we took a first stab at things, even though less than 10% of equipment even had date functionality, the three items of most concern were:

DCSs at power plants The Energy Management System Protective relays and other controls at substations.

We were then anxious to begin testing, but the question was How do we test devices for Year 2000? Which dates do we test for?

====================================

David (Factfinder's) Comments:

Since I didn't begin working full time on a y2k nuclear plant project until mid 1998, I missed most of the "speculation" phase inside the industry. By this time, what I found at EPRI Web and vendor y2k sites was pretty good information. I was quite suprised to find that Intranet (Internal websites) of the plant I went to work at on y2k used a lot of "information" straight off the Internet! Some of this information was good (especially the IT related part), but the embedded stuff was full of the usual myths and misconceptions I would soon come to see for myself on the Internet and in the y2k forums.

==========================================

Malcolm's Comments:

Once we had identified all systems in use, even before we had considered whether or not they had any date functions at all, every individual item (both hardware and software) was assigned to a single person to be responsible for. Our job was first: to prioritize each item into one of four criticality's; second: to establish whether or not there was any likelihood of these systems being affected by Y2K; third: to test each system and report whether or not it was compliant.

The four levels of criticality were: If this system were to fail.....

1: Generation or critical operations would be lost or affected immediately.

2: Generation or critical operations would be lost or affected some time in the future.

3: Generation or critical operations would not be affected, but there would be a loss of efficiency, or business systems may fail.

4: No real effect on any system. (e.g.. fax machine may print a wrong date)

I was assigned a number of applications to check on, including two which were included in the level 1 criticality. Some of these systems did prove to have date functionality.

The method we used was first of all to contact the manufacturer of the hardware, or the supplier of the software and to enquire whether or not there may be any Y2K issues. If we received a reply that indicated there could be a problem, then that item was immediately put down for remediation or replacement. A negative reply, or no reply at all meant that we had to devise our own tests. We were never prepared to simply accept someone's word that a particular system was compliant.

The applications testing proved to be a long boring process. First the computer that each application was running on had to disconnected from the main network, and setup in a stand-alone mode (after a full backup of the application and any data had been made). Then the date and time was set to slightly before rollover and the application was run during what would have simulated a rollover. The computer was then shutdown for a few minutes and reboot to ensure that the application would continue after a reboot in the new year. The process was repeated, but this time the computer was shut down a minute or two prior to rollover, and the computer was re-boot shortly after.

These steps were then repeated for Jan 31st, Feb. 29th, Mar 31st, and Dec 30th and 31st. Remember some of these applications take quite some time to run, so this testing took quite a while.

As a result we found a number of Level 1 items that would be affected by the roll-over, however at our particular group of Power Stations there none that would actually cause a machine to trip. The actual results of these tests will be dealt with in a later chapter.

We were suprised to find far fewer items that had any date functionality than we first expected in the main power system, however we did find some strange programming techniques, and some very complicated methods of achieving rather simple results. At this point we were becoming optimistic that complete remediation would be achievable well within the time frame, but at no stage did we let ourselves become complacent.

It was also at this stage that I started searching the net for Y2K sites in order to share in whatever information others may already have. I initially went to CSY2K expecting to find out a lot about software, but instead I found a site full of arguments, and terms such as Polly and Doomer, and "If you live within 5 miles of a 7-11 you'll be toast". My progression around the net will follow logically with the rest of the series.

===========================================

Jim Cooke's comments:

(Since 95% of my work was with embedded systems that's the perspective I'll be speaking from)

Thanks for the opportunity to add my comments.

Our first problem was to get a complete inventory of all embedded systems. When we we first looked at this issue in late 1997 we realized we had no idea how many embedded systems that we had and how exactly they all worked together. The operators felt that they knew but no one could put together a flow chart that gave an overall picture.

We scheduled an inventory in March, 1998. The criteria was "If it uses a date, count it". We hired an outside company to do the inventory so we'd have a hands off approach. They did exactly what they were told. Unfortunately, this wasn't a very good criteria since we ended up getting things like microwave ovens, clock radios, and computer monitors and keyboards as part of the inventory. We spent several months just normalizing the inventory and finally ended up with about 65,000 line items. The problems of did we get everything and did we test everything were to bedevil us until the end of the project.

None of us were even aware of the "iron triangle" idea. We knew that having gas and electricity was absolutley critical to both the economic and social structure of the country and we were determined to do all we could to make sure we were ready.

The next big issue was "What's critical and what's not" which I'll talk about next time.

=============================================

Discussion question: What were your first areas of concern? Early on in the Y2k debate, there was mention of an iron triangle of electricity, banking and communications. Which area were you most concerned with, and why?

Chapter 3 is next: Early test results and encounters with vendors

-- Dan the Power Man (dgman19938@aol.com), April 20, 2000

Answers

It is hard to pick one of the three points of the triangle, but for me it was banking. I do not know for sure if I bought into the idea of "cascading cross-defaults", but I was aware of just how intricate the world's banking system was, and I understood the how crowds react under stress. So I did have a fear of bank runs.

I am by no means a technical person, but I would have understood if people were bringing actual samples of how a partucular system would fail on roll-over. I scoured the net for this information, and most of what I found was innuendo. I never became convinced of a tragedy because no one could show me exactly what systems within the triangle would fail, how they would fail, and, most importantly, no one ever convinced me the remediation would not be done.

Jim, you often speak about how we always muddle through with imperfect technology, and I am AMAZED at how we do it. Trillions and trillions of banking transactions takeplace every day, and most of us will never see an ATM screw up a deposit or withdrawel. Sure errors are made with scanners at supermarkets, tellers make errors in the bank, but all in all, considering how many machines and systems interact every day, the failure rates we see are infinitely small.

My kudos to folks like you who saw the problem and corrected it. Thanks.

-- FutureShock (gray@matter.think), April 20, 2000.

FS:

My big worry was whether people would react emotionally and cause things like gas shortages and bank runs. I was certain that the technical stuff would work. I was a machine polly and a people doomer. My lesson was that I was much more cynical about how the average person would react than was justified. People examined the information and reacted appropriately. There was never a wave a fear or panic and that was more suprising to me than the lack of technical failures. Just as we muddle through technical glitches, the vast majority of people are able to muddle through human problems without overreacting. I have a lot more faith in my fellow man today than I did last year.

-- Jim Cooke (JJCooke@yahoo.com), April 20, 2000.

For those who missed the initial installment, here's the link to chapter 1.

-- Malcolm Taylor (taylorm@es.co.nz), April 22, 2000.

If the iron triangle included power, communications, and banking, I'd have to say I was only worried about power. If power stayed up, anything else could be fixed. It's unclear why I never worried about communication. I suppose it could be because communication has so many options. There are cell phones, walkie-talkies, CB radios, etc. I noticed on the internet that some folks were concerned about 911 systems working. 911 was only fairly recently introduced to society.

I was never concerned about bank runs. We had several conversations regarding banks on TB2000 last year and on other Y2k fora. I had no intention of removing my money from the bank, had no intention of removing my mom's money from the bank, and knew absolutely NO ONE IRL who had that intention. I did an informal survey in 1998 of everyone with whom I communicated via E-mail at the time. Not one person was concerned about Y2k causing problems, and that included folks in the IT field and those outside. It was apparent to me that Y2k was a significant event ONLY to folks immersing themselves in Y2k information. This 5% or less of the population would not be enough to affect the banking system. I WAS concerned that folks could make last-minute runs on their ATM's. This concern led me to extrapolate that some vendors COULD run out of smaller bills with which to make change. For that reason, I started hoarding smaller bills in the summer of 1998 timeframe so *I* wouldn't be in a position where I'd have to pay $20.00 for a $2.00 item. In addition to hoarding these smaller bills, I felt there a possibility of increased burglaries around the new year because folks had been advised to withdraw SOME money JUST IN CASE. For this reason, I neatly placed small bills in sandwich bags and duct-taped these bags in "hidden areas" around the house. I never used the same type of area twice. One "baggie" could be taped behind a drawer. One could be taped behind the sink bowl under a vanity, etc. I kept a record of where each one was located on an unlabeled floppy disk. I never figured out which floppy disk that was, but memory served quite well when I needed money. When my daughter began moving out this past weekend, I found another "baggie". The duct tape had finally grown tired and it fell from behind a brace in a closet she'd just emptied.

In addition, I felt the perception of petroleum shortages could cause folks to run to gas stations at the last minute. For this purpose I purchased two 5-gallon gas containers. Storing fuel makes me jumpy, so I asked the owner of the house in which I live if HE thought it necessary that these 5-gallon gas containers be filled in the last weeks of 1999. In addition, I asked if he would feel COMFORTABLE with 10 gallons of gasoline sitting in the garage attached to the house. He didn't feel it necessary, so the containers remain empty in the garage.

-- Anita (Anita_S3@hotmail.com), April 22, 2000.

Anita: Thanks for your comments. You were wise to be concerned about storing fuel. I love the story about the "baggies" of small bills. :)

-- Dan the Power Man (dgman19938@aol.com), April 24, 2000.

Does it amaze anyone else that this thread has not received more attention on what was formerly a Y2K forum? That Elian is more worthy of our time than an examination of what these folk knew prior to roll over?

Has everyone given up on learning what there is to learn about what was one of the biggest overreactions in the history of humankind?

I thought we were a bit more intellectual on this forum, but alas, myself included, we cannot avoid gossiping about the news.

Guys, keep up the work and maybe some folk will notice and contribute to this noble cause.

-- FutureShock (gray@matter.think), April 25, 2000.

Dan, David, Malcolm, Jim,

Since I'd worked on early (1992) Y2k remediation in the financial industry (see my history in the Chapter 1 thread), I had confidence that electronic funds transfer and most ATM software would be okay, and that the financial industry overall was taking things seriously. OTOH, I'd seen a few "skeletons in the closet" -- aaaccckkk!

Electricity concerned me most, because it seemed that the consequences of that point's failure could be more basic, more devastating, than banking or communication failure. Maybe because I was living in a northern state (WI), and the rollover would occur in winter. I spent my first 28 years farther south (OK), and have never gotten over a certain deep fear of winter cold since I moved north. I never felt life-threatened in any Oklahoma power outage, but the rapidity with which my ill-insulated townhouse cooled during my first Wisconsin winter power outage was realy really scary. (Where I now live is better insulated.)

- - -

FutureShock,

I hope you continue to find ATM failures rare. My coworkers and I spent thousands of hours in test rooms with a dozen ATMs lined up with their covers off and guts exposed, currency trays filled with play money, so that we could systematically run through hundreds of combinations of error scenarios to make sure that each failure is handled in the intended manner to maximally protect the customer.

Most recent ATM models include multiple internal protections to ensure that currency jams or miscounts are corrected before currency is presented to the customer, and reported to the ATM network's control center and that all other forseeable errors are prevented from disrupting customer transactions and reported to the control center. Whenever an error condition prevents further proper operation of customer transactions, the design is to shut down the ATM rather than risk making a mistake in a transaction.

Generally, our testing scripts had a few dozen normal transactions to run through all the combinations of possible customer requests, then page after page of a couple of hundred error conditions. Y2k testing just added a few more lines.

Of course, ATMs, like VCRs, are complicated electro-mechanical devices, and their manufacturers don't always think of everything. A certain early model had a propensity for dispensing "free" money if the customer pressed a certain button in a certain way. Naughty, naughty. Well, they don't sell that model anymore.

>Does it amaze anyone else that this thread has not received more attention on what was formerly a Y2K forum?

Well, that's good ol' human nature again. Not too many of us enjoy analyzing like this.

>Has everyone given up on learning what there is to learn about what was one of the biggest overreactions in the history of humankind?

Among my pre-Y2k predictions, which have proved monotonously wrong, was that there would be many thousands, perhaps a million, academic theses on the subject of Y2k written during the next century. One of that prediction's flaws was that it was based on the supposition that the rollover was going to be a lot more exciting than it actually turned out.

Because the main event turned out so flat, folks lost interest fast.

Do we think the overreaction has anything to teach us? Well, maybe we can start by dissecting the origins of the overreaction. Is there anything there of interest? Religious apocalyptic fervor -- maybe Y2k stole the spotlight from this? Fears of technology -- fell flat because there wasn't any collapse. Y2k ended not with a bang, but with a whimper.

It just turned out to be quite unexciting. A few taxi meters, a few rainfall monitors.

I agree that there is a real pre-Y2k story here, but the flat denouement has spoiled our passion for it.

Richard B. Woods

Milwaukee, Wisconsin

-- No Spam Please (nos_pam_please@hotmail.com), April 27, 2000.

Richard:

Thanks for the "inside" look at ATM's. Thanks for your work on remediation. And thanks for your other comments here.

-- FutureShock (gray@matter.think), April 27, 2000.

I would not be so sure that the interest in this series is small. I for one, am following it closely but do not have anything to contribute at the moment. I'm sure others appreciate the details from the different sectors.

Well, maybe we can start by dissecting the origins of the overreaction. Is there anything there of interest?

Yes there is something to it, there was an entire generation trained to accept the possibility of "end of the society" scenarios. I will write about it later.

-- Cherri (sams@brigadoon.com), April 27, 2000.

Future Shock: Yes, we do tend to follow the sensational stories, but such is human nature. Don't worry--I plan to "keep up the work" and finish all the chapters.

Richard: I did have an opportunity to Y2k test the power company's version of the ATM, as we had several portable units around town where people can pay their bills. It was an OS/2 system, and it tested just fine. We fed in checks as well as $20 bills.

Cherri: Thanks for letting us know that you are reading the series. I'm trying to gear the discussion questions so that everyone can participate...

-- Dan the Power Man (dgman19938@aol.com), April 28, 2000.

Dan:

I look forward to the continuation of this series.

-- FutureShock (gray@matter.think), April 28, 2000.

Dan- I too look forward to the rest of the series. I believe the lack of replies are due to the somewhat "technical" nature of this chapter. It's my guess the remaining "chapters" will be more interesting, offer more feedback opportunities and receive much greater attention.

Future chapters:

Experiences with the media

Experiences with Y2k message boards

Pre Y2k dates: 1/1/1999, 4/9/1999, 8/21/1999, and 9/9/99

Y2k Power Industry Heroes (and of course, Villains!)

Confessions: A few things you never knew about, including what the lawyers advised us NOT to talk about

Good job. Keep 'em coming

-- CD (costavike@hotmail.com), April 28, 2000.

Hi No Spam,

Sorry to get OT here, but since you deal with ATMs, I thought you would get a kick out of this story.

A few years ago, I went to an ATM to take out $100. The machine was out of $20, so it gave me the money in $5. But when I counted it, I had only $95. I counted it a couple of times, and made sure that 2 bills weren't stuck together. Since I was only out $5, I forgot about it.

When I got my statement, it showed the $100 withdrawl, but also showed a $100 credit, posted the next day! Being an honest guy, I called my bank to explain what happened. The girl told me, that since it wasn't their ATM, there was nothing she could do! She told me not to worry about it, and that the error was already written-off.

So I went from being $5 short, to $95 ahead! That paid for a couple years of ATM fees!

One of my few good experiences with a bank... <:)=

-- Sysman (y2kboard@yahoo.com), April 29, 2000.

I am reading these chapters with GREAT interest. They are my glimpse of what really went on.

When I saw the first one I thought, "AH, here's what I have been looking for--the nitty gritty on power."

No comments indicate no expertise--not a lack of interest.

-- Pam (jpjgood@penn.com), April 29, 2000.

Sorry everyone, for this commercial interruption. I was reading this post quietly. Then I saw Cherri's post, and I had to stand up. Cherri stated a "whole generation had been trained to accept the end of Society". Who da hell she talking about? We know she was a Chief (big inflated deal) in the military. Is she saying the American military is teaching protection for the end of society? Big lie, if that was her inference. Maybe her military experience was other than American. So we can rule out the American military being trained to accept the end of the world. Hell, they are too busy with physical fitness, why would they put their bodies through such physical torture if they believed the sky would fall tomorrow? Whom do we have left for this "training?". Can't imagine it to be the young two worker, two kid, baseball, soccer practice family. They are too busy trying to work, cook, wash clothes. Cherri, with your clothes dryer, running water, inside toilet, your tv clicker, your many dogs in your lap (God forbid you should visit the elderly). Guess who is left Cherri? People with an experience, outside of yours, you had to be there.....

-- To Cherri (fl@mes.com), April 29, 2000.

oops,forgot to say Cherri, some of us did not see it as the end of society, but saw a possible disruption in modern conveniences. Went on a two year vigil of reading, and it wasn't pretty, this was Data Mining (I borrow this phrase). It was an oddesy for me. Your mind will be opened by Data Mining. Open your mind, search for any subject on the internet. Dr. telling your cash challenged Aunt that only one expensive drug will work? Search that topic to death. Kind people, along the Cyber Space path, may provide an inexpensive alternative.

-- To Cherri (fl@mes.com), April 29, 2000.

Errr.....Maybe Cherri was talking about those of us who grew up in the 50's and 60's who pretty much believed the "end of society" was not only possible but probable due to nuclear war. Maybe you're too young to have done the duck and cover drills under your desk at school.

Why did you think she was talking about Y2K and attacking you in particular?

-- Jim Cooke (JJCooke@yahoo.com), April 30, 2000.