Subject:	End of June 1999 Was: VA in BIG Trouble
Date:	1999/06/30
Author:	cory hamasaki <kiyoinc@ibm.XOUT.net>
	Posting History

On Wed, 30 Jun 1999 05:19:03, "Harlan Smith" <hwsmithXOUT@cris.com> wrote:   > Where are his references. He has provided none. Show me one report of a > remediation effort that was blocked by inability to procure an orphaned > part.   We've given up looking for the IBM 3330-11.  Although this is a disk drive, much like a big ZIP drive or EZ135, the problem is similar to the one of orphaned devices.   25 years ago, these were state of the art and available from primary and secondary sources.   15 years ago, these were being phased out, replaced in new applications by better devices.  Serviceable units were available for the hauling.   5 years ago, they were gone.  No units available anywhere.   Today, no substitions or re-engineering is possible in the time remaining.   In an abstract sense, this is exactly the problem that Tim May detailed.   The IBM 3330-11 is a parallel channel connected disk drive.  Think of a SCSI connected ZIP.    The media contains source code, executables, and data.   I can't buy a replacement drive from IBM, Stortek, Memorex, CDC, or any of the several companies that traded in these things.  I've scoured the reseller market, the scrap dealers, and have spoken with several museums and historic societies (There's one in Pennsylvania.)   We considered contracting for the construction of a head-on-a-stick flux change reading device.  With the flux changes, I only have to reconstuct the electronics that understands error correcting, data encoding, CKD, and solve the problem of aligning the tracks into cylinders.  The electronics can be built in software and I know people who have solved part of this problem.   Having done that, and not lost synchronization, I have to write the data onto a modern device in such a way that standard IBM access methods accept my data as system information.    If I can't do that, I have to reverse engineer VTOC, VSAM catalog, VSAM, QSAM, BPAM etc.   In the specific case of embedded controllers, some of these will not be found, some will be found but will be unfixable, some will fail.  What is unknown are the consequences. When these break, does the chemical plant blow up or does a production report contain bad numbers, " -3,422,034 gallons refined this month."   ===========   Look people, we're arguing the nits now.   Step back, the remediation failed last year.  Remember, December 31, 1998 and all of 1999 for testing?    March 31, 1999, I'm shaking my fist to show you that I'm pumped.   Then it's "Lets run a Y2K Test and show them that everything works!" Well no, Peach Bottom 2 blanks the monitors, City hall traps the LA mayor, Van Nuys floods the park with 3 million gallons of sewage.   June 30, 1999 and everyone, absolutely everyone is ready or my name isn't Albert Gore, uh, John Koskinem, no...  whatever...   We have 184 Days now, and the DC Government has announced plans to drag in locomotive sized generators to run the water pumps AND in the same report, the hospitals are hoarding, uh, storing up to 90 days of water.   It might be prudent to be at least as ready as a DeeCee hospital.   Add to that the NRC reports of "Good News, Nuclear plants might be ready soon." except in the fine print they say that a bunch of displays and radiation monitoring systems have Y2K problems.   If you step back and look at the last year or so, paul milne is tending his garden, probably a little bigger this year than last, tim may has ordered a few more tools and books, frank ney has his back to the wall but seems to have his sense of humor.  I've spoken with lots of doomies who are deciding on whether or not to add a few more small cans of ham, another couple cases of canned green beans, all of which they will use and enjoy.   On the big brother side, the horn-hairs around here are SCREAMING, "NOBODY PANIC, we're renting locomotive sized generators", we've told the cops they'll be working as armed guards at the food and water stations, uh, let's call them "warming centers".   Paul finishes his water tower, say, there's a nice view from here and hundreds of miles away, the DeeCee horn-hairs are trying to get another, what was it, 60 million dollars from the feds; what's this for? Snacks, tidying up a few loose ends.   WRP 123 has a revealing exchange between "Darth" Bennett and OMB on the June 30, 1999 status report.  Say, how *should* we spin this?  Bennett puts it to OMB, has 'em squirming.   How did we get into this mess?  How did the conventional wisdom pass us so suddenly?  I don't have a locomotive sized generator.  I have a 10 watt solar panel and will borrow a small generator for the roll.  I never expected a multi-million gallon sewage spill.  And we have 184 days to go.   cory hamasaki 4,432 hours, http://www.kiyoinc.com/current.html 

-- a (a@a.a), June 30, 1999

Answers

oops, I reached the wrong thread. I was looking for some pollys. Don't expect to find any here. See you later a.

-- Yes we see (thebigpicture@dragon.ranch), June 30, 1999.

He compares apples to oranges, obselete tech to modern embedded, and then leaves it up to the reader to swallow it whole. Have you looked for parts for an IBM-PC lately? Nope, and you aren't going to. You will buy a replacement. Apparently these characters have no backup for program data that was never archived but kept live on disk. Now that is a MAJOR no no in the IT industry. I don't think I need to document that - everybody knows you are urged to keep backups.

So ok - this problem has to be pretty uncommon.

-- Paul Davis (davisp1953@yahoo.com), June 30, 1999.

Paul, first of all, he's talking about multimillion dollar mainframes, not personal PCs. You don't just walk down to Circuit City and tell them you've heard about that nasty y2k thingy and you'd like to look at their line of super minis.

Secondly, the significance of the amount of code that has to be inventoried and accessed, inspected and remediated, tested and integrated into production is apparently lost on someone with as little software experience as yourself.

BTW - I heard today that the Corp of Engineers has declared the drainage ditches on Nebraska's state roads "protected waters". Were you in on that? :)

-- a (a@a.a), June 30, 1999.

Paul, I have no tech knowledge, but I do have YOU figured out, and consequently wouldn't trust my son's King snake with you. Please DO comment on Cory's comments BELOW the tech discussion, and quit acting as if that portion doesn't exist. Beginning with "Look people, we're arguing the nits now" (which is EXACTLY what you so covertly attempted to do *just now*)

-- Will continue (farming@home.com), June 30, 1999.

Paul,

::Apparently these characters have no backup for program data that was never archived but kept live on disk. Now that is a MAJOR no no in the IT industry. I don't think I need to document that - everybody knows you are urged to keep backups.

:: So ok - this problem has to be pretty uncommon.

This is MUCH more common than you would imagine. I've seen this happen over and over in my 15+ years of programming experience. Someone does a housecleaning and chucks the wrong tapes or printouts, or someone decides to reuse the wrong set of backups and WHAM! No source.

Also, MANY companies DO NOT get source created by consultants. If they are smart enough to ask about it (and you'd be suprised at how many SMEs do not) it usually goes into an escrow account on tape. If that tape can't be read (and I've seen that happen too) then no source.

How often does this happen? In my personal opinion I'd say that maybe 80% of the large corps, and near 100% of the smaller ones have had this happen to them at least once. Now I'm NOT saying that 80%-100% of the SOURCE is lost, I'm saying that 80%-100% have at one time or another lost source to one or more systems. Just ask anyone who's been in the biz for more than few years, they'll tell you...

-TECH32-

-- TECH32 (TECH32@NOMAIL.COM), June 30, 1999.

AAMOF Paul, just today I was confronted with this problem. A system I am attempting to make y2k compliant is not compatible with the latest version of one of the software subsystems. And the old system libraries will not work with the new OS. So it was off to the source repository to try to make a "new" old binary. Only the source in this case is so many versions behind, as is the compiler that built it, that it is probably easier to reengineer the new software than to dig up the old code, if it even still exists.

-- a (a@a.a), June 30, 1999.

I have worked for and around several DP shops - and all of them archived with a religous passion. Where I am working now, I set up the archive system currently used, and they got pretty upset when I told them they either had to shut down the email system during backups, or the backup of the database would miss many recent messages when restored, as the database would not be in synch over all its parts. (Could still be restored, but would have to be repaired, which would result in removing all recent messages.) How important is the last couple hours email - at midnight?

So I have to stand by what I said - this just can't be a common problem that affects most or even a sizeable percentage of DP shops.

BTW Will, a buddy of mine says king snake is pretty good if you clean them, soak in salt water, cut into 2 inch pieces and saute in butter. I recommend a white wine.

-- Paul Davis (davisp1953@yahoo.com), July 01, 1999.

WC - and what is to comment on in the bottom part? That is all editorial opinion - and CH is entitled to have any opinion he wants. Just like you or me. I think his writing style is awful, and have said so before, but what do you want me to say about his opinions? That I don't agree with them? I don't, that's hardly a secret, not even from CH - he knows that perfectly well.

He makes some assumptions that I find dreadful - like the assumption that the sewage spill problem in LA won't be fixed and will recur on 1/1/2000. Finding out where the problems are is why you test - if you wanted them to happen in production you wouldn't bother. And that testing procedure was flawed - you shouldn't run tests on live systems. As for the locomotive size generators - is there a link for that? GE 16 cylinder diesel/electric units are about the size of what he is talking about - but you don't drag them around anywhere - they are meant for fixed mount use - generally on a mobile platform like a ship.

-- Paul Davis (davisp1953@yahoo.com), July 01, 1999.

Paul: We are not talking about your rinky-dink department that uses the same puny 486 as a "development" system (I guess in this case that means creating Lotus macros) and email server. We're talking major IT shops here. The code is spread over hundreds of systems and thousands of media devices, some seperated in different parts of the globe.

As for the "locomotive size", that was the quote used by the DC authorities to describe their "preps"

As usual, you have made youself the laughingstock of the thread by injecting random BS that you obviously know little about.

-- a (a@a.a), July 01, 1999.

Didn't sound to me like the problem here is data recovery. Sounds like the problem is reconfiguring a system that currently expects a 3330 and is going to have to do data addressing, whatever, on some device with different characteristics.

He didn't say they failed to back it up, he just says they can't find parts now, and there's no time to convert to something else.

-- bw (home@puget.sound), July 01, 1999.

Paul Davis:

"He [Hamasaki] makes some assumptions that I find dreadful - like the assumption that the sewage spill problem in LA won't be fixed and will recur on 1/1/2000."

Funny how people reading identical text can come up with such divergent interpretations. I saw no such assumption in what Cory wrote -- he simply remarked on a singular and modestly disastrous event occurring during Y2K testing. Of course the utility learned from this -- of course that valve problem will be fixed. The significant part of this event is simply that it happened. A moment's reflection suggests the possibility that some other problems of equal or greater impact will not be found and fixed before the rollover. How probable is that "possibility"? Who can tell? Not me, for sure.

But that very contingency is implicit in what Paul himself writes in the same paragraph:

"Finding out where the problems are is why you test - if you wanted them to happen in production you wouldn't bother. And that testing procedure was flawed - you shouldn't run tests on live systems."

Right on. No question about that. Question is, how many utilities won't find all their glitch points in time?

Uncertainty continues to dominate this problem.

-- Tom Carey (tomcarey@mindspring.com), July 01, 1999.

Tsk, tsk a - wern't you the guy who was all over the place about personal attacks the other day? Shame on you.

Let's see here - an Amadahl mainframe configured to look like an IBM 360, a Honeywell mainframe connected to a WAN, and a system consisting of two Honeywell mainframes and a Prime mini, connected to a LAN via a PC used as a port/router/translator. I programmed on the first two, the wife was operator on the last. All were passionate about archiving, none fall into the 486/PC macro weenie class. Got a problem with that?

-- Paul Davis (davisp1953@yahoo.com), July 01, 1999.

Paul - you ar emissing the key point in these failures. Of course the specific failure will be fixed. That's why I'm glad to hear that people (companies) are testing.

Known and publicized major failures (sewage spills, welfare check errors, FAA radar control failures, and other failures like the 250,000 gallon Bellinham WA gasoline spill, the refinery shutdown and process abort, the Northwest Airlines failure(s) during testing, the loss of control displays, etc.) have occurred AFTER remediation, and in most cases, during manual operation or during tests after the repairs were complete. That is, when the operators are most careful, using specific procedures to check things. It happens now, when the distraction (and cause) of secondary failures is eliminated. When all support services ans emergency services are still available.

And yet catastrophic failures still happened. In some case, lives were lost.

NOW, what will happen to companies, agencies, and processes who have done nothing? What will happen when unexpected errors occur when emergency support itself is strained, overloaded, or not available?

To phrase it differently, what happens when 2500 people try to get in a "warming center" intended for 250, but actually only capable of handling 25?

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), July 01, 1999.

Cory,

From the message you posted with bits and pieces re: 3330-11, I can see that you do have a problem, but not one which is insurmountable.

In a former life (1987) I co-founded a corporation in Grand Prairie (DALLAS), TX which specialized in the refurbishment of disk devices and data recovery. They are still in business (and doing quite well) and are on the web as: WWW.TLSI.NET - So I do know a bit about DASD and basically all magnetic storage devices.

"The IBM 3330-11 is a parallel channel connected disk drive. Think of a SCSI connected ZIP."

Actually the 3330 (like all of it's predesessors and the 3350), this device used an interface called "SMD". This interface has been, for all intents and purposes, dead since SCSI-II made the scene in RAID devices prepared for mainframes by companies such as EMC.

"I can't buy a replacement drive from IBM, Stortek, Memorex, CDC, or any of the several companies that traded in these things. I've scoured the reseller market, the scrap dealers, and have spoken with several museums and historic societies (There's one in Pennsylvania.)"

Look SOUTH Cory - South America - that is. Contact service bureas there to get your disk pack data transfered to tape (be prepared to read it at 1600bpi though). Much of the older technology landed in countries such as Brazil, Venezuela and Argentina. There are other places here in the USA which can set you up with the right equipment, but what you really need is a Service Bureau to do it for you. A list of links to look at is here:

http://www.compufacts.com/used1.htm

But, you will also need the 3330 control unit to interface to your 3xxx mainframe. The same control unit was not used in the 2314 or 3350 - only in the 3330 series.

"We considered contracting for the construction of a head-on-a-stick flux change reading device. With the flux changes, I only have to reconstuct the electronics that understands error correcting, data encoding, CKD, and solve the problem of aligning the tracks into cylinders. The electronics can be built in software and I know people who have solved part of this problem."

A solution can be prepared to read the data from the pack but it is not a very pretty solution. The issue here is synchronization - I have prepared special head stacks for reading data in the past and the big problem is that the sectors are not aligned in the exact same location (rotationally) on adjacent platters. In other words, you can get the data from every track on a platter, but it will be only a fraction of the data from each cylinder. Drive electronics will start writing at the first available sector on a track - then when that track is full, the head select circuitry will select the next head and wait for the first available sector and start writing again. Now, which head was selected in which order? Well, that data is encoded in the VTOC and translation table under head-0. This is only a smidgen of the trouble in reovering data - The part that most people don't understand is that DASD was never "hard sectored" and it has **ALWAYS** been analog data. This means that unless your timing is absolutely perfect, you will never synchronize on a single track let alone a cylinder. In other words, data recovery - while possible - is not a likely solution.

There are still more alternatives - Reverse engineering of the source code. This is not that difficult of task if you know what the source was originally compiled as. There are, again, service bureaus which specialize in the recovery of source code from object code - you can find them right here in the WWW. I have done it on PC's but not mainframes - but there are plenty of those out there that do.

I have to ask how something like this happened to begin with. Having been in the DR business, I know that it happens frequently in the PC and Client Server arenas - but I have only heard of one other case where DASD with source had not been backed up and the DASD was summarily lost. It has been the mantra of IT shops for as many years as I have been involved with them - BACKUP / VERIFY BACKUPS / BACKWARD COMPATIBILITY OF BACKUP-RECOVERY SOFTWARE etc.. Lost source code? Someone should get their pee-pee slapped with a steel print ruler for this one.

Yours in COBOL... Dino!

-- (COBOL_Dinosaur@yahoo.com), July 01, 1999.

Just another thought Cory - you might look in the trade rag "PROCESSOR MAGAZINE" which has literally hundreds of surplus ads in it and comes out on a bi-weekly basis. I don't have one here but I am sure they are on the web somewhere.

Yours in COBOL... Dino!

-- (COBOL_Dinosaur@yahoo.com), July 01, 1999.

In a slightly different direction from the "recover old disk info" comments, but relating to the original title:

From Power Engineering magazine (Feb 1999), representative costs of a large "locomotive size generator" so glibly mentioned in the Washington DC contingency plans:

2,850kw diesel genset, $545,000.00, installation and testing = $250,000.00 (or more if a building and sound silencing is needed), lead time from order -> power = 6 months

84,000kw gas turbine genset (like new! says the ad), $12 million, installation = $500,000.00; time = 9 months.

20,000kw diesel genset, low hours (5 units available, but they are 50 Hz; so they couldn't be used here!) $1.0 million each, $3.8 million for the set. 5 months lead time required (if overtime used.)

And these are "used car" prices, for diesels and gas turbines already built!

Bottom line? if Washington hasn't written the check today to begin construction tommorrow, any large generator they contract to buy won't be available until February or March.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), July 01, 1999.

Basically all a loco-sized genset consists of is the diesel prime mover (engine) and alternator set out of a surplus diesel locomotive. They're refurbished then installed on a frame or trailer with hook-up provisions. Add a radiator, some controls and a fuel tank and it's ready to run.

During the Quebec ice storm, their alternative was to "borrow" a couple of 3600 horsepower locomotives, get them off the rails at a handy grade crossing and then "drive" them down the streets to the locations in need of power. There is a throttle setting which just happens to put out 60 hz power. I'm aware of one hospital and local command center which were powered this way.

In South Korea using locomotives to supplement the power grid was a standard practice during the sixties, seventies and into the eighties.

I'm sure this could work in DC and the streets there might actually be improved by the passing of a 200 ton locomotive. I'd take trenches running in the direction of travel over moon-crater potholes any day.

WW

-- Wildweasel (vtmldm@epix.net), July 01, 1999.

That's pretty interesting WW, but I think that if and when the day comes that diesel locomotives are plowing up M street, DC will have bigger problems!

-- a (a@a.a), July 01, 1999.

The 16 cylinder jobs I was talking about are about that size, and I am sure they aren't cheap. But they are built to last forever. Matter of fact, the generator side has GE Canada stamped on the field coil housing. So at least part of them was made in the North.

I didn't know you could throttle a locomotive to produce 60 cycle current - that is a possibly useful fact to remember. Thanks.

-- Paul Davis (davisp1953@yahoo.com), July 01, 1999.

But asgain - look at the lead times required. If DC doesn't actually begin construction NOW (dislike cap's, but it is critical to emphasize the date here), they simply cannot be installed in time.

If the units themselves must be "built" - forget it. Can't be done in time.

These are big, very heavy machines. Overall, 20-30 feet long (depends on how many cylinders), 15-20 feet high. Simply moving them into position from the nearest factory may take a week. The ones I operated have 8-10 inch diameter cylinders, require complex support systems (cooling water, oil, fuel supplies, fuel regulators and backup's, filters, lube oil, lube cleaners and coolers, generator controllers, coolers, lube oil, mechanical supports, civil supports (building, foundations, overhaed, cooling/heating/silencers/control stations, electrical links - even if temporary, etc.)

It's not a simple job. It can be done - but will require 5-6 months - once the dirt starts moving.

-- Robert A. Cook, PE (Kennesaw, GA) (cook.r@csaatl.com), July 02, 1999.

Cory Hamasaki is looking for a disc drive. He has the media.

Why is Paul Davis talking about file maintenance when Cory is looking for a drive?

.

-- Tom Beckner (tbeckner@erols.com), July 02, 1999.

HEY, DIANE! I just received this to my "private" email! WOW! Let's research ASAP, everyone:

"Yeah, guys, but you see, since we have to get the locos in place by Dec. 31 and our plan has penciled in those dates, by golly, it will all happen by then. Just like it has for the remediation effort itself. After all, we're the same guys who have worked on the remediation all this time. Trust us."

Confidential Memo to TB2K Forum from the D.C. Year 2000 Bill Clinton Gala and Contingency Planning Gang

-- BigDog (BigDog@duffer.com), July 02, 1999.