I have worked in telcom as a tech for many years. (not directly on the telcom switches, but directly with the telco company techs themselves to install our telephony equipment) and I have visited many co-locates and switch rooms. If they are rebooting the maintenance and management parts of their systems because of bugs, then they will be screwed in short order with more y2k bugs and other anomalies. Those terminals they use are old monochrome pieces of crap. They are used to send simple commands to the switches (resetting lines, cards, status requests, etc) Without those terminals, the switches cannot be maintained. I should tell you, that all the times I have been in "switch rooms" around the US, I have never been in one that didn't require CONSTANT switch maintenance, and Im talking about a REGULAR day...imagine the phone traffic for new years. Couple that with outages, and people calling just to see if they can call, and you have a recipe for disaster.

Another thing is that these switch rooms normally only have one or two of these maintenance terminals and employees to use them. If multiple problems start to occur, a backlog of trouble tickets will stack up to the cieling in a very short time. Even in dealing with Sprint just yesterday, it took them 5 hours to get one of their techs to look at my t-1 to see what the problem was. This is because they already have too much to do as it is. (all he had to do was punch it up on the terminal, and find the problem) The problem was that they had a complete DS3 go down (DS3 = 28 t-1's). They had no clue until I called, and they checked it out 5 hours later. OOPS! My t-2 was only 1 of 28 that was down, and they had NO IDEA. (and this is only at one small co-locate facility)

Most switch rooms and telco co-locates are a freaking mess. Very old equipment mixed in with new equipment, miles of cables looking like a huge bowl of spaghetti because the techs over the years have moved stuff around, installed new equipment on top of old equipment, or just plain didn't have enough pride in their work to keep things clean and organized. Furthermore, there are usually 1 or 2 techs that will run a medium sized switch room, and many times they have NO CLUE where to find things, or what is plugged into who, etc. etc. They have to consult each other, and take their sweet time to locate a trouble spot BECAUSE THEY ARE HIGHLY UNORGANIZED.

In short, telecom is a sloppy business, and it is the product of years and years of stacking newer equipment on top of the older equipment. I can tell you for a FACT that if true Y2K remediation was happening Nationwide like we have been told, we would have seen COUNTLESS outages just caused by THE REMEDIATION ITSELF!!!!!!!!! You cannot replace these systems, or even parts of these systems without having outages. You may be thinking that they could just route calls through another area or switch. While this may be true in SOME cases, you have to keep in mind that the large telcos out there aren't the only fish in the sea. There are THOUSANDS of small and medium companies that provide service all over the US. Also, these companies many times are sharing resources throughout their systems. They cannot simply re-route traffic- they do not have the RESOURCES. Even when they do this stuff in the wee hours of the morning, to aviod public dispare, they screw up constantly. There are hundreds of telephony protocols and configurations. One screw up and its a big fat NO- GO. I have spent DAYS with a switch tech from NYNEX (Bell Atlantic) SPRINT, ELI, US WEST, GST just to get one stupid T-1 configured to work with our system. OUR SYSTEM USES A TEXTBOOK PROTOCOLnothing special!!!!!!!!!!!!!!!!!!

For example: I bought the t-1 connection in my office from a company called ELI (Electric Lightwave). They charge me $999/month for this connection. However, they are getting the t-1 from a company called Nextlink, who is getting the t-1 from Sprint. Each one of these 3 companies at some level, has my t1 running through their equipment. If any one of these 3 has a problem, then my t-1 goes down (and has 3 times already in the 4 months I have had it). This is an EXTREMLY common occurrence. All these telcos large and small share resources, and they all use THOUSANDS of different brands of equipment (of all different ages). Think about how old our phone system is. It is impossible to just update hardware in these systems. You can upgrade PARTS of the system, but it is impossible to replace everything. Some of the equipment out there is upwards of 25+ years old.

You cannot simple start unplugging and replacing this stuff, and I can tell you that industrial telecom equipment is so expensive it is insane. Many of these companies live off of TINY profit margins so that they can be somewhat competitive with the MEGA companies. It is ludicrous to think they can just REPLACE or upgrade this stuff.

The point of all this rambling is simple. If there exists even a few Y2K bugs that are repeated throughout our telephony systems here in the US (and the WORLD), then you can expect major phone problems. The telcos are simply not designed to handle this type of crap. ESPECIALLY the small and Medium companies who make up a larger part of the system than you think. I do not know the diffent hardware (and all of its ages, incarnations, etc) to be able to say how Y2K sensitive they are. I only offer insight into the nature of this MESS we all Telecom. I just hope the date sensitive systems are few.

I'll be happy to answer any questions as best I can.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999

Answers

Thanks for your insights, that took courage in our current legal enviro. If that testimony doesn't make it feel real for folks, then nothing will!

Do you have a link to your predictions?

-- Hokie (nn@nn.com), December 03, 1999.

Well, I didnt really say anything controversial, and I in no way consider myself an expert. I spoke only of my experience in the years I have visited various switch sites, and worked with several different telcom companies. I find them IN MY OPINION to be unorganzied. Not only that but in my experience, I found TONS of interconected systems of various equipment of various ages, and MANY MANY messes of hardware and wires.

PLEASE DO NOT PANIC about my above statements. As I said, the mount of problems will be related to how Y2K sensative the telecom hardware is, and I am in NO WAY an expert on all these systems. I doubt anyone can be an "expert". *IF* the hardware is Y2K sensitive, then things *could* get out of hand and BACKLOGGED very quickly.

I AM NOT AN EXPERT AND ALL OF THE STATEMENTS MADE IN THIS POST AND THE ONE ENTITLED "MY TAKE ON THE TELECOM SYSTEM" ARE PURELY MY OPINION!!!!!!!!!!!!

There..that should be a good enough disclaimer :)

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

Also, I DO NOT have any predictions. It is impossible to predict something like that because of the sheer magnitude of the Telecom system. Once again, the possibility of problems is connected to the ammount of UNFIXED - Y2K sensitive equipment out there. Obviously, this is largely unknown by us average folks.

If I had to take a wild guess though, Id say the smaller telcos are at the most risk because of the cost of remediation, and the fact that many of them don't have alternative resources if something should happen to their primary resources.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

Mr(?) Hill:

I too work closely with our internal phone switch and am the point of contact for the telco companies at our site. Dealing with MCI, Sprint or any other vendor has turned into a cruel joke. It takes 6- 8 weeks for a new circuit install, and often the technical support people's skills are questionable at best.

My concern is this: If people are going to be stupid and test their phone at midnight for a dial tone (which is a virtual certainty), this will undoubtedly take out a large amount of switching equipment in the overall system. What will this do to WAN links? Even though voice and data are separate functions, they are technically routed and switched through the same network (the phone system is the network in this case.)

Unless you actually ran cable directly from one location to another, the connection has to be routed through the phone system somehow. I don't see much promise of many WAN links being operational after the first. After that, knowing the skill level and availability of our glorious telco providers, I don't believe the functionality will be restored very quickly.

I have to chuckle when the home office mandates a call to them at fifteen past midnight to alert them of any problems. "Use the cell phones if the regular ones don't work" is the backup plan. Like cell phones are magic and don't require any equipment anywhere to function.

Think about this. IMHO, people are overlooking this aspect of their planning. The failure of voice communications has all but been conceded by the phone companies, but I don't believe people realize the impact on the data side of things. Potential services disrupted:

Internet ATM transactions 911 services Power plant monitoring and reporting Credit card transaction processing Connectivity to the home office or satellite locations The air traffic control system Stock and financial market information dissemination Any operation that requires phone or data networks to connect to any other location.

This will be the ultimate lesson in interconnectivity, and I believe a painful one.

-- ariZONEa (not@the.phone), December 03, 1999.

Yeah, I totally agree. I really do hoe the telecoms have fixed everything (laff), because if we start having phone outages, that means we will also have data outages because, as you said, they are for the most part, the same system.

I know what you are saying about Sprint being slow. Here in Vegas I have been trying to get a single phone line in for two months in my new home. I just got it yesterday after countless weeks of delays. The same thing happened with our business lines. I really don't see how they could handle multiple outages when they can't even keep up with regular service demands.

I guess we will see. If we get through this ok, then I will have an All new respect for the phone system.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

Cory,

From your posting its obvious that your knowledge/info on the telco equipment is quite dated. Most of the Operators (read AT&T, Sprint, and Bell South) utilize sophisticated software, Operations System Support (OSS), running on UNIX machines at Network Operations Centers to monitor and trouble shoot their systems. A single NOC may handle approx 5 - 10 switches - AT&T has a single NOC that handles all of southern CA without any difficulty. Additionally, I am very much affronted by your "broad" characterization of the technicians and engineers working within this industry. I have worked with all of the above operators as a Systems Design Engineer (Hardware and Software) for the past 8 yrs and have been amazed at their competence (competence does vary across USA and the World with our equipment) - some of the Technical Staff know our equipment as well as our engineers. Moreover, one should take a look at In Service Performance (ISP) for various operators/vendors (majority of the operators have an ISP < 10 min/mo) before making such a WILD ASS STATEMENT based on antedotal (or limited) data. Regarding, mixing of equipment - yes, every operator utilizes a mixture of vendors but than again all vendors design around an open spec and perform exhaustive interop tests (whats the big deal). Our switches (most popular Central Office Switch in the world - both wireless & wireline) can function with any LUCENT, NORTEL, ALCATEL, NOKIA, TANDEM equipment without any problems.

Concerning the mixture of Old and New equipment - you would be suprised how many "CHANGEOUTS" have occurred within the past 5 - 10 yrs. Prices have fallen and demands for features increased leading to the operators replacing the equipment in rapid fashion - of course the new rage is Telephony over IP (Can you say bye-bye circuit switches and hello routers and servers) which will lead to another round of changeouts. Finally, the Telecomm Industry is not in such dire straits as alluded to within your posting - and this is from someone that has a first row seat regarding System Design and Remediation. If desired, I can provide many links and info to illustrate/support the successful testing performed by the industry. Regards,

-- william holst (w_holst@hotmail.com), December 03, 1999.

This is a side note but worthy of reporting. The power company my husband works for has a contignency plan in place if the phones are not operating at midnight. They plan to send someone directly to our home if the phones are down and request that my husband go to work. This is my own personal opinion, but I suspect ITSHTF and regardless if you answer the phone or not, if they need you to work, they will come and get you. Though my husband is not expected to be on call, his boss told them yesterday that that was the plan.

-- bardou (bardou@baloney.com), December 03, 1999.

As I said, the small and medium companies are more at risk than the Big Budget, Mega phone companies. They simply do not have the resources and money to handle such a big job.

My experience is not dated, I work with this stuff everyday, and I am on the phone with INCOMPETENT switch techs on a daily basis.

If you are experienced in Telecom, then you must have seen switchrooms, and co-locates that made your skin crawl. Tangled messes of wires, equipment plugged into the wall instead of a UPS power source or DC (with the wire dangling in the air for any moron to trip over), old equipment half working, half not..new equipment that is not configured correctly because of lack of tech training, rooms that were WAY too hot for the equipment to run stable, techs that are so unorganized they are a joke, the list goes on.

I assure you these things are happening TODAY. There is nothing dated about this information.

As for the big players, they cant even get a circuit up in a decent time because they are in RED TAPE hell. Then when they do get it up and running , misinformation abound on the configuration, because it has been passed through so many different people. Forget about residential/business Trouble tickets!

Also, even though these telecoms have updated to newer networks, fiber loops, multi-million dollar switches. etc - they still sometimes have to utilize older facilities to get the service to the end user.

Lets also not forget the possibility that they simply did not have enough time to remediate. I worked with customers who took over a year, with countless committee meetings, and arguments before they would spend $50,000 on some equipment. (and $50,000 worth of telecom equipment isnt much equipment at all)

I guess it depends on what end of this industry you work on. I have to deal with the switch people. I have to spend entire days on the phone just to get a standard t-1 protocol working correctly, and I have to deal with my customers when their system goes down due to a telecom outage. Then I have to get on the phone and ARGUE with the telecom switch guy about WHOSE hardware is causing the problem, when I KNOW its not mine. Maybe you havent dealt with this end of it.

Then there is that little thing called human error. Hehe

Thanks for your input!

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

William Holst you rock! I love reading the words of someone who ACTUALLY HAS SOME EXPERIENCE AND BACKGROUND in the topic at hand rather than a loose cannon who thinks he has knowledge to discuss the telecomm industry just because he's talked to their lowest tier reps..

-- Forget it (garynorth2@yahoo.com), December 03, 1999.

Hey "forget it"

You forgot one little important detail in your pathetic attempt to "debunk" my opinion. The "lowest tier" as you call them, are the people who work FIRST HAND and HANDS ON with the HARDWARE and software in the phone system.

I work with those people. Do you think the people in upper management have a better handle on the state of the hardware which makes up the various phone systems out there? If so, please DO explain!

I have 8 years of experience, get a life.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

"before making such a WILD ASS STATEMENT based on antedotal (or limited) data. Regarding, mixing of equipment - yes, every operator utilizes a mixture of vendors but than again all vendors design around an open spec and perform exhaustive interop tests (whats the big deal). "

LOL

this is assuming the hardware/software was:

installed and configured correctly working with hardware/sofware that is on the "compatibility list" is being managed be a competent tech or group of techs

clearly you must have experience in the larger companies, and very little experience in the smaller CO's, colos and switch sites.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

TROLL ALERT

For those not aware, Cory Hill is a college student who is just here playing games. He admitted as much on the psychopathic debung-E's web board. Just look thru their archives and you can read his lies with your own eyes. (just don't stay too long; you might get infected by an "idiot" meme!)

-- Troll Patrol (busted@nd.exposed!), December 03, 1999.

also, my experience is with RANDOM switchsites all over the US. I have been about 50 random sites, and have seen the same PROBLEMS at each one...I find it hard to beleive that I landed in the only 50 trouble spots in the US.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

LOL....I told everyone on this forum that the whole 'College Kid" things was a sham that I used just to mess with the debunkers. I also appologised for my stress induced outburst on BOTH forums, and kept quiet for a month as part of my appology.

You can call me what you want. I am a 28 year old computer/internet/telecom tech (I work in computer telephony). I have 8 years of experience ON SITE with these "switch techs" and their various hardware configurations.

call me a troll as you will...i could care less.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

So you admit you lied before, but now we can 'trust you' huh, Hill?

Riiiiiiiight, troll, riiiiiiight.....

Go back to the debung-holes. Your 'double-agent' schtick is worn thin. Go home.

-- Troll Patrol (busted@nd.exposed!), December 03, 1999.

Say what you want..if it makes you feel special inside. I dont care if you beleive me or not. Im here, I work in the industry..I have the insight YOU DO NOT.

Take your bitter little soul elsewhere.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

Troll,

What's your area of expertise? Can't think of one?...or are you a just specialist in dirty laundry? Go away!

-- TM (mercier7@pdnt.com), December 03, 1999.

My expertise is in Computer Telephony and Internet Dev. Specicically, Prepaid calling card platforms, Prepaid Internet Access, Prepaid Wireless (cell/GSM/PCS), IVR systems, Windows NT, Windows 9x, Windows 3.11, Novell Netware (admin up to version 5) PC hardware and software (general). WEB/FTP servers (IIS, Netscape Enterprise).

Specific telephony: T-1, E-1, R2 (various international "versions" of this protocol), Dialogic telephony interface boards (240SC-T1 and 300SC-E1 used primarily in our systems now expanding to include Natural Microsystems, and a couple others)

Over the past 8 years I have built, installed, and supported these systems in 17 countries. The systems connect directly to T-1, E-1, and/or ISDI (PRI) lines provided by various local and long distance providers. While I didn't travel out of the country much, I supported out international techs locally.

I also have a few years experience in web page development including HTML, Dynamic HTML, Flash 3 (and now 4) and I am now getting my hands into XML and Perl.

My experience installing the telephony systems put me in direct contact with countless switch personel from several US and Foriegn carriers time and time again. My job on site is to install the system, and work with the switch techs to make sure the trunk groups and lines (t-1, E-1, or ISDN) are configured correctly so that our system would process the calls.

I am now learning Voice over IP using Dialogics DM1 boards, and am directly involved in the engineering of our new products, including SS7 and ATM interfaces

here's the part where you say ... "yeah riiiiiiiiiight"

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

I'm not going to answer any more of your rude posts, so save your breath.

-- C. Hill (pinionsmachine@hotmail.com), December 03, 1999.

Thanks for the information. The mess you have been describing is not only believable, but probable. Human nature doesn't change in any industry. People will do the least amount of work necessary to get the job done and they won't spend much time fixing a predecessors mistakes if it isn't absolutely necessary. (If it ain't broke, don't fix it)

Keep the information rolling.

-- James Chancellor, PE (publicworks1@bluebonnet.net), December 03, 1999.

As one who has worked in the telephone industry as an engineer/planner, I believe that basic telephone service will continue to work. The industry devotes substantial resources to keeping their software up to date for hte basic telephone switches. They are only dealing with two or three large vendors for this software such as Lucent and Nortel, which makes the job much easier. I am more concerned about the maintenance/support/admisistrative software, which is highly date dependent and probably more vulnerable to y2k problems.

-- Dave (dannco@hotmail.com), December 03, 1999.

Dave,

I would suggest that Ericsson has the biggest responsibility since 63% of Telecos world-wide employ the AXE. We have been busting our butts remediating switchs in China, South America, Russia and Europe with excellent results...Our customers deserve alot of the credit as they follow the guidelines for maintenance and keep the S/W Revision current - things have become alot easier since I joined E/// we now can perform remote updates to the customers switchs without having to travel. Regards,

-- william holst (w_holst@hotmail.com), December 03, 1999.

Hmmm. It appears as though someone is using my name. Will the "new" Troll Patrol please choose a different name so as to avoid confusion? Thank you.

-- (TrollPatrol@don't.org), December 03, 1999.

About 10 years ago, I took a job with GTE as a switching tech, starting at 50 cents below top union wage. The reason the locale supervisor offered the job to me was because none of his techs could fix the solid state battery charging equipment. Not much to it I thought.

After one year, I was so bored doing nothing most of the time, I went back to on the road contract work.

My impressions : Telephone techs are the mostly over-paid and incompetent technically. Anything requiring a little up-to-date knowledge was/is done by outside contractors.

The union protects them and if they tried to get a job that required passing an electronics/software test of any kind, they would be SOL. Just like working for a government agency, I suspect.

Has their overall tech abilities improved with time ? What do you think ? Parisites with a monopoly license don't have to, do they.

-- LongTimeLurker (qwer@hjkl.com), December 03, 1999.