I’m in an email group, and someone sent out a link about a company that tried to bring back Craftsman tool manufacturing to the States. It failed. TL;DR: they had to buy their equipment from overseas, and get new parts and tech support from overseas. No one left in the US that could do it.
That brought out other stories about the loss of knowledge in manufacturing. Apparently there are only a handful of people in the US who really know how to repair toilet paper manufacturing gear.
But it’s not just manufacturing. In my latter days in telecom, I — officially — worked broadband support; DSL and fiber Internet. But one day, the supervisor from another group came to me in a tizzy, wanting to know if I knew Frame Relay.
I doubt that it’s used much anymore, but back in the day, FR was much beloved by customers who didn’t want to shell out the bigger bucks for dedicated T1 or T3 service. The company still had quite a few legacy FR customers, and one was out of service. And no one in that other group knew how to troubleshoot FR anymore. Apparently some manager remembered seeing the words “Frame Relay” on my resume.
The supervisor gave me the circuit number, and I pulled it up. LMI Down. “Problem’s at customer prem. His equipment is down.”
“How can you be sure?”
“Cuz that’s what LMI Down means.” (Basically, it’s our FR switch saying, ‘I’m fine, but the customer’s equipment stopped talking to me.’)
“But how can you be sure?!”
-sigh- So I looped the smartjack, ran a clean BER test, dropped the loop, and LMI Down again. “We’re good. It’s the customer.” Then I had to explain why the test I ran proved that. In small words. To the allegedly experienced technical supervisor for a multi-state telecom company.
After that, I seemed to get every Frame Relay trouble ticket, and every time it was LMI down.
Another company: I had recently started in the Indy office. One day, the shit hit the fan when a phone customer called 911. And the call was answered in… Evansville, I think. The office was new, and someone had screwed up the switch routing tables, and previous tech I replaced had never tested it.
Imagine you’re having a heart attack and you’re trying to get an ambulance from the wrong end of the state because a tech didn’t know how to do his job.
That lack of knowledge can bite you in the ass at lower levels than your technicians. If janitors don’t know what they’re doing….
At yet another telecom company, a customer’s circuit (also Frame Relay, by coincidence) started going dead Friday nights at about the same time. After a few weeks of that, the customer finally admitted that 1) their FR equipment was powered from a switched electrical receptacle on the same circuit as the room lights, and 2) their janitor was turning off the lights (and their gear) when he was done cleaning. After that, their circuit only went dead about every other Friday night, as the janitor was told not to turn the lights out, but didn’t always remember.
That customer was a nationwide trucking company. The Frame Relay circuit was how they coordinated loads between facilities for movement across the country. If you recall the shipping fiascoes during peak “pandemic,” you can guess what havoc the janitor was causing. Didn’t get your live saving meds on time?
Same company: Youngish engineer knocked out an entire SONET ring. He was doing potentially service affecting work, and shorted out a DC bus. When asked why he was doing that in the middle of the day, He replied, “The manual said it wouldn’t be service affecting.” But it was, because he didn’t know how to do the work correctly, or know enough to take in account the possibility of an error. That one affected a lot of businesses, including hospitals. Way to go, Sparky.
Then there was the high school that had an OC-12 SONET system. I started dropping Monday mornings, and coming back up about 15 minutes later. Turned out the janitor, for lack of a more convenient outlet, was stupidly unplugging the OC-12, so he could plug in his floor buffer. (I’ve heard a hopefully-apocryphal story of a hospital janitor doing the same thing with patient equipment; specifically a ventilator keeping the person alive.)
I’ve got more stories about ignorance causing problems; from the manager killing an FAA T3 connecting Air Traffic Control centers for coordinating flights, to a hospital IT manager who — I kid you not — didn’t know that his electrically operated computer terminal needed electricity to operate (and wanted the phone company to tell him when the electricity would come back on).
We’re running out of people who still know how to keep our technological systems running, when everything doesn’t work as planned. And it’s going to be killing people.
