A number of years ago, sometime in the 90's, I was called out to Nexen Gas Plant at Balzac, just outside Calgary. It was 2 AM and normally the on-call person should have taken the call, but the guys at Nexen always called me at home, directly. Before I went to the site for their problem, I asked a few questions and the answers were very perplexing. First of all they said that all the controllers were powered down. These were Provox controllers spread across 3 cabinets, and one cabinet had a bunch of I/O in it and an Allen Bradley PLC. They said EVERYTHING was dark in the cabinets, not one light was on. That begged the next question, "Is the plant running?" They said yes, in fact it was running quite well. I thought these guys were on drugs because without control, how can plant processes run? Even more confusing was the fact that this system had triple redundancy power backup, 24 volt batteries, a 110 volt UPS and redundant power supplies. Lose any two and you should still would have power!
I got to site quickly and met with two electricians, one instrument technician and two supervisors. The control room and back room where the controllers were strewn with large drawings as everyone tried to figure out how this system worked.
I knew the system quite well, having spent much time there. Opening up the cabinet doors, I verified that there was a total loss of power, but only to the cabinets. The Envox configuration station was still running, all the network stuff that wasn't in cabinets was okay, their SCADA system was also okay. Since they had remote I/O these devices were holding their last value to the plant instruments, explaining why the plant was still running. I looked at their computer screens and saw indeed that everything was running smooth, all trends were drawing nice straight traces! This was a wolf in sheep's clothing scenario. Operations thought there was no problem but the problem was there was no control in the plant. Once some external event occurred, like the sun coming up in a few hours, process variables would start to change as vessels started to heat up. Then they'd know they were out of control.
The first thing I did was check the 40 ampere fuse feeding the system. As I did so, the Bakelite case crumbled in my hand as the fuse fell out. Both ends of the fuse holder were very corroded from the high currents moving through this system. It became apparent that despite all the redundancy, everything had to pass through this one fuse, i.e. a single point of failure. I asked if they had any number 8 wire because I had a plan to bypass the fuse. They said they had none of this type of wire, but some guys scrambled out to check anyway. I didn't want to wait because the plant could start on its slippery slope any minute. Moving to another cabinet with few controllers in it, I quickly cut out a cable that had been tie-wrapped with others and was being used for redundant cabling to the redundant power bus bar. I quickly removed this length and stuck it into the location formally held by the fuse. When I saw the big spark I knew I should have shut off all the power supplies to the controllers before doing this, but no matter, I was committed now and just pushed the screw into its threads on the power bus and screwed it down. Everything came up, alarms started sounding and the trends started bouncing around.
The guys that were watching were amazed and I became an instant hero. I filled them in on what the problem was and then hurriedly went back to Spartan in Calgary (at 3 AM) to get another fuse holder and fuse. Returning to the plant, only one guy who was called out was there, everyone else had gone home. I tripped the controllers by shutting each power supply off, then removed my temporary patch cable. After installing the fuse, I brought the system up one controller at a time. Nexen lost no production.
About a week later I got a nice letter from Nexen management thanking me for my quick decisive action. They also told this to their sales guy, Guy Fiset who wrote the situation up so I could get a Spartan award.