Taking my newly acquired but very cheap scope meter with me, I tried to look at noise on the HART wires but could see none. However, a physical inspection told me right away that there were wiring issues. Because HART is an analog signal operating at low levels, about 500 mV, any noise in that range could swamp the signal, causing the controllers to lose the message. After a few tries, it gives up and flags a major error which was, in this case, causing the control loop to go to manual mode. When this happened, operators scrambled to get control of the loop, and others in the field had to swing valves to maintain the process variable. It was a panic situation, occurring at totally random times. The wire that should be used is shielded twisted pair, or STP, definitely not the single strand non-shielded wire they were using. Husky told me that this was the wire recommended by the engineering company, Kenonic Controls.

I couldn't get anywhere with my analysis because I couldn't see the noise with my cheap, slow scope-meter, but the problems persisted. I called Dave Brown, Emerson's site installation guru for advice and asked him if he could come down. Guys in our office were under heavy pressure to get this fixed and since it was now a service issue, it was on my plate. (It was always like that. Whenever there was a new installation, as soon as the project guys deemed the project "delivered", it was out of their hands and now the responsibility of service. In this case however, it was warranted because the project guys like Alex did all they could to zero in on the problem and could not.) The issue at the plant was even more serious. Not only was the operation being disrupted at possible loss of product, but there were 28 electricians on standby getting paid but doing no work, waiting to complete the commissioning. Husky told them to wait until Spartan figured out what the problem was so I had the fate of 28 electricians in my hand.

Dave flew in on a Friday and I picked him up at the airport and got his big box of test equipment cleared through customs. We hit the road early the next morning. Dave is kind of an ornery, arrogant guy, the kind I like because they don't stand for bullshit and tell it like it is. But he and I got along well together, likely because he was my mentor and he knew I did good work for him when he wasn't around. In other words, when I call him, it's because there is a real big problem that requires his credentials.

As soon as Dave saw the wiring he knew this was a serious problem and he concurred with my initial analysis. But we needed more hard evidence. When Dave unpacked his color Fluke oscilloscope I was in love. What a great piece of equipment this was. In most cases I was the guy hooking up the wires while Dave did the interpretation, but Dave let me go through all the functions on my own, because he could see how interested I was. It didn't take long before the two of us saw the noise on the wires swamping the HART signal. But where was it coming from? We went around the plant with Alex in tow and the electrical foreman. We started at the signal "source" at an instrument called a DVC (digital valve controller), looked at the installation and followed all the wires back through a number of JBs (junction boxes). The DVC was not grounded on its case as it should have been. Filters inside the instrument are required to shunt any noise on the signal wires to ground through this case ground, so this was not happening. I checked the cable coming in to make sure the shield wire was not connected to metal because shield wires are only connected at one end and we saw they were connected at the DeltaV controller end. This was okay; the shield was not connected at this end. We followed the cables through the JB and saw that all the wiring, including the shield wires were connected properly, not shorted to ground, and continuing through the terminal blocks. The DVC that was not case-grounded had another problem in that the 4" aluminum cable tray that supported the signal cables was not bonded to the major 24" cable tray above the plant floor. In fact, none of this tray was bonded to each other. When discussing this with the electrical foreman he told us that Kenonic said this "wasn't necessary", despite reservations by Husky. Now, all the tall man-lift equipment that was returned to Edmonton had to be brought back in to get these trays bonded. Dave could only shake his head because by this time Kenonic had been bought by Emerson and the logo on Dave's coveralls said "Emerson". He was taking all the heat for something KENONIC had done. He was really, really pissed.

The problem boiled down to 16 feet of cables (about 100 pairs) between the marshalling cabinet and the DeltaV I/O blocks that was NOT shielded wire. The parallel run wires from different channels were creating magnetic fields, inducing voltage spikes in adjacent wires and thus in other channels. These were the spikes we saw with the oscilloscope. The 16 wires filled a 6" Panduit cable tray and was about 5" (well okay, 6 inches) in diameter.

We also spotted problems with the UPS used at site. Because the source was so far away from the power consumer, impedance effects and noise in the plant created high noise levels in the neutral-ground connection. These in turn found their way into the power supplies feeding the control system.

While Dave was watching the oscilloscope I shut one of the cabinet doors and he yelled, "Hey what did you do?" So I repeated it and once again he noticed a big spike on the channel we were monitoring. Every time the door opened or closed, it shut off the 4 foot fluorescent light in the cabinet. For those that don't know, fluorescent fixtures contain a transformer called a ballast that can generate huge voltage spikes from rising or falling magnetic fields. This is what was happening. The fix was to put in smaller fluorescents (with smaller ballasts, or electronic ballasting) but in Husky's case, they just put in a large explosion proof incandescent bulb. This fixed THAT problem, and also got me on mission to make sure no cabinets leaving Spartech would have 4 foot fluorescents. (Actually, Spartech always used the smaller fixtures as matter of course. The Husky cabinets had been made by another company. In other words, we were always doing it right in the first place!)

When we met with Husky personnel, Dave didn't mince words: the wiring would have to be replaced, isolation transformers would have to be installed for the UPS noise abatement, and all proper grounding practices would have to be followed for the DVC, including following the electrical code for bonding all the cable trays. The electrical foreman was adamant about how he had discussed many of these issues with Kenonic/Emerson but was told none of his concerns were warranted. In other words, Husky was aware of potential problems that Kenonic said was non-existent.

This was a weekend and for some strange reason the replacement instrument cable required was not available in Edmonton and had to be flown in from Ontario, along with the isolation transformers. Dave and I were still there when the electricians were swapping out instrument cables, a very laborious and potentially dangerous job, because the plant was running and could not shut down for this swap. Each pair of wires had to be identified from source to destination across two cabinets, tested to make sure it was the pair, then wires removed. This had to be coordinated with operations who had to put the loop in manual and swing bypass valves, to keep the loop working. Just pulling the wires back was a chore, because they turned and twisted amongst each other, creating a big rat's nest. Then, despite not having room in the Panduit cable tray, the electricians had to run the new cable, test it, then commission it. They had to do this for about 100 loops, and so had 4 electricians, two per side doing this, 24 hours/day for a couple days, until it was done.

The power conditioning with the iso-transformers was done about a week after which I went back to do some testing, using Dave's oscilloscope. (He trusted me with this $5000 piece of equipment). I confirmed that the problems we saw had now disappeared, and verified by the smooth running control system.

On the way back to Calgary after a week at site, Dave called Bill Elliott and told him that I could have solved these problems IF I had had an oscilloscope like his. And yes, I put Dave up to it because I believed a good piece of diagnostic equipment like this would be beneficial for MY work. (Not everyone did troubleshooting to the depth I did, so others didn't require the use of this tool). At first Bill was reluctant because it was an expensive piece of equipment and would it really be used? Unlike the real time corrosion meter I convinced Bill to buy me, the oscilloscope WAS used and easily paid back the cost of purchase. In order to convince Bill, Dave pulled out his "I'll tell Mike Begin and he'll buy this if I tell him to" trump card. Bill said okay.

When we returned to Calgary a few days after the weekend, Dave wanted to meet his old friend Mike Begin and we caught him just going into his office. Dave praised me up and down for not only my management of the issue, but my work at site, and the way I took care of him, a US citizen. He also said there were only 5 people around the world that he trusted to do the kind of troubleshooting (he used the word "work") we did, and I was one of them. Dave also mentioned how I should have an oscilloscope like his and Mike said there would be "no problem".

The electricians finished their work and wanted to get back to commissioning these loops and the rest of the plant, but were held back until Dave's full report was complete. This was a Kenonic request. Unfortunately there were 28 electricians not doing anything while the report was being written and there was possibility they would have to be laid off in the middle of a job. Not good. When Dave's report came out it was full of grammatical and spelling errors and Kenonic still did not want to let the report out. At issue of course were many items that Husky had to fix, as detailed in the report. It wasn't only the loop wiring, but also the isolation transformer, tray bonding, grounding of DVCs. The Husky commissioning manager, an electrician himself, called me and asked for the report but I told him I could not release it. (I had it and was doing some minor edits for Dave. Plus I was adding all the photos and drawings.) He persisted but I told him I could be fired if I released it. He said he would have to lay off people if he couldn't get them working on some of the fixes.

I made a judgment call and told him I would give him the report but he was not to share it with anyone , nor tell anyone I gave it. I did not tell my boss because he might have decided to tow the company line, although we (boss and I) might have been able to go to our VP and get it done. Regardless, I didn't want to take the chance of being turned down, so I gave the guy the report.

I only told my boss, Bill E about it after I retired, in an email I called "Sins of the Worker"