Black Hole of Cogema
Cogema was a company owned by the country of France, local companies and the government of Saskatchewan. They mined uranium in northern Saskatchewan where there were no roads, and refined it to a product called "yellow cake". The refining complex was huge as it typically is for any kind of mining, and this was just the refining side of things, not the mine itself. This site used Provox control system components and they had maybe 7-8 nodes that had controllers and two data highways. All communication was done through 75 ohm coax with a couple nodes fed by a fiber optic repeater.
Shortly before Christmas in 2004, I was called by the Saskatchewan rep who said the company was experiencing communication errors, even after replacing many components of the communication system and asked if I could go up to take a look. I rented some communication equipment from Norpac controls out of Vancouver so I could test the cables and fiber optic lines at the plant. I left Calgary on Dec 16, knowing that the earliest next plane back would be on the 21st of December.
I packed and flew to Saskatoon where I was picked up by the Saskatchewan rep the next morning at 6 AM. He took me to the hangar where a private plane was taking out the next shift of workers to this site. At a place called Points North in the middle of the wilderness of Northern Saskatchewan, we landed, then took a snowy cold 1 hour trip on a rickety old school bus out to site. All the windows were iced up and no one could see outside.
When I got to camp and they dropped us off at the main building to check in, I took a photograph of the large thermometer outside the building. It was set at -40C (or -40F, the only place they are equivalent)! It was going to be an experience for sure
I had to take a couple hours of indoctrination, mostly safety, but also methods for changing clothes. All personnel travelled from the camp where they ate and slept in a bus for maybe 1/2 mile trip to the refining section for the beginning and ends of their shifts, and they would enter the building on the "clean side" of the locker room. There they (we) would get out of their clean clothes, change to work clothes, then walk to an area where the REAL dirty, ie contaminated clothing was located. We had a box that contained all this stuff, hard hat, boots, coveralls, smocks, etc that would lift up to the ceiling in the dirty room via a steel cable. We'd drop it down, put our dirty stuff on and then go into the plant. At the end of the shift, we would repeat, taking a shower before we put back our regular clothes. Showering was mandatory. Our work clothes was given numbers and washed at site. No dirty clothes was allowed to be let off site and there was an entire group in the warehouse that checked clothing and equipment with Geiger counters.
My first day there was kind of a wash, where I did all the paperwork and orientation, met the guys and took a tour of the location. The main controller node where the communications and fiber optic equipment was located was on the 3rd floor of this one building and I had to climb those stairs maybe 20 times a day, with equipment. It was brutal.
I completed my task by checking all voltage and light levels (for fiber optic cables) to determine where the losses were and I then proceeded to ensure all connections were clean and tight, until I was able to match the spec put out by Fisher Controls. (At the time it was Fisher, now it is Emerson). I removed all contaminated silver pins in the copper comm cables, set up diagnostics so I could monitor events, checked grounding, power, cables, and all connectors. There were numerous places where there were elbows attached to controllers and these were "not supported" by Fisher because they added 2 more points of contact. I took them out and went straight in with the cables. I also evaluated the fiber optic transceivers and determined that one of them was faulty, which the company thought was a good one because it had come back from repair.
The room (photo at right) I was given was not too bad: it had two beds in it and so was quite large. It was also close to the bathroom, though it was a community bathroom and the room was located in the main building so I could walk to the kitchen/eating area in my slippers. The room had no television but I was close to the games room where there was one if I wanted to watch TV.
A year later the problems escalated at the plant to the point where areas were shutting down and ruining product. This was a much more serious matter and I had to hop on a plane to Saskatoon only 2 hours after I got the call.
This time, the room they gave me was absolute crap! I couldn't believe that they were housing a control systems specialist, one that was - hopefully - going to save their asses, in such a hell hole. My room was in a trailer that shook every-time someone entered the doors at either end. It was about the size of a standard home bathroom with one window, one light, no television or desk, not even a goddamn chair! I had to sit on the bed to read a book or lay down, those were my choices. When I went to the community washroom in my stocking feet, I found myself standing in some liquid around the toilet. Arrrggghh! The place was filthy too.
That night, about 10 PM, I get a knock on the door. It was the maintenance manager telling me the plant was down again and would I come out. I said, no, that they had gone this far with this problem, they would have to weather another 8 hours like this because I was not going to work all night. Geez, I had been up since 5 AM!
Later, I hear a key in my door, and the door opens up to a big 300 pound native guy who apologized profusely for having crashed my pad. The idiots at the camp office had given him the same key as mine.
This was only day one and I hadn't even started the hard work yet.
I gathered as much information about the system layout and equipment types as possible. Since I had an instrument/electrical guy as my shadow I used him as much as I could to get familiar with what they had. I also looked at diagnostics which they had never done, and quizzed everyone, including operators about the issues. While I was there I saw some of the symptoms.
My first plan of attack was to simplify the system. Some equipment was on the network but was shut off or was on but not being used. So I shut it down and took it off the system by taking the unit number off the network. I checked cabling and proper terminations and found some issues. Problems with I/O were fixed so that the errors they reported wouldn't mask the other problems. We checked power and grounding and traced every coax and/or fiber optic cable. I narrowed the problems down to one or two nodes and inspected every connection. Many were loose, only being hand tight if at all and this was a problem. Inside each screw connector I found little bits of metal that had worn off the threads. The pieces were bridging the insulator between the outside conductor and inside conductor and in my mind were reducing the impedance of the line, or maybe even shorting it out. We cleaned all these connections and the plant started to quiet down. I checked their spares and equipment they had recently swapped into place. I determined that some of the boards were defective and made sure the ones I installed worked properly by installing them for a day or two and monitoring the status. The comm stats in diagnostics gave me a wealth of information and I could see how these guys were running around like chickens with heads cut off if they weren't using the diagnostics. This feature was letting me know just how well I was fixing the problems. After I got most of the plant fixed, I did a complete power and ground audit on all nodes, all buildings.
After 8 days of grueling work, through one weekend, I was able to leave on the next plane out. A number of years later, a technician I had worked with said I had done a great job because all the problems recordema had did not resurface and they were now replacing it anyway with DeltaV system.