“Sh_t happens” is what I was told when I inquired about a recent training accident at a State Fire Academy. Professional firefighters were running a scenario involving an aircraft prop. What they ended up doing was spaying jet fuel/water mixture onto the fire via an emergency shutdown system and hand lines, causing two minor injuries and currently over $600,000 in damages to equipment and environmental clean up. An independent investigation has learned that the training academy uses an oil/water separator so that the water and fuel can be reused. In 2010, the worker who had operated and maintained this oil/water separator since 1997 retired. Two new workers were hired; however, the report states…
The scope of this investigation did not include what role this change in personnel may have had in the events that occurred in October of 2013 but there appears to be a loss of institutional knowledge after XXXXXXXX left as well as a change in procedures. And this leads me to the learning opportunities.
There are AMPLE learning opportunities in this incident that can be applied to just about ANY hazardous activity. From missed communications to poorly designed emergency shutdown system, we can, AND SHOULD, all learn something from these series of mistakes. Those in process safety should take SPECIAL note to how these failures can find their way into our process safety management systems…
1) POOR DESIGN of the EMERGENCY SHUTDOWN SYSTEM
Report states:
In the control tower XXXXXXXX hit the emergency shutdown button. This action had no effect on halting fire growth which is to be expected since one of the features of the emergency shutdown system is to apply water to the props using small nozzles located on the props. Since this water system is the same as the hydrant system it would be expected that instead of water, fuel was being applied to the props.
Knowing that the fire training system was using an oil water separator and that this system could fail, the EMERGENCY SHUTDOWN WATER supply should have been from a SEPARATE and CLEAN source and used ONLY for the emergency situations and the fixed emergency shutdown system. This should have been the design REGARDLESS of how well the oil separator worked!
2) PROCEDURES and TRAINING
From 1997 until 2010 the maintenance supervisor operated and maintained the oil/water separator. Apparently he was successful in his methods; however, the report does not state this factual, it implies that after his retirement these duties were passed on to another individual. This new individual did things a bit differently and did not appear to have a full understanding as to why the previous caretaker did things the way he did. The report states the previous caretaker ran the system overnight, but came in routinely throughout the night to take samples and check on the system. The new individual ran the system over night, yet he did not monitor the system or take samples throughout the night. His reason, the report states…
XXXXXXXXX stated that he did not think it was important to check regularly since the academy no longer charged consortium partners for water usage, which was previous practice. The reason for a tight control on the processing of the wastewater is a safety issue and not a money issue.
3) Maintenance of Critical Equipment
It appears there were no procedures or established maintenance schedule on the oil/water separator. Given that the worker who operated/maintained the oil/water separator was involved with the initial development as well as the ongoing issues of maintaining the ARFF prop he understandably achieved a high level of organizational knowledge regarding operating the prop. He also described his routine of maintaining the fuel/water separator by checking the level in the two cisterns and taking a water sample from the clean water cistern. He stated he would do a visual inspection of the water sample and based on this sample he would make minor changes to the separator by adjusting the water supply or powder supply. He stated that he would do this every two hours that the separator was running. When asked if he had ever run the system all night he stated he had on occasion run the system all night and when he did he followed the same routine of checking every two hours. He stated that this fact could be checked because he submitted time cards showing the night visits. He also stated he kept a “safety” level of water (Safe Upper and Lower Limits!!!) available between the pump outlet and the top of the water in the clean water cistern thereby reducing the chance of fuel getting drawn into the hydrant system. He described his maintenance schedule for the ARFF prop as follows:
- Solenoid valves – Rebuilt every 6 months
- Cleaning rings – Serviced monthly
- Prop Nozzles – Cleaned every 6 months
The new worker charged with operating and maintaining the oil/water separator stated in the interview that he would turn on the system and check on it periodically throughout the day. He stated that he did not think it was important to check regularly since the academy no longer charged consortium partners for water usage which was previous practice. The reason for a tight control on the processing of the waste water is a safety issue and not a money issue. He stated that he sometimes ran the system through the night did no checks on the system throughout the night. When asked about the maintenance schedule of the ARFF prop he was uncertain on when the equipment was serviced. REMEMBER it was the props emergency shutdown system that failed miserably!!!!
4) COMMUNICATIONS and RESPONSE to NEARMISS(S)
October 14th – After an initial routine morning of training the ARFF prop experienced an ignition failure. The maintenance staff was notified and after attempts were made to fix the issue, which was unsuccessful, training was completed using an alternative ignition method. As part of the attempt to test the ability of the prop to self-ignite a quantity of fuel was flowed that was not ignited.
October 21st – After several training evolutions were held it was observed that there was a black sludge coming out of the hydrants being used by the crews to re-fill the crash trucks. According to the statement of a Captain with the POSFD and longtime state certified prop operator who was present, this occurrence had not been seen in the past and after a brief discussion with academy maintenance staff it was decided to halt training for the day and the water/sludge that was in the crash trucks was ordered to be processed back into the prop for recycling. Since the POSFD was going to conduct additional training in four days (October 25th) the Captain with the POSFD stressed to a maintenance tech of the academy maintenance staff that the hydrant system needed to be flushed prior to the training on the 10/25. The Captain talked to the maintenance tech and was assured that system would be flushed prior to that training.
Prior to the 10/25 training a phone call to ARFF Training Director was made to ensure that the hydrant system was flushed and everything was operational for the training on October 25th. The ARFF Training Director confirmed that the hydrant system was flushed. With that information it was decided to hold the training on 10/25. It is unknown who ARFF Training Director checked with to determine if the hydrants had been flushed. October 25th – At approximately 0730 two ARFF Training personnel, and a Captain with the POSFD entered the pump house building to turn on pumps. One of the ARFF Training personnel is part of the full time staff at the academy and one was the state ARFF instructor for the day. They found what they described as a large amount of fuel rich water on the floor of the building as well as a strong fuel smell inside the building. The water/fuel separator was still in operation. A ARFF Training member phoned the maintenance supervisor for the academy, and the supervisor advised the trainer how to shut down the separator. The maintenance supervisor made a phone call to the maintenance manager to discuss the situation. Although he is the maintenance supervisor he has been at the academy a limited amount of time and the maintenance manager has more experience with the prop. After the call supervisor advised the POSFD that the situation had happened before and training could continue. (The long time academy maintenance supervisor from 1997-2010 confirmed that similar situations of fuel splashing on the ground had occurred in the past) The decision was made by a Captain with the POSFD and the POSFD Training Chief that training would take place.
Around this time Captain with the POSFD was informed by POSFD crews that the hydrant water still had the same issues with discoloration and sludge that had shut down training on 10/21. The Captain with the POSFD discovered that the system had not been flushed and per his statement he gave the order to have the hydrant water flushed back into the prop for recycling. Instead of this being carried out the hydrants were flushed onto the drill ground which resulted in an environmental issue. Since the water available for the props and hydrants is a closed system this also reduced the water supply by several thousand gallons.
The investigation is not over, but it does not sound as if it will go deeper into the failures that led up to this incident. It appears that the POSFD did their checks and double checks, but even they failed in their decision to go forward with the training that day once they discovered fuel and water issues. The report states
It is unknown who XXXXXX checked with to determine if the hydrants had been flushed.
As I said this is an incident that most of us can take away several learning’s and apply them to our facilities and/or management systems. I particularly like how these failures line up with process safety management program elements.
CLICK HERE to view the preliminary report.