The Risk Management Program (RMP) is implemented by the Office of Emergency Management (OEM) in EPA’s Office of Solid Waste and Emergency Response (OSWER). EPA and state and local implementing agencies conduct inspections at RMP facilities to determine compliance with RMP regulatory requirements, but because resources for conducting inspections are limited, within the past few years EPA has begun focusing more attention on high-risk facilities for RMP inspections. OEM requested an outcome evaluation of the RMP focusing on the role of inspections. OEM and ESD agreed that prior to committing to a full evaluation, they would conduct an evaluation scoping project to:
1) assess whether current data are adequate to support an evaluation, and
2) identify additional data collection that might be required.
Here is what they came up with…
EPA could study associations between chemical accidents and RMP inspections with existing data:
RMP Info includes accident histories for facilities that have, and have not, been inspected. Linking the inspection data in ICIS to the accident data in RMP Info would enable EPA to compare accident histories across inspected and uninspected facilities. EPA could also examine the accident histories at inspected facilities before and after an RMP inspection was conducted. This approach would indicate the association, if any, between RMP inspections and chemical accidents. The ICIS inspection data and RMP accident data can be linked to 80 percent of RMP facilities at present; the other 20 percent of RMP facilities would be excluded from the analysis pending completion of EPA’s “bridge table” linking the two databases. The ICIS inspection data only go back to FY 2007, limiting the timeframe of the analysis.
Existing data are not adequate to assess changes in facility behavior from RMP inspections:
Neither database (RMP Info or ICIS) contains reliable indicators of changes in facility behavior. Limitations include data quality concerns regarding the self-reported RMP data, limited information about inspection results, and lack of information on changes in the behaviors that RMP inspections may influence (e.g., adoption of safer practices). The best way to assess changes in facility behavior is to conduct follow-up inspections – but this has not been done at most RMP facilities. Many facilities have never received an RMP inspection, and very few have received more than one RMP inspection. The lack of follow-up inspection data is a significant evaluability limitation.
Existing data are not adequate to assess the effects of changes in the RMP inspection strategy:
Assessing the effects of the strategy (to designate some facilities as “high risk” and to devote more inspection resources to high-risk facilities) on facility behavior is not possible because RMP Info and ICIS do not provide reliable indicators of changes in facility behavior. In addition, it would be premature to compare the accident histories of “high-risk” and other facilities given the short time that has passed since the strategy was adopted, and the corresponding lack of inspection and accident data for “high-risk” facilities vs. other facilities.
EPA has not systematically studied the effect of RMP inspections on facilities that are not inspected:
The literature suggests that inspections not only influence inspected facilities but can also influence uninspected facilities by increasing their awareness of regulatory requirements and the perceived threat of inspection. While EPA has studied this issue in general, it has not systematically documented examples that are specific to the Risk Management Program. Understanding the conditions under which RMP inspections can influence behavior at uninspected facilities would require further research, including interviews with facility managers and inspectors.
The evaluation team suggests that EPA takes steps to improve the quality, accessibility, and usefulness of the data:
• Review the RMP data, notify facilities about potential errors, and verify that facilities submit needed corrections.
• Expand the “bridge table” between RMP Info and ICIS to include more facilities.
• Review accident history and inspection data to establish chronology and account for duplicate records.
• Add new data fields to RMP Info and ICIS to reliably capture changes in facility behavior.
• Ensure those inspections are properly linked to resulting enforcement actions in ICIS.
• Continue to work with EPA’s Regional Offices to ensure the quality and consistency of inspection reports.
• Backfill the ICIS database using pre-FY 2007 inspection reports, to the extent feasible.
• Develop a clearer understanding of when and how different regions have implemented the “high-risk” strategy.
• Verify that EPA can link facility risk status (high-priority list) to inspection results and accident history.
• Study and document cases of general deterrence (effects of inspections on uninspected facilities) in the program.
• Consider implementing a pilot study to target a subset of facilities for more frequent inspections.
Here is the study report fs-eval-oswer-risk-management-program.pdf