The Federal Motor Carrier Safety Administration’s Safety Measurement System doesn’t capture enough safety-related data on carriers to establish a link between carriers’ SMS scores and their crash risk or to reliably compare carriers to one another, the Government Accountability Office concluded in a report released Feb. 3. In its long-awaited review of the Compliance, Safety, Accountability program, GAO said that even as an internal agency tool for targeting SMS, the program lacked the precision to be as effective as it should be, and it questioned both the way FMCSA presents SMS data to the public and FMCSA’s plan to use SMS scores to determine carriers’ fitness.
“Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores,” GAO found in the report, which was ordered by Congress in a recent appropriations act. “FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation.”
Even though FMCSA requires a minimum level of information to generate an SMS score, that minimum still doesn’t generate enough information to produce a reliable SMS score, GAO said. “FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.”
For SMS to be a more effective enforcement tool, GAO said FMCSA should score only carriers with more information than those scored today, allowing it to better identify high-risk carriers likely to be involved in crashes. To test the impact of this idea, instead of the few inspections now required to qualify for SMS scoring, GAO considered an alternative that eliminated the use of safety event groups and set at least 20 inspections or 20 vehicles, depending on the CSA BASIC, as the minimum threshold for a CSA score.
Using this approach, GAO said that 67.1% of high-risk carriers would have had a crash during the period it reviewed as opposed to 39% under FMCSA’s current approach. “This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA’s scarce resources.”
Carrier fitness and public awareness
If SMS isn’t precise enough to be used as an effective internal tool, GAO raised further questions about using them for safety fitness determinations as FMCSA currently plans. It said that “basing a carrier’s safety fitness determination on limited performance data may misrepresent the safety status of carriers, particularly those without sufficient data from which to reliably draw such a conclusion.”
GAO also questioned how FMCSA presented SMS data on individual carriers, although it stopped short of suggesting that the agency pull the data altogether. The watchdog agency noted a contradiction between FMCSA’s official disclaimer about SMS scores being intended for agency and law enforcement purposes and the agency’s own statement that SMS provides stakeholders with valuable information to make safety-based business decisions.
FMCSA itself has released a mobile phone application – SaferBus – to provide safety information, including SMS scores, to consumers selecting a bus company, GAO noted. Meanwhile, the Dept. of Defense has written SMS scores into its minimum safety criteria for selecting carriers of hazardous munitions, and multiple stakeholders have reported that entities such as insurers, freight shippers and brokers, and others use SMS scores. “Given such uses, it is important that any information about SMS scores make clear to users, including FMCSA, the purpose of the scores, their precision, and the context around how they are calculated.”
Industry reaction
Given that criticism of CSA and SMS has grown over the past three years, it’s not surprising that the trucking industry welcomed the GAO report.
“The GAO’s review of FMCSA’s Compliance, Safety, Accountability program was comprehensive, thoughtful and balanced,” said Bill Graves, president of the American Trucking Assns. “While ATA has long supported CSA’s objectives, we can’t help but agree with GAO’s findings that the scores produced by the program don’t present an accurate or precise assessment of the safety of many carriers.”
“Given GAO’s findings, FMCSA should remove all carriers’ scores from public view,” said Dave Osiecki, ATA executive vice president and chief of national advocacy. “Since scores are so often unreliable, third parties are prone to making erroneous judgments based on inaccurate data, an inequity that can only be solved in the near term by removing the scores from public view.” Osiecki also said that that “it would clearly be improper” for FMCSA to proceed with the rulemaking on carrier fitness determinations based on the data until FMCSA corrects problems that GAO identified.
The Alliance for Safe, Efficient, and Competitive Truck Transportation (ASECTT) – a group that has challenged FMCSA in court over CSA and SMS – said that “the GAO’s own words are a powerful endorsement of many of the arguments [we have] been making for years.” Several years ago, ASECTT’s legal challenges resulted in changes in how FMCSA presents SMS data on its website, and the coalition is still in litigation with FMCSA over SMS and CSA. GAO confirmed SMS’s bias against smaller carriers, ASECTT said.
ASECTT said it agrees with GAO’s recommendation against using SMS as it is today for determining carrier fitness, but the group “continues to also believe that until the system has been fully validated for individual carriers, SMS data should not be presented to the shipping public. The FMCSA should live up to its federal duty to determine which carriers are safe to operate on the highways and shut down the unsafe operators.”
Meanwhile, FMCSA tried to put GAO's findings in perspective, saying that it uses a "strategic, data-driven approach" to identifying and prioritizing high-risk carriers.
"While we are always looking for ways to improve our safety oversight methods, and will carefully consider the GAO’s latest recommendations, research shows that CSA is already more effective at identifying motor carriers with a greater risk of crashing than the system we replaced in 2010," FMCSA said. The agency added that under the Obama Administration it has tripled the number of unsafe bus and truck companies taken off the road.
