Within the design context, determining which tools are most appropriate for risk assessment and risk characterisation will depend upon the nature of the design deficiency being managed including:
systems that fail probabilistically (mostly aircraft avionic systems and some mechanical systems)
aircraft structures and engines
systems that do not fail probabilistically (eg software)
systems with simple failure modes that cannot be determined probabilistically or deterministically
systems that are not included in the certification basis (eg hazardous materials and other non-specified design deficiencies).
Design deficiencies in systems that fail probabilistically. System safety programs (SSPs) provide a range of tools for analysing, assessing and (following measures to eliminate risk and where not practicable to do so, minimise risk) subsequently characterising risk of failures in systems that fail probabilistically.
The Hazard Risk Index (HRI) matrix, for example, can be a particularly useful tool for characterising safety risks when a design fails to meet the aircraft Certification Basis. The HRI approach is primarily applicable to systems whose failure can be probabilistically determined on a usage scale such as 'failures per flight hour'. Most often, these are the same systems that were designed to meet a Failure Probability Objective (FPO) by the OEM during initial aircraft design. These FPOs are extremely small, typically in the order of 10-6 to 10-9 failures per flight hour, depending on the type of aircraft, criticality of a particular function, and the requirements of the certificating authority. FPOs are used by designers to demonstrate that a particular upper level aircraft function will satisfy the overall safety goals of the aircraft, during both normal and degraded operations. Consequently, FPOs and the HRI can be a particularly useful tool for characterising the safety risk due to an observed in-service design deficiency.
A common mistake, however, is to use these tools for systems whose failure cannot be probabilistically modelled. Another common mistake is to use these same tools for systems that are used infrequently. For example, characterising risk on an annual basis for a system that is used either rarely (eg ejection seats, crash protection measures) or only a few times per year (eg flare systems, aerial refuelling) is likely to be grossly misleading. Characterising these risks through the HRI will almost inevitably, and misleadingly, result in an assessment of 'low' risk regardless of theseverity of the latent defect. Such systems pose hazards that are better assessed on a per usage basis (or perhaps the total number of uses through to aircraft Planned Withdrawal Date), rather than per flight hour.
Summarised, the system safety tools and framework are only appropriate for characterising risks for certain aircraft systems and functions. For the remainder, different approaches are needed to assess and analyse, characterise and communicate the risk.
Design deficiencies with Structure and Engines. Primary aircraft structure, including critical engine and airframe dynamic components, have a catastrophic failure condition by virtue of their role, and defects such as cracks, corrosion and dis-bonds cannot be easily or reliably statistically modelled. Progression to failure is also very sensitive to flight and ground usage, and as such, the risk cannot always be well characterised into the future. Given the complexity in making such judgements, and the potentially catastrophic consequences of incorrect assumptions and actions, DAVENG-DASA engagement is mandatory in all risk determinations and treatments for primary structure and critical engine and airframe dynamic components.
Design deficiencies in systems that do not fail probabilistically. This includes systems such as software or other system hazards that arise systematically from logic failures of a design, not component failures. These risks are best characterised through the consequences of their failure, or through a framework such as the MIL-STD-882E Software Control Category. Assistance is available from DAVCERT-DASA if required.
Design deficiencies in systems with simple failure modes that cannot be determined probabilistically or deterministically. Technologies such as oxygen delivery systems will reliably fulfil their role, but can quickly degrade due to a variety of external factors. This degradation is usually controlled through on-condition maintenance and inspection, or provision of emergency and backup systems. Similar to the above, HRI’s may be misleading if used to characterise the risk posed by a particular design deficiency. A qualitative assessment against the original design standard might be more appropriate.