Hogeschool van Amsterdam

Kenniscentrum Techniek

Do experts agree when assessing risks? An empirical study

Risk matrices have been widely used in the industry under the notion that risk is a product of likelihood by severity of the hazard or safety case under consideration. When reliable raw data are not available to feed mathematical models, experts are asked to state their estimations. This paper presents two studies conducted in a large European airline and partially regarded the weighting of 14 experienced pilots’ judgment though software, and the calculation of agreement amongst 10 accident investigators when asked to assess the worst outcome, most credible outcome and risk level for 12 real events. According to the results, only 4 out of the 14 pilots could be reliably used as experts, and low to moderate agreement amongst the accident investigators was observed.

Although quite alarming results, this paper does not aim at raising concerns about the skills of experienced employees; rather, we urge organizations to comprehend the distinction between experience and expertise, and focus on training their stuff in published expert judgment methods.

Gepubliceerd door  Kenniscentrum Techniek 23 mei 2016