An incident has happened. It's not the first time we’ve seen an incident like this one. Everyone agrees that there is a bad pattern repeating.There's some data that has convinced upper management to produce a mandate on the new way to do things. But that new mandate may not do much to address the underlying issues.
If you've been in a situation to notice this, you were likely very close to the issue itself: perhaps a responder or an operator.
You may have even wondered why no one just asked you about it. Post-incident review processes often overlook this step and, as a result, miss a lot of what could be learned — and a lot of what helps create safety and reliability. Incident analysis that involves the people closest to the work can help overcome that dynamic and provide more insight into what is actually happening.
This same dynamic is playing out in Australia in the sport of rock fishing: a sport where people fish from rocks jutting out into the ocean. This lets anglers reach deeper water — and much more fish — without having to use a boat.
The downside to this, at least as the media frames it, is that you can be washed into the ocean by "freak waves". With this in mind, and with a goal to reduce drowning deaths among rock fishers, the Australian government made a decision to mandate life jackets for rock fishers.
Luckily, a team — Peter Kamstra, Brian Cook, Eveline Rijksen, and Shane Daw — decided to investigate and analyze what was going on using a Naturalistic Decision Making (NDM) approach [1]. NDM is a way of studying how people make decisions in the real world — in contrast to lab-based studies which examine one specific aspect of decision making, usually using toy problems and non-experts in the domain (i.e. student volunteers).
What Kamstra et al. discovered is that mandating life jackets was unlikely to help reduce the number of drowning deaths. Even worse, the decision to require the use of life jackets may overshadow more productive strategies that could help fishers create safety. They discovered this by talking to the people closest to the incident, closest to the work itself.
The researchers met with the anglers, dividing them into groups based on their level of experience with the sport. 28 rock fishers with more than 3 years of fishing experience were considered the expert group.
Kamstra et al. worked with this expert group to develop a categorization of the reasons that rock fishers can get swept into the ocean. These are:
- Lack of experience
- Panicking and running from large waves
- Falling in while retrieving snagged lines or large fish
- After falling into the water, swimming directly back to the point of entry
- Unknown
This categorisation was strikingly different to the official data, where “unknown” was the most common category. The largest category of incident causes being "unknown" is a genuine problem if you're responsible for managing or otherwise influencing these outcomes. In the absence of data, those in authority are likely to fall back on their existing experience and mental models, which call for solutions like life jackets and lifeguards: solutions which are not specific to the rock fishing context.
Unlike swimmers, rock fishers move around a lot. They may be at one site one day, a different one the next. A given rock fishing site might have many rock fishers on a particular day, or one rock fisher, or none. This means that rock fishers need to manage risk and safety for themselves.
The researchers discovered that while anglers agreed that life jackets were very important for people who weren't strong swimmers, life jackets didn't address the most important way that rock fishers address safety: not ending up in the ocean in the first place, and, if they do find themselves in the water, behaving in ways that are less likely to result in injury.
The media generally asserted that the major safety hazard facing rock fishers was “freak waves”. Interestingly, according to the official National Coronial Information Systems (NCIS) data, "freak waves” are actually fairly low on the list of factors causing rock fishing accidents.
Nevertheless, a line of argument became prevalent that the rock fishers needed to be educated about the dangers of “freak waves”. For example, the NCIS data in one case says that “Weather data obtained from the Bureau of Meteorology showed that the water conditions at the time were dangerous.” [1]
On the surface, this seems reasonable: the water conditions were dangerous (in what way they were dangerous is unspecified), and the anglers are very close to the water. Case closed: they just need to know when to stay away from the dangerous water, right?
However, it isn’t quite so simple. Sam, a rock fisher with 20 years of experience, explained that “On some days, they'll send out a warning that there is a 4m swell and that it is dangerous to fish but it is only dangerous depending on where you go [fishing]. If the 4m swell is a southerly [south swell direction] and you are fishing in a southerly direction [facing the swell] of course it's dangerous, but if you go to the other side of the headland, on the north side, how is that massive southerly swell going to hit me? It's fine.” [1]
Something similar can happen in post-incident reports in software: "We went back over the logs and found [something]" with no explanation of how or even if that thing contributed.
Framing the problem of rock fishing safety as one of education or warning of wave hazards masks the strategies that rock fishers are already using in response to the risks they experience.
One example of such a strategy that Kamstra et al. discovered is that the rock fishers wear special cleats that help to keep them anchored to the rocks, reducing the likelihood of them entering the water. Another strategy is to avoid taking risks — such as going right to the edge of the rocks, or even into the water — in order to retrieve snagged equipment or large fish. Finally, the experienced fishers did not run from approaching large waves, knowing that doing so creates a worse risk: that of being hit from behind by a wave and swept away.
Expert fishers also behave in particular ways if someone does enter the water. When someone ends up in the water, inexperienced fishers nearby will usually go to the edge of the rocks and hold out their rods, encouraging the person in the water to grab on so they can be pulled back in. The problem with this is that because of the rocks and algae, it can be incredibly difficult to climb back onto the rock at the point of entry. It is also very dangerous to be in the water close to rocks: waves can smash you against the rocks or pull you under the water. The expert rock fishers advocate swimming away from the point of entry and finding somewhere safer to exit the water.
The official data did show some risks that the expert fishers did not discuss. While fishers identified the hazard of fishing alone, they did not identify the risk of leaving the group: the data shows that 25% of the drownings involved someone leaving the group to move to another part of the rock, or to grab their fish.
Experienced fishers identified fishing alone as a hazard: in groups, people can call out if they see a large wave approaching and assist each other in other ways. Despite this, the data shows that more experienced fishers drowned fishing by themselves than inexperienced ones. This points us to the possibility that expert fishers may become overconfident in their ability to handle risk. Inexperienced fishers may be more cautious and less likely to fish alone.
By listening to both the official-experts and the experiential-experts we obtain a much richer understanding of safety in rock fishing. If we dismissed either strand of expertise, we'd be missing a large part of the picture.
The same thing can happen in our world: folks who are near the work and thus near the hazard are already adapting to fill gaps and adjust to the environment. Without understanding what those adaptations are, we're unlikely to be successful in prescribing a solution.
Performing post-incident reviews that help surface the sort of information and insights that practitioners have can help guide decision making around what (if anything) really needs to be done after an incident. This is also why keeping incident retrospectives focused on learning — as opposed to action items — is more fruitful, shifting to generating solutions too early can easily obscure the things that responders are already doing.
All the researchers who studied the experiential-experts view of rock fishing safety needed was time and access. Time can be a very scarce resource, but spending time intentionally can pay off well into the future and help prevent a lot of time being wasted working on the wrong things.
In technology, we are fortunate that we can easily see the work in context, often using the same tools and environment that the work was done in. For example, with incidents, some of the tools that were used during the incident may be the same tools that investigators would use to ask you about it!
We can seek multiple, diverse perspectives, especially from different levels of the organization, including people closest to the incident (or the work in question), and even people who rely on the service.
We can ask questions that help elicit more information than just "why?" such as the questions that Sidney Dekker suggests in The Field Guide to Human Error [2]:
- Were you reminded of any previous experience?
- What were the limitations that you were facing?
- Did you discuss or mentally imagine a number of options or did you know straight away what to do?
- Were there conflicts or trade-offs to make between goals?
We work to elicit, understand, and preserve the experiences and understanding of those who were involved in incidents, rather than seeking one single perspective or "truth". This means we don't have to (and shouldn't!), correct or otherwise editorialize the stories of participants. This way we preserve the details that make these stories so valuable, and that provide us with the means to fill in the gaps in our understanding.
