4.3. Noticing brittleness

From DARWIN
Jump to: navigation, search
Provide feedback on this topic
The link takes you to a secure and anonymous Google form.

Light bulb icon tips.svg Open the link in a new tab or window if you want to see the guideline content in parallel

The interventions proposed here aim to support organisations to identify sources of brittleness in order to invest in their correction.

Brittleness is experienced in situations of goal conflicts and trade-offs, or when there is a competition for resources and a need to establish priorities under time pressure. Other difficulties emerge when an organisation struggles to manage functional interdependencies between different parts of the same organisation, or when there is insufficient buffer capacity to provide additional resources. Noticing brittleness also means observing operational variability and comparing work-as-done with work-as-imagined, so to reveal how the system might be operating riskier than expected. In addition, brittleness manifests itself when the organisation is unable to learn from past events, such as near misses and accidents.


Implementation

Introduction

What is needed to notice brittleness:

  • Engage personnel at all levels of the organisation in understanding and noticing brittleness.
  • Create the conditions for personnel across the organisation to expose and discuss things that do or might not go well in crisis situations.
  • Implement recommended activities regularly to facilitate the personnel's capacity to notice and discuss brittleness.
  • Rely on external experts if resilience or safety managers familiar with notions of resilience are not available.
  • Select methods for the identification of possible sources of brittleness with the involvement of roles and actors at different levels in the organisation, making sure to account for an adequate diversity of perspectives. In order to achieve such diversity, combine individual interviews and workshop-based techniques, taking into account time constraints and availability of resources.
  • Plan the methods around triggering questions to be used as guide for the analysis (see examples of triggering questions below for the phases ‘Before’, ‘During’ and ‘After’ a crisis).
  • Use the outcome of your analysis to revise your internal guidelines or to create ad-hoc ones.

Note Brittleness is a useful concept because it can be easier to describe and notice when systems can break down. However, this focus on "what goes wrong" is complementary to the approach described in Identifying sources of resilience. It would actually be counter-productive to only focus on the negative aspects of systems and operations: it is fundamental to also understand the nature and characteristics of resilience and how it exists in the organisations considered.

ATM logo

Air Traffic Management implementation - Introduction

Including specific question "what is needed to notice brittleness" when applying - Toolkit:Systems Thinking for Safety/Principle 5. Resources and Constraints " Practical advice Consider the adequacy of resources. With field experts, consider how resources (staff, equipment, information, procedures) help or hinder the ability to meet demand, and identify where there is the opportunity for improvement. Consider the appropriateness of constraints. Consider the effects of constraints (human, procedural, equipment, organisational) on flow and system performance as a whole. Reflect on the implications for individuals and the system when people have to work around constraints in order to meet demand." (see Toolkit:Systems Thinking for Safety/Principle 5. Resources and Constraints)

Before a crisis

The assessment of potential sources of brittleness can be performed in two types of situations:(1) on a periodic basis, as part of established self-assessment activities; (2) In anticipation of specific events, to ensure resilience capabilities are in place. Relevant examples of the latter case include especially:

  1. Anticipated surge in demands (e.g., due to seasonal peak of activity, or to the approach of an identified threat)
  2. Relevant change brought to the system of interest (e.g. a new technology, a new policy, a new role being introduced).

In all of these cases, the analysis should aim to reveal and discuss potential issues that the system under investigation might experience when handling a crisis. For those organisations which have already identified a list of mitigation measures in case of accidents and crises (e.g., in classic risk management activities), the assessment of brittleness should also focus on understanding what might go wrong when applying the mitigation measures.

What is needed to notice brittleness Before a Crisis
For both the situations described above, noticing brittleness can be achieved through the organisation of a short workshop or focus-group for which:

  • participants are introduced to principles of resilience,
  • a facilitator leads a discussion about anticipated crisis situations and potential pitfalls,
  • the discussion is guided by the triggering questions presented below (the full set or a selection of them).

In such workshop or focus group, it is possible to use actual past events or fictional scenarios, to ground and direct discussions (see Practice 1 for an example related to surge in demand and Method 2 for an example associated to a technological change).

Triggering questions

Lack of Resources (human, technical, material)
  • Are there situations in which the resources we expect to have to respond to a crisis/emergency may not be available?
  • What can we put in place to relieve, lighten, moderate, reduce and decrease stress or load?
  • Where could we easily add extra capacity to remove stressors?

Lack of Information

  • Can we anticipate situations in which we will lack the necessary information to handle a certain event?
  • Do we have a protocol in place to gather the missing information?
  • Can we anticipate situations in which we may experience uncertainty based on the history of our operations?
  • Which processes and/or plans are insufficiently defined and may represent a source of uncertainty?

Goal Conflicts

  • What goal conflicts and trade-offs may arise or increase?
  • In such situations, will we be able to establish priorities?
  • Can some goals be temporarily relaxed or sacrificed to reduce the trade-offs?

Constraints and Bottlenecks

  • What constrains us in our ability to execute?
  • What conditions may push our system towards its limits?
  • Who will be most heavily loaded/stressed?
  • Can we anticipate situations in which our operations will be constrained by other organisations?
  • Can we anticipate situations in which our operations act as a constraint for other organisations managing a crisis?

Difficulties to adjust

  • Do we have the capacity to reallocate existing resources if needed. What may prevent us from reallocating them?
  • Do we have a policy that allows us to modify normal operations when needed?
  • Do we expect that major mismatches between official procedures and actual practices may occur?

Limits of mitigation plans

  • If we have safety/emergency plan, what can go wrong when applying the planned mitigation actions?
  • What could prevent us from applying some of the mitigation actions?

HC logo

Healthcare implementation - Before

Example of situations of relevance to healthcare:

  • Periodic assessment of potential sources of brittleness can be performed during the regular evaluations of the capability to answer the population health needs.
  • The anticipation of specific critical events is illustrated by the risks of influenza peaks. Every year influenza virus changes bringing about a crisis to cope with. The virus could be a novel one that needs to be covered by a new vaccine. Healthcare organisations therefore need to ensure that resilience capabilities are in place at all levels, specific prevention measures are taken, to contain the crisis and reduce risks.
  • Relevant changes to the system may be due to the introduction of a novel technology, for instance a new healthcare device or a new kind of vaccine.

In general, some common actions can be identified to assess potential sources of brittleness in situations that are relevant in a healthcare perspective:

  • carrying out a rapid assessment for a quick and efficient identification of sources of brittleness;
  • selecting indicators that could be predictive of a certain type of brittleness;
  • measuring the predictors identified to improve the preparedness.

See in addition the Healthcare Practices, Methods and Tools below.

During a crisis

During time-critical types of crisis, it may be difficult to use triggering questions as a checklist to be read step-by-step. However, it is important that all the professionals involved in the management of the crisis are fully aware of the topics addressed by the triggering questions and can consider such topics, even without reading them.

For crises that develop over longer time (e.g. Icelandic volcano eruption, or Ebola outbreak) it is possible to organise workshops or operative meetings to reflect with other colleagues on the possible sources of brittleness, and use the triggering questions to support the reflection. The same approach can be used during a drill or a simulation by a facilitator to guide the simulation and stimulate participants to notice brittleness.

Triggering questions

Lack of Resources (materials, information, personnel..)
  • Do we need additional resources (human, technical, material) to manage the event?
  • Are other part of our organisation able to renounce to some of their resources, to support us in managing the event?

Lack of information

  • Is there additional information available to address the crisis that we are not considering?
  • In case of lack of relevant information to handle the situation, can we put a protocol in place to gather the missing information?
  • Can we ask the advice of a colleague who is not involved in the crisis, to support us in correctly interpreting the situation?

Constraints and Bottlenecks

  • Are our operations during the crisis blocked by member of other organisations?
  • Are we hindering the operations of the members of other organisations during the crisis?

Difficulties to Adjust

  • Are we in a capacity to reconsider our priorities?
  • Can we delay the achievement of some goals, in favour of more urgent ones?
  • Can we consider deviations from normal procedures to manage the event?

Difficulties to learn from the crisis

  • Are we able to capture experiences from the crisis, in a format that support the dissemination of “lessons learned” inside the organisation
  • Will the format of such “lessons learned” encourage remedial actions by the management?

Difficulties to learn from previous events.

  • Are we adequately considering “lesson learned” from the past?

HC logo

Healthcare implementation - During

  • During time-critical type of crisis, health first responders organisations and local health units working on the territory are fully engaged on managing the emergency in the field. Methods and tools (i.e. triggering questions) to notice brittleness are hardly applicable. However, operational personnel need to be fully aware of them (e.g. by integrating them to their everyday practices at no-crisis time).
  • For crises developing over longer time, as in the case of infectious diseases, interdisciplinary work groups/ ad hoc crisis units are established according to the emergency to analyse the crisis situation, identify criticalities and set-up a response strategy. The generic triggering questions of this card - related to the during phase - could be used within these groups, to evidence possible sources of brittleness during the application of the mitigation actions.

At international level, in case of highly impacting infectious disease (i.e. Ebola), the European Centre for Disease Prevention and Control and WHO regularly perform risk assessments by means of which roadmaps are provided to countries. Roadmaps include indicators with the Countries’ capacity assessment to cope with the crisis.

After a crisis

Adverse events usually provide information that helps identify sources of brittleness (similarly to the way accidents and incidents can be used for safety-related purposes). However it should be emphasised that analyses must focus on processes, i.e. how operations were conducted, rather than on outcomes, i.e. what the consequences were.

What is needed to notice brittleness after a crisis . Depending on time of implementation, resources and objectives, organisations can:

  • Conduct quick assessments based on methods such as the focus groups described in Practice 1, for instance during debriefing sessions.
  • Conduct more in-depth analyses based on methods that focus on understanding operations in context (e.g., CTA – see Method 1). Data used in such analyses can come from data recorded during the crisis experienced, investigation reports or debriefings, whether it was an actual event or an exercise.
  • Across longer timeframes, assessments need to be conducted about how the organisation has reacted after crisis events, for instance whether it has prioritised and invested resources in the analysis and enhancement of resilience. Failures to do so correspond to forms of brittleness (see Method 3).

Triggering questions

Lack of Resources
  • Were our resources (human, equipment, material) adapted to the scale of the event?
  • Which were the missing resources, competences, strategies (if any)?

Lack of Information

  • Did we experience cases in which the information we had was insufficient to effectively handle the situation?
  • Were there difficulties to put in place protocols to gather the missing information?
  • Did the crisis we experienced reveal wrong assumptions we had about the nature of threats we are exposed to, and about our capacity to handle them?
  • Did the crisis we experienced challenge the plans we had established?

Goal Conflicts

  • What goal conflicts and trade-offs did we experience?
  • Were the goal conflicts unusual or unexpected?
  • Were we able to establish priorities?
  • Did we sacrifice any goal in a way that reduced our ability to adapt to certain circumstances

Constraints and Bottlenecks

  • What were the bottlenecks?
  • Where our operations dependent on others?
  • Were the operations of others' dependent on ours?
  • Was collaboration with other organisations effective? If not, which were the constraints?

Difficulties to adjust

  • Were we able to deploy or mobilise additional resources when needed? If not, what prevented us from doing so?
  • Were other parts of the organisation able to renounce to some of their resources when needed? If not, what prevented them from doing so?
  • Were we able to adjust goals and priorities when needed? If not, what prevented us from doing so?
  • Were we able to modify normal operations when needed.
  • Did we observer an excessive mismatch between official procedures and actual practices during operations.

Difficulties to learn from the crisis

  • Were we sufficiently able to capture experiences from the crisis and collect them in a format easy to share inside the organisation?
  • Were we sufficiently able to use these experiences to promote "after action review" inside the organisation?

Difficulties to learn from previous events

  • Have past, potentially similar, events in our own organisation sufficiently helped us being prepared for this crisis?
  • Have similar events in other organisations or domains sufficiently helped us being prepared for this crisis?

Limits of mitigation plans

  • If a safety/emergency plan was available, what went wrong when applying the planned mitigation actions?
  • Did we miss any mitigation action that would have been necessary?
  • What prevented us from applying some of the mitigation actions?
  • Did some mitigation actions result insufficient to handle the associated hazards?

HC logo

Healthcare implementation - After

Case studies are usually implemented to evaluate what went wrong when applying the mitigation measures.

A differential analysis of brittleness factors needs to be performed to identify: a) temporary factors to take into account in reviewing emergency plans; b) structural factors concerning institutions and policies to be recognised in order to start a change process that needs a wider temporary perspective.

In the case of Ebola, the analysis of data collected during the crisis and its management, allowed the review of the reference legal framework (i.e. International Health Regulation).

ATM logo

Air Traffic Management implementation - After

Triggering questions can be implemented as part of lessons learned within ATM. In Skybrary - ‘Lessons learned’ is validated working knowledge derived from successes or failures that, when reused, can significantly impact on organisation’s processes. (Secchi, 1999). The EUROCONTROL advisory material to regulations ESARR3 - Use of safety management systems by ATM service providers gives generic guidance on the processes to be established for lesson learned and dissemination that includes collecting lessons, dissemination and training" (see Lesson Dissemination)

Also airlines performance relies on reporting culture which encourages the pilot community to report high and low level incidents to enable the company to learn possible lessons from these incidents to help avoid recurrence. The preface to the Flybe Operations Manual reads: "All employees are urged to help provide the highest levels of safety in the industry, and so are encouraged to report any information which may affect flight or ground safety. To promote a free flow of information the Company will not normally take disciplinary action against any employee reporting an incident affecting safety. The only possible exception may be where someone has acted recklessly or maliciously or omitted to take action, in a way that is not in keeping with their training, responsibilities or experience. In such cases, the fact that a person has made a report will be taken into account in their favour. The Company will take very seriously, however, occasions where an incident is discovered that has not been reported. Not reporting anything which could affect flight or ground safety is considered serious misconduct." (see Skyway Spring 2013 - p44-45)



Contract.jpg
Expand.jpg

Understanding the context

Detailed objectives

As part of the assessment of resilience, noticing brittleness is the approach that aims at revealing and understanding deficiencies in and challenges to resilience in the system under consideration.

The opposite of a resilient system is a brittle one. Brittle systems break down especially in the face of surprising situations at the boundaries of what the system typically handles. In those situations, they are unable to accommodate even minor disturbances without ceasing to function. Examining the factors that undermine resilience is important in order to identify the most effective measures to actually enhance resilience and reduce brittleness. This assessment supports preparedness (e.g., related to planning or training) and the avoidance of situations that would result in potential harm or damage, for instance by anticipating potential bottlenecks in the response to a crisis situation.

Targeted actors

Managers are expected to implement the interventions in two ways:

  • setting up regular activities that lead to discussions about brittleness and its identification;
  • involving actors at all levels of the organisation, in particular team leaders and other operational personnel who are engaged in crisis management activities.

In addition, members of the organisation familiar with resilience notions (e.g., resilience or safety managers), possibly with the help of external experts, play a key role in conducting events, leading and moderating discussions about brittleness.

HC logo

Healthcare actors

Actors should be identified in the following areas:

  • scientific experts in the fields;
  • policy makers and regulation bodies at different levels: International Organisations (WHO, ECDC), Ministry of Health, Regions/ Counties;
  • operational institutions that operate on the territory (hospital, local health units, etc.).

ATM logo

Air Traffic Management actors

The roles and responsibilities of involved actors change according to the type of crisis and the related environment of operations. "Noticing brittleness" must encompass most of the activities of the organisation, at all levels starting from senior management to front line operators.

The actors involved are those listed below:

  • Air Navigation Service Providers (both civil and military)
  • Aircraft owners and operators
  • Aircraft manufacturers
  • Aviation regulatory authorities (National and International)
  • ATFCM (Air Traffic Flow and Capacity Management)
  • International aviation organisations (i.e. EUROCONTROL, ICAO, CANSO, etc)
  • Investigative agencies
  • Airport operator (if airports and/or ground operations are concerned by the crisis)
  • Firefighters (if airports and/or ground operations are concerned by the crisis)
  • Police (if airports and/or ground operations are concerned by the crisis)

Expected benefits

Understanding brittleness in the system allows organisations to address its sources and underlying factors and avoid situations that would result in potential harm or damage.

Relation to adaptive capacity

Noticing brittleness occurs through understanding when the system lacks adaptive capacity, or, more generally, faces challenges with adaptation. Through investigating brittleness, organisations can notice signs that indicate that their adaptive capacities are either eroding or ill-matched to the demands that are about to occur, allowing them to invest in order to adjust those capacities. This can happen before, during, or after a crisis event.

Relation to risk management

As part of the Resilience Engineering paradigm, noticing brittleness affords proactive safety management. Brittleness relates to how the system under investigation behaves under stress, more than to specific characteristics of the system or of threats. This approach contrasts with the traditional industrial safety paradigm of counting errors after accidents or mishaps and deriving specific risk-based interventions to reduce this count.

Illustration

A firefighting case and analysis illustrate the assessment of brittleness during operations:
Companies arrive on the fire scene and implement standard operating procedures for an active fire on the first floor of the building. The first ladder company initiates entry to the apartment on fire, while the second ladder gets to the second floor in order to search for potentially trapped victims (the ‘floor above the fire’ is an acknowledged hazardous position). In the meantime, engine companies stretch hose-lines but experience various difficulties delaying their actions, especially because they cannot achieve optimal positioning of their apparatus on a heavily trafficked street. While all units are operating, conditions are deteriorating in the absence of water being provisioned on the fire. The Incident Commander (IC) transmits a ‘all hands’ signal to the dispatcher, leading to the immediate assignment of additional companies. Almost at the same time, members operating above the fire transmit a ‘URGENT’ message over the radio. Although the IC tries to establish communication and get more information about the difficulties encountered, he does not have uncommitted companies to assist the members. Within less than a minute, a back-draft-type explosion occurs in the on fire apartment, engulfing the building’s staircase in flames and intense heat for several seconds, and erupting through the roof. As the members operating on the second floor had not been able to get access to the apartment there due to various difficulties, they lacked both a refuge area (apartment) and an egress route (staircase). The second ladder company was directly exposed to life-threatening conditions.

In spite of the negative outcome of the situation described, it illustrates a practice of noticing brittleness during the response to a crisis. The Incident Commander (IC) recognised and signalled a ‘all hands’ situation, in order to inform dispatchers that all companies were operating and to promptly request additional resources. ICs are particularly attentive to avoid risks of lacking capacity to respond to immediate demands as well as to new demands. The ‘all hands’ signal is a recognition that the situation is precarious (brittle) because operations are vulnerable to any additional demands that may occur.

HC logo

Healthcare illustration

Lack of overseeing capability as a source of brittleness.
A first responder organisation operating in Rome relies on the recruitment of associations of volunteers in crisis periods, whose accreditation is not subject to a proper assessment. During large scale emergencies, this organisation would integrate additional front-line staff, usually provided by externally accredited associations of volunteers. However, the regional institution responsible for releasing such accreditation lacked in control and monitoring capability — in particular check of personnel skills. Therefore, the leaders of the first responder organisation were aware that during large scale emergencies they had to deal with the additional burden of managing low competency staff, a condition that can contribute to operational brittleness. The situation highlighted a source of brittleness that is external to the concerned organisation and that, therefore, requires a system-level intervention to be addressed (DARWIN, 2016).

This example shows how a potential source of resilience becomes a source of brittleness. This because, in case of insufficient buffer capacity of the healthcare organisation, additional resources were provided, but not systematically monitored and assessed.

ATM logo

Air Traffic Management illustration

Several ATM illustrative cases and lessons learned are available on [1]. The website presents the most [2] and related accidents and serious incidents. For each incident/accident, a description, analysis of the event and main findings of the investigation are reported.

The more the Safety Culture is spread in ATM organisation, the more illustrative cases and lessons learned are available.

Implementation considerations

Challenges

  • Noticing brittleness requires that actors are familiarised with the principles of resilience. It is nonetheless a perspective and skill that can be learned (see Practice 1).
  • Enhancing resilience also requires understanding why things go right. Noticing brittleness is a useful way to anticipate, react to, and learn from challenging situations, but should not be the sole focus of a resilience assessment.
  • Because noticing brittleness focuses on how the system behaves under challenging situations, it is also different from understanding the threats or vulnerabilities of the system.

Implementation cost

Some of the methods described can be carried out in short amounts of time, e.g., through workshops or focus groups (e.g., Practice 1, Method 2). However, they require:

  • to be carried out by appropriately trained and knowledgeable people who can act as facilitators;
  • to involve a sufficient diversity of participants to yield the most information and best results.

Cognitive Task Analysis (see Method 1) is a well documented and practiced method coming from the field of human factors. However, it is a resource- and knowledge-demanding method, best carried out by experts in the field.

Noticing brittleness requires that actors are familiarised with the principles of resilience. Resources need to be anticipated in order to develop the associated perspective and skills (see Practice 1).

HC logo

Healthcare implementation considerations

Associated challenges

Healthcare is a complex adaptive system in which the non-linearity, the unpredictability and tensions are inherent. Within this complexity, people are at the same time source of brittleness and of flexibility and resilience for the system (Nemeth et al., 2008). Brittleness is a theoretical concept that is not necessarily part of the vocabulary of healthcare personnel, neither at managerial nor at operative level. Healthcare personnel need to familiarise with this concept and the principles of resilience. This process will support the personnel to move beyond the blame and shame cultures that have hampered the open flow of information and learning about vulnerabilities in healthcare (Nemeth et al., 2008).

Some other factors – internal to the healthcare domain - could hinder the application of the noticing brittleness principles within the contexts, among them (Vincent, 2006):

  • Hierarchical structure of the healthcare system. Hierarchies within professions tend to be rigid and relationships between professions and specialties complicated by issues of power and status;
  • Organisational culture and professional groups cultures;
  • National culture may be also influential (for example different approach to seniority, hierarchy, etc.);
  • Inability of the healthcare system to efficaciously communicate with the generic public in order to reduce sources of brittleness (for instance, an epidemic spreading due to a lack of vaccination).

Furthermore, the implementation of brittleness exercise requires an organisational context - and also the management support - that gives value to a proactive approach to crisis response (for instance by reporting errors and failures). Brittleness assessment requires an organisational context where personnel can express critical aspects (DARWIN, 2016).

Minimum viable solution

  • The triggering questions proposed are relevant to be considered during a workshop before and after the crisis, both to increase the awareness of potential sources of brittleness in a preparedeness perspective, and to explore the after crisis-phase. In case of crises that develop over longer time, the triggering questions can be also used in operative meetings in order to reflect on the effectiveness of the mitigation measures applied.

ATM logo

Air Traffic Management implementation considerations

The concept of "Brittleness" in ATM is strictly linked to the concept of Just Culture and Safety Culture which represent internal factors that could help in facilitating the identification of brittleness in each organisation. Notwithstanding the concept of “Just culture” [3] "has become better understood and accepted by people employed in the aviation industry, the need for a “just culture” is generally not understood by many legislators and therefore not accepted within their State judicial systems. This issue causes increased fear of sanctions against the reporter, particularly if partly or fully responsible for the reported occurrence. Furthermore, certain elements of the media may deal aggressively with apparent breaches of flight safety within certain airlines and ANSPs. These factors - punishing Air Traffic Controllers or pilots with fines or license suspension and a biased focus by some media on aviation safety issues – may have the cumulative effect of reducing the level of incident reporting and the sharing of safety information. This hinders safety improvement and as a cascading effect resilience. Concerns about possible misuse of information regarding “Brittleness”: One of the major problems with systematically collecting and analysing information is that such information can be a very powerful tool and, like any powerful tool, if used properly it will provide great benefit. However, it can also be used improperly and if that occurs considerable harm can be caused


Contract.jpg
Expand.jpg

Relevant material

Relevant Practices, Methods and Tools

Practices

  1. Brittleness assessment practices in industrial maintenance. Lay and Branlat (2014) describe how the necessary participants’ skills can be built through the use of study groups that aim at observing and discussion resilience and brittleness at play. A table in the document summarises examples of observations of brittleness at play. A workshop can be conducted prior to anticipated peak season (increased demands and risk of events) during which a facilitator helps participants notice brittleness. The document describes a set of guiding questions.
  2. “All hands” alarm in firefighting operations. The ‘all hands’ signal is used by an Incident Commander and by the dispatcher to quickly request additional resources when all companies on site are busy. It is a recognition that the situation is precarious (brittle) because operations are vulnerable to any additional demands that may occur. See illustration in this card and Woods and Branlat (2011).

Methods

All of the methods below are relevant to both Noticing brittleness and Identifying sources of resilience; these topics simply represent different focus of attention during the discussions. The corresponding cards can be used conjointly during the implementation of the methods.

  1. Cognitive Task Analysis (CTA) - TRL 9 - CTAs are typically based on different techniques that capture aspects of the situations under consideration. Analyses can occur after situations were experienced. CTAs can be conducted during training situations, which provide rich and more controlled situations during which crisis-relevant data can be captured more easily. See Crandall, Klein, and Hoffman (2006).
  2. Resilience Engineering assessment guidance - TRL 6 - The method was developed as a complement to a traditional safety assessment, in the context of technological changes in the Air Traffic Management domain. It focuses on understanding the variability the system (people and technology) needs to handle in everyday operations, how it currently adapts and handles the more challenging situations, and, finally, to anticipate how adaptation might be hindered or improved after the implementation of the new technological system. The method relies on short workshops/interviews led by a resilience assessment expert and involving relevant stakeholders such as operators (direct users of the system or operators they interact with), managers and designers of the technology.
  3. Q4 Framework - TRL 2 - Visualisation to assess how the organisation is prioritising and investing in safety, how it has reacted to adverse events. Assessment could also include measuring brittleness and evaluation of cost-effectiveness of countermeasures. See Woods, Herrera, Branlat and Woltjer (2013).

HC logo

Healthcare Practices, Methods and Tools

Practices

  • Periodic assessment of potential sources of brittleness: an example is provided by the monitoring activities periodically performed by the Italian Regions to evaluate their capability to answer the population health needs. This assessment system is based on indicators established at national level.
  • Anticipation of specific critical events: In Italy, the Ministry of Health performs a situation analysis before seasonal epidemic peaks and provides recommendations to all levels of the national health system to set up a response strategy. These recommendations include information on case definitions, analysis of data collected during the previous year, notifications, actions, institutional PoCs, reference laboratories.
  • Relevant changes to the system: in Italy, every time a new technology is introduced, a Health Technology Assessment (HTA) is performed and a national inquiry is provided for data analysis on existing similar technologies, possible relevant issues, costs and benefits ratio. HTA refers to the systematic evaluation of properties, effects, and/or impacts of health technology (i.e. medicines, medical devices, vaccines, procedures and systems developed to solve a health problem and improve quality of life). The assessment is conducted by interdisciplinary groups using explicit analytical frameworks, drawing on clinical, epidemiological, health economic and other information and methodologies. HTA is used to inform policy and decision-making in healthcare. More information about HTA available at: [4]
  • Pre-drill brittleness assessment. The brittleness assessments can increase ecological validity of drills if included in their planning phase. The brittleness assessment is an opportunity to really understand the capacities and challenges of responders during a particular scenario. A deep understanding of these factors could provide greater insights about real difficulties and challenges that can arise during an emergency (DARWIN, 2016).

Methods

  • Business Process Modeling (BPM) allows to represent processes of an organisation, so that they may be analysed and improved, in order to increase quality and reduce criticalities, also in terms of costs. Often, it supports change management programs (Scheuerlein et al., 2012).
  • Cognitive Work Analysis (CWA) and its modified form, Team CWA. Typically the CWA was used in healthcare as an approach to understand how people work in complex environments involving technology. It supports people making better and quicker decisions (Vicente, 1999).
  • Hazard Vulnerability Assessment (HVA) consists in: a) recognizing hazards that may affect demand for the health care system and infrastructures; b) identifying assets and resources of the system; c) assigning quantifiable value/ rank order and importance to those resources; d) identifying the vulnerabilities or potential threats to each resource; e) mitigating or eliminating the most serious vulnerabilities for the most valuable resources to improve the preparedness (Arboleda et al., 2009; Du et al., 2015).

ATM logo

Air Traffic Management Practices, Methods and Tools

EUROCONTROL has initiated Skybrary ([5]) which is an electronic repository of safety knowledge related to flight operations, air traffic management (ATM) and aviation safety in general. It is also a portal, a common entry point, that enables users to access the safety data made available on the websites of various aviation organisations - regulators, service providers, industry. With specific reference to Brittleness Skybrary provides a list of generic system thinking methods that can be used in ATM relevant for Brittleness. In the section called "Toolkit:Systems Thinking for Safety" it includes systems methods, observation, discussion, data and document review and survey methods, for more information: (see Toolkit:Systems Thinking for Safety - Principles in action)

  • Threat and Error Management (TEM) is an overarching safety concept regarding aviation operations and human performance. It has been developed as a product of collective aviation industry experience. (http://www.skybrary.aero/index.php/TEM)
  • Normal Operations Safety Survey (NOSS) is based on the TEM framework. It provides the organisation with a picture of the most pertinent threats and errors in a specific operation, how they are managed and how effectively any resulting undesired states are managed during normal ATC operations.
    An enhancement of NOSS considering brittleness and questions proposed within DRMG could be possible. More information about NOSS available at: [6]

NATS promotes several activities (i.e. Events, Seminars, workshops, training, etc) in order to improve the management of Emergency situations. Some of them are:

  • TRUCE (Training for Unusual Circumstances and Emergencies) which is a course for pilots that includes discussion and practical simulations to cover various scenarios that could happen in the air or on the ground – anything from severe weather to aircraft or passenger-related issues [ref. http://nats.aero/blog/2016/01/working-with-atcos-to-safely-handle-emergencies-a-pilots-perspective/]
  • STAC (Scenario Training for Aircrew and Controllers) which is a forum for pilots and controllers offering the possibility to jointly explore the risks and hazards inherent in emergency situations, and to promote mutual awareness of the protocols and options to be observed or considered.
    The workshops use actual emergency scenarios to help promote increased awareness by all participants of the separate and often competing demands on attention and responses in unusual and emergency situations.
    They are facilitated by NATS TRM Specialists and airline CRM instructors and follow structured discussions relating to:
    • Communication issues within the flight-deck and externally with ATC agencies
    • Sharing situation awareness in an emergency scenario within and between the two groups
    • Issues of overload and decision making for both parties
    • Handover issues between controllers, and sharing the situation within and between the aircraft crews
    • The use of SOPs, including emergency quick reference checklists by both groups


(see STAC Workshop Information)

References

  • Crandall, B., Klein, G. A., & Hoffman, R. R. (2006). Working minds : a practitioner’s guide to cognitive task analysis. Cambridge, MA: MIT Press.
  • Lay, E., & Branlat, M. (2014). Noticing Brittleness, Designing for Resilience. In C. P. Nemeth & E. Hollnagel (Eds.), Resilience Engineering in Practice: the Road to Resilience. Farnham, UK: Ashgate.
  • Woods, D. D., & Branlat, M. (2011). Basic Patterns in How Adaptive Systems Fail. In E. Hollnagel, J. Pariès, D. D. Woods, & J. Wreathall (Eds.), Resilience Engineering in Practice (pp. 127–144). Farnham, UK: Ashgate.
  • Woods, D. D., Herrera, I., Branlat, M., & Woltjer, R. (2013). Identifying Imbalances in a Portfolio of Safety Metrics: The Q4-Balance Framework for Economy-Safety Tradeoffs. In I. Herrera, J. M. Schraagen, J. Van der Vorm, & D. Woods (Eds.), Proceedings of the 5th Resilience Engineering Association Symposium (pp. 149–154). Soesterberg, NL: Resilience Engineering Association.

HC logo

Healthcare references

  • Arboleda, C. A., Abraham, D. M., Richard, J. P.P., & Lubitz, R. (2009). Vulnerability assessment of health care facilities during disaster events. Journal of Infrastructure System, 15, 3, 149-161.
  • DARWIN (2016). Deliverable D4.2. Initial evaluation of the guidelines. Available at: http://www.h2020darwin.eu/project-deliverables
  • Du, Y., Ding, Y., Li, Z., & Cao, G. (2015). The role of hazard vulnerability and prevention in China. Military Medical Research, 2, 27.
  • Nemeth, C., Wears, R., Woods, D., Hollnagel, E., & Cook, R. (2008). Minding the gaps: Creating Resilience in Health Care. In: Henriksen K, Battles JB, Keyes MA, et al., editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 3: Performance and Tools). Rockville (MD): Agency for Healthcare Research and Quality (US); 2008 Aug. Available from: https://www.ncbi.nlm.nih.gov/books/NBK43670/
  • Scheuerlein, H., Rauchfuss, F., Dittmar, Y., Molle, R., Lehmann, T., Pienkos, N., & Settmacher, U. (2012). New methods for clinical pathways-Business Process Modeling Notation (BPMN) and Tangible Business Process Modeling (t.BPM). Langenbeck’s Archives of Surgery, 397, 5, 755-61
  • Vincent, C. (2006). Patient Safety. West Sussex, UK: Wiley-Blackwell Wiley-Blackwell.
  • Vicente, K. J. (1999). Cognitive Work Analysis, Toward Safe, Productive, and Healthy Computer-Based Work. Mahwah, NJ: Lawrence

ATM logo

Air Traffic Management references

darwin

Terminology

  • Brittleness
    Brittleness describes how rapidly a system's performance declines when it nears and reaches its boundary conditions (Source: Woods, 2015).

  • Buffer capacity
    Size or kinds of disruptions the system can absorb or adapt to without a fundamental breakdown in performance. (adapted from Woods, 2006)

  • Functional interdependence
    Interrelationships (mutual dependence) between functions of a system.

  • Operational variability
    Variability and uncertainty are inherent in complex work such as disaster response; the conditions and challenges that manifest themselves are many and various. These can take the form of changes experienced in the daily life of operational units everywhere; or surprises that emerge from the interface of system elements that interact in unusual ways (e.g., hidden interactions); or challenges such as volcanic ash that defy prediction capabilities.

  • Work-as-done
    Work as done refers to he assumptions or expectations of what other people do [as part of their work] is called Work-as-Imagined (WAI), while that which people actually do [as part of their work] is called Work-as-Done (WAD) (Hollnagel, 2018, p. 17).

  • Work-as-imagined
    Work as imagined refers to the assumptions or expectations of what other people do [as part of their work] is called Work-as-Imagined (WAI), while that which people actually do [as part of their work] is called Work-as-Done (WAD). The term 'imagined' is not used in an uncomplimentary or negative sense but simply recognises that our descriptions of work will never completely correspond to work as it takes place in practice - as it is actually done (Source: Hollnagel, 2018, p. 17-18) and how work is being thought of either before it takes place when it is being planned or after it has taken place when the consequences are being evaluated (Source: Wears and Hollnagel, 2015).


Navigate in the DRMG