--- other: private: false title: Incident Management --- # Incident Management ## Definition Confirmed issues include malfunctions, performance degradation, service disruptions, or errors within Fluid Attacks products or components. These issues must have already affected or have the potential to impact a minimum of two users, regardless of whether they have been reported or not. This criterion applies universally, including internal issues affecting hackers or other team members. ## Incident Manager The Incident Manager is responsible for creating and managing incidents from the time of detection until the incident [postmortem](#postmortem) is published. ## Assigned developer The assigned developer is responsible for resolving the [issue](#issue). Their duty involves promptly addressing the case with the utmost priority, keeping the [Incident Desk](#incident-desk) updated on all progress and obstacles or any eventuality that arises during the process. Furthermore, the assigned developer provides all necessary information to the [Incident Manager](#incident-manager) to identify and adequately document the root cause and the development of the [postmortem](#postmortem). ## Detection This refers to when an external or internal user identifies an incident and subsequently reports it for publication and resolution. Incidents are usually reported through [help@fluidattacks.com](mailto:help@fluidattacks.com), but can also be notified through an [issue](#issue), internal chat, a direct email, or other means of contact. ## Issue When an incident is detected, the help area generates a detailed [issue](https://gitlab.com/fluidattacks/universe/-/issues) describing the problem. It is the responsibility of the [Incident Manager](#incident-manager) to assign a developer to resolve it. While most issues originate from the help area, they can also be created by other Fluid Attacks employees or external users. ## Publication After assigning a developer, the [Incident Manager](#incident-manager) must promptly access the incident panel to create the publication, considering the following fields: - **Incident name**: Reflects the nature of the problem clearly and concisely - **Incident status**: Upon publishing, the status should always be set to 'Identified' - **Message**: Description of what is being affected and how - **Workaround**: Description of how to avoid the issue through alternative methods, accompanied by a link to these instructions within the documentation - **Affected components**: Use checkboxes to select the affected components, which include: - **Platform (Integrates)**: Responsible for the Fluid Attacks application and its API - **Web (Airs)**: Contains the Fluid Attacks homepage and all information about Fluid Attacks and its products - **Docs (Docs)**: Contains Fluid Attacks documentation - **Agent (Forces)**: Responsible for the client-side part of the DevSecOps agent - **Cloning (Melts)**: Allows downloading the end users' code repositories and a few other utilities that Fluid Attacks hackers require - **Scanning (Machine)**: Security vulnerability detection tool that scans source code, infrastructure, and applications and reports its security problems - **Extensions (Retrieves)**: Visual Studio Code extension to visualize reported vulnerabilities in the [platform](https://app.fluidattacks.com) by pointing to its specific file and line of code - **Mailing**: Responsible for sending notifications to users of Fluid Attacks products - **Severity**: Options include major outage, partial outage, degraded performance, and operational - **Notifications**: Keep subscribers informed (ensure this checkbox remains selected) - **Long-running incident reminders**: Ensures admins are consistently updated on the continuous persistence of the incident (keep this checkbox selected and set reminders at one-hour intervals) Once the above fields are completed, click on the Create button. The incident will be immediately published on [status.fluidattacks.com](https://status.fluidattacks.com), and subscribers will be notified. ## Incident Desk The Incident Desk is an internal chat group consisting of at least four members with extensive knowledge of the business. Its purpose is to provide the developer tasked with incident remediation. Whenever a developer is assigned to resolve an incident, the [Incident Manager](#incident-manager) adds them to the group and posts a description of the problem, who the accountable developer is, the issue's URL, and the incident's URL. This initiates a discussion thread around the incident. The assigned developer is expected to dedicate 100% of their working time to resolving the incident. Additionally, they must promptly inform about any mitigations, solutions, obstacles, or other relevant updates regarding the case. Upon successful resolution of the incident, the Incident Manager removes the developer from the Incident Desk group. ## Follow-up It is the Incident Manager's responsibility to stay in contact with the developer responsible for the incident, tracking its progress, and keeping the Incident Desk updated on the incident's status or providing support if any obstacles arise. For long-running incident, the Incident Manager must constantly report the incident status, either through the crisis desk, responses to status page interval notifications, or directly through incident updates. Additionally, a brief update is provided every hour through the status page, ensuring clear and relevant information for understanding the incident's current status. ## Closure Once the incident is resolved, the [Incident Manager](#incident-manager) concludes the process by updating the incident status to 'Resolved' with a concise two-line message indicating the resolution and that the affected component is operating normally again. Clicking the 'Update' button publishes the new incident state on the status page, sends notifications to subscribers, and unlocks the option to write a [postmortem](#postmortem). ## Postmortem A public report detailing the incident is provided through four specific sections: Impact, Cause, Solution, and Conclusion. This report must be generated when the incident is closed ('Resolved') can be published within the incident using the **Write Postmortem** option in the incident management panel. The [Incident Manager](#incident-manager) is responsible for writing and publishing the postmortem. This involves conducting an investigative process around the incident, seeking support from the developer who resolved it and other stakeholders who can provide necessary information. The postmortem should be crafted for general understanding, avoiding technicalities, and striving for clarity and conciseness. Each of its sections is described below. ### Impact This section should be written with the following elements: - How many users were affected and how - Timestamp of the incident's timeline (on `UTC-5 [1] `) where: - `[1]` is the date of the merge request that caused the incident. - How the incident was discovered, whether proactively (by someone on the Fluid Attacks team) or reactively (by a customer) - Elapsed time of the incident's detection (`[1] Time to detect was `), where: - `[1]` is defined as **Time To Detect (TTD)**, that is, the elapsed time from the date the failure reaches production to the date it is reported through any of Fluid Attacks' support channels. TTD is always expressed with a precision of up to one decimal place. The unit of measurement for `` must be defined as follows: - `if elapsed_time < 1 hour, then X minutes`. - `if 1 day > elapsed_time > 1 hour, then X hours` - `if 1 month > elapsed_time > 1 day, then X days` - Elapsed time of the incident's detection (`[1] Time to fix was `) where: - `[1]` is defined as **Time To Fix (TTF)**, that is, the elapsed time from the date the failure is reported through any of our support channels to the date the incident is resolved. TTF is always expressed with a precision of up to one decimal place. The unit of measurement for `` must be defined as follows: - `if elapsed_time < 1 hour, then X minutes` - `if 1 day > elapsed_time > 1 hour, then X hours` - `if 1 month > elapsed_time > 1 day, then X days` - Who reported the incident and through which medium it was communicated - Elapsed time of the total exposure period of the incident (`[1] Window of Exposure `) where: - `[1]` is defined as Window of Exposure (WOE), that is, the elapsed time from the date the failure reaches production to the date the incident is resolved. WOE is always expressed with a precision of up to one decimal place. The unit of measurement for `` must be defined as follows: - `if elapsed_time < 1 hour, then X minutes` - `if 1 day > elapsed_time > 1 hour, then X hours` - `if 1 month > elapsed_time > 1 day, then X days` - It is essential to include the reference to the [issue](#issue) created to resolve it. ### Cause This section requires a thorough investigation to identify the root cause or causes behind the incident. At this point, it is essential to collaborate with the [developer](#assigned-developer) responsible for remediation and any other stakeholder who can provide the necessary information to identify and write up the cause successfully. By the end of this section, it should be clear what precisely the root cause of the incident was. For this section, it may be helpful to rely on the [5 Whys Technique](https://en.wikipedia.org/wiki/Five_whys). If the cause was introduced in a merge request, this URL must be linked as a reference (it is always the merge request URL and not the commit URL). ### Solution This section requires collaboration with the [developer](#assigned-developer) who remediated the incident and aims to clearly explain the remediation process. If code intervention was involved in resolving the incident, include the URL of the merge request (not the commit) that fixed the problem as a reference. ### Conclusion In this section, the [Incident Manager](#incident-manager) should document the lessons learned during the incident process. This includes understanding what architectural aspects allowed the problem to reach the user (Production environment), detailing the actions taken by the responsible developer to remediate the root cause, and finally, adding a [Taxonomy term](#taxonomy-term) or [Taxonomy tree](#taxonomy-tree). ### Taxonomy term These terms serve as tags to categorize various incidents that may occur. Their purpose is to succinctly and systematically summarize and classify the nature of a problem or error. This provides a quick and structured understanding why a particular issue reached the production environment. | Term | Definition | | ----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | COMMUNICATION_FAILURE | Occurs when it was an expected behavior, but it was not specified in the documentation, or there were insufficient instructions to perform the task correctly. | | DATA_QUALITY | Refers to a situation in which the data used within a process is deemed inadequate or insufficient in accuracy, completeness, or relevance, potentially leading to erroneous results or compromised outcomes. | | FAILED_LINTER | Occurs when the linter and its configured rules do not identify a syntax error. | | FAILED_MIGRATION | Denotes an error during a migration process, negatively impacting processes or data integrity. | | IMPOSSIBLE_TO_TEST | Occurs when the flow or nature of the operation does not allow for testing. | | INCOMPLETE_PERSPECTIVE | Refers to a situation where certain aspects were not considered during the planning or development of the functionality/process, resulting in failures or unexpected behavior. | | INFRASTRUCTURE_ERROR | Refers to a situation when misconfigurations within the infrastructure as code setup result in component failures, reduced performance, or even complete service interruptions. | | LACK_OF_TRACEABILITY | Occurs when insufficient loggers or tools are available to identify the root cause of the error. | | MISSING_ALERT | Occurs when technologies in the development environment and the continuous integration do not trigger an alert on the error, allowing it to go unnoticed in production. | | MISSING_TEST | Refers to the absence of specific tests, which can lead to undetected errors and result in product failures. | | NO_SPECIFIED | Refers to an error that, regarding its nature, has no reasonable or clear explanation. | | PERFORMANCE_DEGRADATION | Refers to a decrease in system performance that impacts the user experience or overall operational efficiency. | | ROTATION_FAILURE | Occurs when failures in the credential rotation process lead to specific components losing access to third-party services, compromising the operability, performance, or availability of Fluid Attacks' products. | | THIRD_PARTY_CHANGE | Refers to a situation wherein a third-party service or technology provider implements alterations in their infrastructure, potentially resulting in operational disruptions or unanticipated behaviors for their end-users. | | THIRD_PARTY_ERROR | Refers to a disruption caused by the failure or downtime of a third-party service provider, leading to unavailability or reduced functionality. | | UNHANDLED_EXCEPTION | Occurs when an exception happens without an associated error-handling mechanism. | ### Taxonomy tree A taxonomy tree is a hierarchical structure that describes a complex problem's root cause. It is employed when the incident arises from a series of interconnected errors or failures rather than having a clear cause. The hierarchical structure helps break down the problem into its parts and connections, making it easier to identify the main issue. The structure can be understood as follows: `Principal Failure < Primary Cause < Secondary Cause < N Cause` The structure can be interpreted as: `Failure Occurred Due to < Which Was Caused By < Which Was In Turn Caused By` Here is an example: `DATA_QUALITY < FAILED_MIGRATION` To ensure transparency and keep stakeholders informed, an [incidents page](https://status.fluidattacks.tech/history) is available. ## Other transparency measures - [Complaint management](https://help.fluidattacks.com/portal/en/kb/articles/complaint-management) - [Continuous Hacking to our technology](https://help.fluidattacks.com/portal/en/kb/articles/continuous-hacking-to-our-technology) - [Data leakage policy](https://help.fluidattacks.com/portal/en/kb/articles/data-leakage-policy) - [Ethics hotline](https://help.fluidattacks.com/portal/en/kb/articles/ethics-hotline) - [Help channel](https://help.fluidattacks.com/portal/en/kb/articles/help-channel) - [Information security responsibility](https://help.fluidattacks.com/portal/en/kb/articles/information-security-responsibility) - [Open source](https://help.fluidattacks.com/portal/en/kb/articles/open-source) - [Quality policy](https://help.fluidattacks.com/portal/en/kb/articles/quality-policy) - [Status page](https://help.fluidattacks.com/portal/en/kb/articles/status-page) - [Vulnerability releasing process](https://help.fluidattacks.com/portal/en/kb/articles/vulnerability-releasing-process)