#DevOps methodology

#Incident Response

4 Agile Practices for Better Incident Response

Without question, the landscape of application development has shifted significantly in the last 10 to 15 years. During that time, software organizations adopted agile development practices that helped them deliver high-quality changes to their applications with greater velocity.

Modern development strategies aided in the production of higher quality applications by enabling organizations to accomplish a few key feats. For one, agile development practices helped to decrease the number of issues in each release. These practices also provided organizations with better frameworks for addressing problems that made their way into production. Finally, the iterative nature of agile organizations empowered development teams to continuously refine their processes in order to ensure greater reliability of future releases. Read on for a more in-depth analysis of modern development practices and their impact on application quality.

#1. Limiting the Number of Issues in Production

Modern development teams release production code with fewer issues than ever before. This is due to a combination of strategy and tooling that allows development teams to identify and resolve potential problems within their applications prior to production releases.

Automated Testing and Shifting to the Left

One of the biggest changes to the development cycle is in how early application features are tested. When waterfall releases ruled the day, application testing was done towards the end of the development process. Consequently, bugs would be discovered just before the release deadline, and developers would have to scramble to fix problems at the last minute.

This is not the case today. Effective development organizations have shifted testing to the left (meaning that they begin testing as early as possible) to ensure that features are constantly being validated and working as planned. This is accomplished with the help of automated test scripts that run as part of the CI/CD pipeline with each application commit and build.

The premise behind this change is simple: the earlier that developers know about the problems, the sooner they can begin to address them, and the easier they will be to fix. If a developer commits changes that negatively impact a feature covered by an organization’s automated testing strategy, he or she will know immediately and will likely be able to implement an efficient and permanent fix.

#2. Providing Better Frameworks for Addressing Problems and Limiting User Impact

Today’s development teams have several advantages that enable them to better deal with issues that are missed prior to production delivery. Application performance monitoring tools and modern deployment strategies help simplify incident identification and remediation in several effective ways.

Performance Monitoring Tooling

Monitoring tools, which are often equipped with automated data analysis and alert functionality, provide instant insights into the performance and reliability of application code in production. These insights, along with proper alert configurations, help to reduce the amount of time that it takes for development teams to discover the existence of problems within their applications (also known as mean time to acknowledgement, or MTTA). In addition, the detailed information that is recorded by monitoring software can provide a window into the situation. These details help simplify the process of remediation, which leads to a reduced mean time to resolve application issues (MTTR).

It’s easy to see how reducing MTTA and MTTR helps to limit the impact of these issues on end users. The sooner a defect is discovered and resolved, the less time it will spend in the production environment – and the less frustrated your consumers will be.

Reducing User Exposure to Defects with Effective Deployment Strategies

With shorter development cycles and more frequent releases, development organizations are forced to create deployment strategies that limit risk and downtime. These strategies allow development teams to limit the impact that a critical defect will have on end users.

Popular strategies include blue-green deployments and canary deployments. In the case of a blue-green deployment, two versions of the application are deployed at the same time (with blue representing the old version and green representing the new one). At first, all traffic is routed to the blue version. Later, the traffic is re-routed from the blue version to the parallel green version. The blue version remains up for a set amount of time after the switch, which enables a quick and painless rollback if any significant defects arise within the new deployment.

Similarly, two versions of an application are deployed in parallel in a canary deployment. In this case, however, a subset of the overall traffic is deployed immediately to the new (canary) version. If the new version proves to be stable for that subset of traffic, then more of the overall traffic is routed to it. If not, then all traffic is routed back to the old (blue) version, and the organization can rest easy knowing that a significant portion of their customer base was unaffected by the unstable release.

#3: Ensuring the Reliability of Future Releases

Today’s development practices enable development teams to continuously refine their approach to ensuring application quality. Through improved monitoring and analysis, better data is made available to those tasked with resolving defects. In other words, more insightful information is now available to developers and engineers when problems arise. This information enables them to implement complete and permanent fixes the first time, thereby preventing future recurrences. 

When a release cycle ends, the failures of the previous release should be properly noted in order to provide development teams with insights that will help them improve their chances of more stable releases down the line. These improvements often include additions to the automated testing strategy that increase test coverage where it was previously lacking, thus ensuring that formerly problematic features are properly validated for future iterations of the product.

#4. Shorter Cycles. Gradual Change. Higher Quality.

By now, all forward-thinking software organizations deploy changes to their applications on a continuous basis. This inherently increases application quality by reducing the risk that the current release will have significant issues. Having shorter development cycles means that fewer changes are being made in each release. This allows applications to mature gradually, which in turn enables organizations to maintain greater control over application quality. In other words, by releasing smaller changesets with greater frequency, development teams are less likely to introduce defects with far-reaching and catastrophic consequences.