Contact Us




Change Locale

Writing the Ultimate Incident Response Playbook

Security | Posted on April 18, 2018 by Chris Martin

A cybersecurity incident hits your organization. What you do next makes the difference between containment and a catastrophe.

You may think your incident response (IR) strategy comes into play on Day 0. But without a playbook written and rehearsed in advance, your organization struggles to get back to “business as usual.”

As of last year, the Ponemon Institute pegs the average cost of a security incident at $5 million or $301 per employee. For larger enterprise organizations, these costs can scale to the astronomical. In 2017, several victims of infamous “mega-breaches” posted losses from $300 million to $600 million or more.

You need to know the skills required and recognize where you have gaps. Otherwise, you’re putting livelihoods at risk. Build your plan on the pillars of preparation, collaboration, and testing. This will help you avoid becoming another security statistic.

In our latest Spiceworks video meet-up, our panel discussed the “do’s and don’ts” of incident response. Our guests included Chris Martin, Cisco Practice Team Lead at Softchoice, Brad Garnett, Incident Response Team Lead at Cisco and Jonathan Meredith a member of the Spiceworks Community.

Watch it here:

Remember: Incident Response is a Team Sport

When an incident strikes, a successful containment involves people who wear many hats in their day-to-day lives. For this reason, Brad calls incident response “the ultimate team sport.”

Any response will involve general IT, security, and network engineering by default. But a successful plan also brings trusted, non-technical resources into the “war room.” These include personnel from human resources, communications, logistics or other mission-critical business units. This ensures employees know the plan and are ready to execute while IT tackles the technical stuff. The makeup of your team will vary with your organization’s size and requirements. Every response plan needs a “head coach” to provide executive sponsorship.

Team and organization often present a challenge. “It comes down to how we have a lot of lean IT shops. We can’t monitor 24/7,” says Chris. Your organization may be too small to have a dedicated security and threat monitoring facility, or security operations center (SOC), for monitoring security at an organizational level.

Your IT team may even lack the resources to keep a static incident response team in place full-time. In either case, tapping into cross-organizational expertise beats relying on IT alone.

Your Playbook is a Living, Breathing Document

Having your plan in place ahead of time is the most important aspect of the incident response. Keeping it flexible is just as important.

When an incident strikes, the foremost goal is to get back to business as soon as possible. Your plan will falter if the team lacks the context to mount a quick, effective response. This means documenting the full inventory of applications on your network. It also means developing “tribal knowledge,” or collective wisdom shared across the organization through the documentation on incident response planning.

Organizations often leave business continuity out of the incident response picture. For example: Is it worthwhile to get management systems back up and running? Or do we focus on critical manufacturing or shipping operations? Your plan needs to identify the highest priorities for your business model.

The last key component is testing. Testing through audits, tabletop exercises and penetration tests ensures your plan is realistic. It also means your IR strategy adapts as circumstances change.

It’s Not a Sprint, It’s a Marathon

Plan for your organization’s activities from Day 0 to three, six and twelve months down the line. You must be ready for the technical, legal and commercial implications of an incident.

Often, these do not emerge right away. “The worst thing I’ve seen is under-scoping an incident,” says Brad, “thinking you’ve removed an attacker from the environment, but they still have their malware deployed.”

Misinterpreting a threat or the stage of the attack cycle is a common error. Attackers aim to act on objectives while remaining undetected. Often, an organization will believe they’ve eliminated a threat when it’s still active in the network.

From a public-facing angle, messaging consistency is a challenge for post-breach organizations. The story often changes from day-to-day, from “service outage” to “data breach” to “millions of records affected.” This has a massive impact on public confidence in your brand. Clear and continuous communication throughout the process is key to successful incident containment.

Incident response requires many people working long hours to return users to critical systems. This, too, has a potential for negative impact. “It’s also important to remember the “people” component,” says Brad.

Your Team Can’t Always Run at Top Speed

In incident response, the greatest challenge is recognizing there’s an incident to respond to. “Gartner has the average time for detection of malware or ransomware at 100 days right now,” says Chris.

“It’s gotten better. But people are attacking us all the time.” For organizations fortunate enough to have a SOC or dedicated team in place, threat detection is easier to manage. But for smaller teams, it’s difficult to find time to sift through the noise to tell a true incident from a false alarm.

“You have to try to remove complexity from a complex situation as much as you can,” Brad explains. This means applying simple classification of threats as minor, major and catastrophic. It also means prioritizing defense of the “Crown jewels.” These include your servers, the infrastructure, and the critical data environment. Automation and machine learning help your team distinguish real incidents from mere events.

Know Your Limits

“I always tell folks, ‘it’s okay to ask for help.’” As Brad says, when it comes to building your IR plan, limited resources often mean limited capabilities. It’s important to recognize what you can handle and where you need help from third-party experts.

Retaining a third-party provider will help you build your IR playbook with expert help. Then, you’ll reinforce it through regular audits, tabletop, and penetration testing. Third parties can also assist you with selecting and integrating network security monitoring (NSM) and security information and event management (SIEM) tools.

When an incident occurs, a third-party will help with triage, forensics and onsite coordination. This ensures your organization gets back to business as usual as fast as possible without sidelining your in-house resources. Many cyber insurance providers also have in-house IR teams or contract them through other third parties. Often regulators include incident response measures as a component of audit compliance.

Balance is the key to deciding where you need help from the experts. Weigh the potential cost of an incident versus the price of third-party help.

Get Your Playbook into the Game

The statistics show you have a high chance of encountering a security incident. Last year, 54% of organizations experienced an attack that succeeded in compromising data or infrastructure.

But as Brad puts it, “if you train how you fight, you’ll fight how you train.” With a thorough, well-rehearsed response plan in place, you’ll be ready when things go wrong.

To recap, the keys to tackling incident response with confidence are:

  • Identify the multi-disciplinary skill sets needed to execute your plan
  • Ensure you have an in-depth, well-tested plan ready-to-go before an incident occurs
  • Plan for continuous communication in the short-, medium- and long-term
  • Focus your recovery on the most critical areas of your network
  • Learn to recognize the gaps in your defenses and when it’s time to ask for help

Have questions about incident response planning or contacting third-party providers? Leave your question in the comments section below or reach out to one of our reps.

Related Articles

Cloud | December 20, 2019 by Ryan Demelo

The stakes surrounding data security and risk mitigation rise with each passing year. Data breach costs continue to increase and potential threats grow more sophisticated.  According to IBM, the average total cost of a data breach – after accounting for remediation, reputational damage and regulatory issues – has reached $3.92 million. While smaller organizations may […]

Cloud | December 11, 2019 by Karly Pierce

IT organizations have seen too much of their time consumed by non-strategic tasks. This comes at great cost to their bottom lines and cloud transition strategies.  In fact, a 2018 Stripe survey found that between dealing with bad code, technical debt and various refactors or modifications, most of the typical developer workweek was wasted.   That […]

Cloud | November 28, 2019 by Ryan Demelo

Among the biggest obstacles to IT resilience is the “data dilemma.”  That data has become “the new oil” is a well-worn cliché by now. But clichés earn that status because they originate in the truth. And it’s true that today, data drives the decision-making that moves businesses forward. Protecting it is more important than ever. […]