Most network defenders have never experienced a business-critical incident. An incident commander brings much-needed experience and knowledge to guide multi-functional teams through the process. Speed of recovery, evidence preservation, and security of the restoration process are often in conflict. Balancing them is critical to the long-term security resiliency of the company. High-touch incident responders like incident commanders require a specialized combination of technical, communication, and relationship skills.
The nightmare
Imagine being the CIO of a multi-national manufacturing company that relies heavily on IT services for production and shipping of your company’s products. Early one Saturday morning, your phone rings:
Your VP of Infrastructure woke you to announce that a significant portion of the company’s servers have been encrypted with ransomware. Critical manufacturing systems, ERP, financial systems, and file stores are affected. Employees cannot access the systems needed to do their jobs. The company will likely suffer massive revenue losses for every day of downtime, and customers may walk away if your actions do not quickly restore operations.
This is a scenario too often encountered in a security consultancy. The following thoughts might immediately go through your mind:
- I can’t even wrap my head around this situation! Where do I begin?
- How am I going to explain this disaster to executive management and the board? How could we have let this happen to us?
- We have never experienced anything like this before. How am I going to mobilize and rally my staff to respond to this?
- Will I have a job when this is over?
- I think I am going to be sick!
After calling your legal counsel for guidance and your cyber insurer to file a claim, you should call your incident response firm. If Secureworks® is your incident response firm, an incident commander would immediately be assigned to your case.
The incident commander’s role
The incident commander operates like the conductor of an orchestra, understanding everyone’s part and ensuring that each action occurs at precisely the right time. Accomplishing this balance requires relevant experience, a diverse background, and the ability to adapt and learn quickly. Secureworks incident commanders lead dozens of complex incident response engagements each year. They bring years of experience working very closely with customers and leading customer staff, consultants, and partners to orchestrate an appropriate response. The following are two challenges that incident commanders navigate during major incidents:
- Speed to recovery versus evidence preservation – During a ransomware attack, company leadership will exert a lot of pressure to return IT services to production. At the same time, internal and external legal counsel will want to know how the attack happened and if any data was stolen. Knowing how it happened is essential to remediation, and understanding if data exfiltration occurred informs legal reporting and decision-making. These goals often conflict with a speedy recovery. The incident commander must direct an expedited collection of evidence to conduct an investigation. This investigation assists in securing the environment, determining if data was stolen, and freeing storage for the recovery effort.
- Speed to recovery versus security of restoration – Also in conflict with a speedy recovery is the security of the recovery effort. Failing to properly secure the environment before rebuilding could result in re-compromise. If the attacker’s access is not severed, they often re-encrypt servers, eavesdrop on meetings, and send messages to employees and customers to disrupt and harass the victim. Containing the network and evicting the attacker are essential for a secure recovery. The following high-level activities are usually required for an effective eviction:
- Removing or rebuilding hosts that show evidence of compromise
- Implementing multi-factor authentication (MFA) on all remote access services
- Building and securing Tier 0 of the company’s Active Directory infrastructure
- Resetting Kerberos, administrator, service, and user accounts
- Patching critical exploitable vulnerabilities on internet-facing servers
Conclusion
Incident commanders require technical skills to understand what to do and how to execute prescribed actions in varying customer environments. They also need to be able to communicate risk at every level in the organization, from the administrator to the board of directors. Because of the incident commander’s experience and knowledge of these situations, they often become very close partners with the customer’s IT and security leaders. Relationship-building skills are essential to building trust.
There can be intense pressure on an incident commander to establish order during a time of chaos and to coordinate a response that helps a victim quickly restore operations. But they find great satisfaction in helping customers through some of the most difficult professional situations they may face.
Learn more about the Secureworks emergency incident response services.
