logo

Incident Handling Foundations

Hello. The material we are going to cover this next hour is central to understanding the theory and practice of information security. This is a foundational course, developed for the SANS Security Essentials program. When you complete this course there will be a quiz available from the SANS web page to help reinforce the material and ensure your mastery of it.
Incident Handling Foundations Security Essentials The SANS Institute Information Assurance Foundations - SANS ©2001 1 Hello. The material we are going to cover this next hour is central to understanding the theory and practice of information security. This is a foundational course, developed for the SANS Security Essentials program. When you complete this course there will be a quiz available from the SANS web page to help reinforce the material and ensure your mastery of it. So many companies and people worry about their network or computer systems being compromised, but few address what they would do if they were compromised. If a company is connected to the Internet they will never be able to prevent all attacks. The motto I like to use is “prevention is ideal but detection is a must.” Being able to detect and react to an attack in a timely matter is key. This module covers the fundamentals of incident handling and shows what a company needs to do to properly address an incident. 4-1 Agenda • What is incident handling? • Why is it important? • What is an incident? • Fundamentals • The Six Step process Incident Handling Foundations - SANS ©2001 2 On the Agenda slide for this module we are going to address various aspects of incident handling. We are going to start with the basics and look at what incident handling is and what it means to your company. We are then going to cover why it is important and why a company needs to be concerned and have proper procedures for dealing with an incident. Being able to identify an incident in a timely manner and react is very important. Just as important is knowing what is not an incident so a company does not have to waste any of their time. The fundamentals of incident handling will also be covered along with the 6 step process for dealing with an incident. The six step process is taken from the “Incident Handling, Step-by-Step guide” published by the SANS Institute. For additional details on how to handle an incident, the Step-by-Step guide is recommended along with a full day course offered by the SANS Institute on Incident Handling. 4-2 Incident Handling • Incident Handling is an action plan for dealing with intrusions, cyber- theft, denial of service, fire, floods, and other security-related events • Having proper procedures in place so you know what to do when an incident occurs Incident Handling Foundations - SANS ©2001 3 As stated on the slide Incident Handling, incident handling is an action plan for dealing with intrusions, cyber-theft, denial of service, fire, floods, and other security-related events. This slide makes it clear that the scope of incident handling is greater than just intrusions, it covers insider crime, and intentional and unintentional events that cause a loss of availability. In fact, fires and floods are every bit as much an incident as a hacker attack. A lot of people only think an incident is a hacker attacker but it is a lot more than that. The other key point of the definition is the notion of action. Sitting there watching is not incident handling. You do not want to move too fast, but you do need to get in motion in an incident! Identifying an incident is important but you must act on that information to secure your systems in a timely manner. The best way to act on an incident and minimize your chance of a mistake is by having proper procedures in place. Well-documented procedures make sure that you know what to do when an incident occurs and minimizes the chances that you will forget something. 4-3 Why is it Important? • Sooner or later an incident is going to occur. Do you know what to do? • It is not a matter of “if” but “when” • Planning is everything • Similar to backups – You might not use it every day, but if a major problem occurs you are going to be glad that you did Incident Handling Foundations - SANS ©2001 4 It does not matter how big your company is or what type of business you are in, sooner or later you are going to have an incident. Companies of all sizes and types have had incidents and those that were not prepared and did not handle it correctly in some cases are no longer around to talk about it. When it comes to having to deal with an incident, it is not a matter of IF an incident is going to occur but WHEN is it going to occur. Another important point is the way some companies choose to deal with an incident is by ignoring it, but as you can image this is very risky to do. I bring this up because some companies I talk to say, “I have never had an incident in 2 years, why do I have to worry about it?” In this case the truth of the matter is they probably have had several incidents, but since they failed to detect them they took a stance of ignoring each incident. As we stated, this is very dangerous and it is only a matter of time until this catches up with you. One of the main reasons for a module on incident handling is, planning is everything. If you are prepared and know what to do, dealing with an incident can be fairly straightforward. On the other hand, if it catches you off-guard, there can be many sleepless nights. The key thing with incident handling is planning is very important but not to get discouraged if you do all of this planning and do not use it right away. Do not say, “I have done this planning and have not had an incident in 3 months.” Think of it as backups, you might not need to use them every day but if a problem ever occurs, and it will, you will be so glad that you did it. 4-4 What is an Incident? • An “incident” is an adverse event in an information system, and/or network, or the threat of the occurrence of such an event. • Examples of incidents include: – unauthorized use of another user’s account – unauthorized use of system privileges – execution of malicious code that destroys data • Incident implies harm, or the attempt to do harm – Incident handler reduces or minimizes harm Incident Handling Foundations - SANS ©2001 5 The slide “What is an Incident?” is for the purpose of defining what we mean when we use a word like incident or event. Incident, as we are using it, refers to harm or the significant threat of harm. There are several important points for an incident handler that flow from this definition. • Since we are dealing with harm or potential harm, our task is to limit the damage. We want to be careful to choose courses of action that do not cause further harm. • If the incident is not what is termed an act of God, your organization may well have a legal right. In either case, the incident handler should proceed in a manner that does not preclude using the evidence gathered in a court setting. Some examples of incidents are: •unauthorized use of another user’s account •unauthorized use of system privileges •execution of malicious code that destroys data Notice the key word in several of these examples, “unauthorized”. If a user openly and willingly gives their account information to another user with the intent that they will use the account to access the network, it is not an incident. It is only when someone uses that account without the permission of the owner in an unauthorized manner. 4-5 What is an Event? • An “event” is any observable occurrence in a system and/or network • Examples of events include: – the system boot sequence – a system crash – packet flooding within a network • These observable events compose an incident • All incidents are composed of events, but not all events are incidents Incident Handling Foundations - SANS ©2001 6 Since an incident is composed of events, lets look at what an event is. An event is something that happened in time that you either directly experienced or that you can show actually occurred. An event is something that you saw flash on the screen or that you heard. It can also be something that you know occurred because it was collected in a log or audit file. As part of the incident handling process, you should create forms that you use to record events. These forms can help you write down the information that should be documented. They can help you to be alert for the things you should be looking for. If you need a starting point, the SANS Incident Handling Step-by-Step guide has sample forms you can use and they are not copyrighted. Make all the copies you want and if you have suggestions for improvement, please email these to [email protected]. If there is any chance of the incident ending in a court case, having correlating information is better than a single source claiming the event happened. For instance, if two people saw a message flash on a screen, that will likely have more validity in court than if one person saw it. Further, attackers sometimes use tools to alter or delete their traces in log files. If you can produce two independent sources for the information, this has more validity. This is one reason we really push intrusion analysts to become familiar with a large number of log formats. The key thing to remember when looking at what an incident is, versus an event, is all incidents are composed of events but not all events are considered incidents. For example, an unauthorized logon is considered an incident, while an authorized logon is not considered an incident, yet both of these are network events. 4-6 Examples of an Incident • Which of the following is an incident: 1. An attacker running NetBIOS scans against a Unix system. 2. An attacker exploiting Sendmail on a Unix system. 3. A backup tape containing sensitive information is missing. Incident Handling Foundations - SANS ©2001 7 On the slide “Examples of an Incident,” lets look at a couple of examples to see what is and what is not an incident. The following are three events. Which ones would you consider to be an incident? 1) An attacker running NETBios scans against a Unix system. 2) An attacker exploiting Sendmail on a Unix system. 3) A backup tape containing sensitive information is missing. Actually the correct answer is all 3 would be considered incidents. In the first example, some might not consider it an incident because an attacker running an NT exploit against a Unix system would not be successful and therefore you do not have to worry about it. Remember our definition of what an incident is, a threat or occurrence of an event. So even though this attack was not successful, it should still be taken as a threat and the next time you might not be so lucky. Remember a lot of these attackers are running scripts against a wide range of systems, so in this case you were lucky because the attack was not successful but what if they ran this NT exploit against an NT system that you have or what if they come back and run a Unix exploit against your Unix system? Wouldn’t you rather deal with an incident when it is not successful than when it is? The second example is a successful attack against a Unix system and should be fairly obvious that this is an example of an incident. Someone successfully compromised your system without authorization or permission. The third example is also an incident because a tape containing all of your company’s data has been compromised. This has the same net impact as if someone broke into your system over the Internet and stole all of your information. Even though stealing a tape is not as glamorous as a hacking attack, it is still considered an incident and must be acted upon. Now that we have a good idea of what an incident is and looked at some examples, lets look at the incident handling process in the next slide. 4-7 Overview of the Incident Handling Process Incident Handling is similar to first aid. The caregiver tends to be under pressure and mistakes can be very costly. A simple, well- understood approach is best. Keep the six stages, (preparation, detection, containment, eradication, recovery, and follow-up) in mind. Use pre-designed forms, and call on others for help. Incident Handling Foundations - SANS ©2001 8 A good way to get an overview of the incident handling process is to compare it to first aid. In both cases, time is not on your side. You are under a lot of pressure and mistakes are very costly. To give you an example, my first real job out of college was working at a Defense Mapping Agency. At that time, they wanted to have an internal rescue squad. I volunteered and completed the training and was pretty excited for my first call. They gave me a pager to carry out so I could be notified when there was a problem. Remember this is back in the day when pagers and cell phones were not that popular, so to carry one around was very impressive. I could not wait for my pager to go off. Finally my pager went off and I was racing to the scene. I passed the chief of the squad and he reached out and grabbed me. He said, “Son, if you hurt someone else or yourself, when you get to the scene, you will not be any good to anyone. Now let’s walk down together”. Now, many years later, I am the grizzled old veteran and I want to pass this advice on to you. Law enforcement agents tell story after story of the well-meaning system administrator that ruined the evidence and usually just a couple minutes after the incident. You do need to act, but take time to think. There is a crucial point to this story. No one can run so fast they can outrun a computer with a 650Mhz Pentium III chip attached to a 100Mb Ethernet network. More importantly, when one is working as root, or administrator, or supervisor, there are many operations that do not have an “undo”. Several times we will draw the analogy between incident handling and first aid. It is a solid analogy. In some sense, first aid is a form of incident handling. So to review this slide the three things you have to remember when dealing with an incident is that it will be very stressful. Every minute will count and mistakes need to be minimized. Putting these three things together means you need to work but not so quick that you make matters worse. Remember the saying, “If you do not have enough time to do it correctly the first time, how will you have enough time to do it again?” 4-8 Incident Handling – 6 Steps • Preparation • Identification • Containment • Eradication • Recovery • Lessons Learned Incident Handling Foundations - SANS ©2001 9 On the slide “Incident Handling - 6 Steps,” we list the six steps in incident handling, which are preparation, identification, containment, eradication, recovery, and lessons learned. The steps serve you, the handler, as a compass or a roadmap, a way to keep in mind what you should be trying to do and the things you need to do next. The key thing with this list is, in order to be successful, you must follow all 6 steps. Some people think if they follow only some of the steps they will be in good shape, but in order to be successful at incident handling, this requires following all of the steps. Now each of the steps need to be customized to a particular company and the industry they work in and the following slides will help you do that. 4-9 Preparation • Planning is everything • Policy – Organizational approach – Inter-organization • Obtain management support • Select team members • Identify contacts in other organizations (legal, law enforcement) Incident Handling Foundations - SANS ©2001 10 When it comes to incident handling, planning is everything and preparation plays a key role. It is very important that you have a policy in place that covers an organization’s approach to dealing with an incident. Things that it needs to cover is whether a company is going to notify law enforcement agencies or run silent or whether a company is going to contain and clean an incident or watch and learn. One thing you really want to avoid is having an incident happen and finding yourself in a debate about whether to contain the incident and clean up, or to watch the attackers and try to gather more evidence. The time to make these (career affecting) decisions is before the incident, keeping senior management and your legal staff apprised. The policy should also contain what the policy is for inter organization approach and how a company works with other companies on an incident. It is very important that an incident handling team has management support and buy-in. The last thing a company wants is for senior management to be questioning or doubting the decisions that were made during an incident. Not everybody makes a good incident handler. There are some very smart people that I have worked with whose personalities do not lend themselves to being a good incident handler. People that like to work solo and be heroes usually do not make good team members. You want someone who works well in a team environment and thinks out solutions and not make rash decisions. 4 - 10 Preparation (2) • Update disaster recovery plan • Compensate team members • Provide checklists and procedures • Have emergency communications plan • Escrow passwords and encryption keys • Provide training • Have a jump bag with everything you need to handle an incident Incident Handling Foundations - SANS ©2001 11 During preparation, a company needs to make sure they update their organization’s disaster recovery plan to include computer incident handling. A large organization with over ten thousand computers is going to rack up some incidents. This can cause the incident handlers to burn out. It is kind of interesting; they tend to burn out just as they get really good. After training and seasoning, they do a bang up job on a couple hot problems and the next thing you know they are suffering from various stress effects. The solution seems to be a set of things, rewards and compensation that includes time off. This may run afoul of your organizational culture, but consider this. When do incidents occur? On Friday afternoons at 3:30 PM. Do the handlers and administrators go home and wait until Monday to start on the clean up? No, in almost every case they stay until the job is done. So we need to reward these people and let them get some rest. Our computing environments are complex, no one knows every variant of Unix and so forth. While we are trying to make sure you have a solid grounding in the basics of handling systems, memory fades over time. Having a checklist to refer to on how to bring down a system or back a system up can help prevent errors and reduce the stress on the handler. If they are following the checklist and it blows up in their face, it is not their fault. It is simply time to update the checklist. When I have been the system administrator of a production system, I have never been really comfortable making privileged passwords available to others. However, in an emergency, a handler may need access to critical systems. One organization has a policy where they are kept in sealed envelopes in locked containers. After several years of implementation they report that while sometimes cumbersome, this system has worked well for them. Note well that there is a two-fold responsibility here. The system administrators must make sure the envelopes are kept up-to-date. The handlers must make sure they tread lightly on the systems, keep the administrators up-to-date on any changes they made and above all, never use a privileged password unless they are qualified on that operating system. One thing that is nearly certain to make an incident worse is to have someone who has no clue what they are doing fumbling around as administrator or root. Not many of us can change the way our entire organization does business, but we can certainly be responsible for the way that we do business. Encourage people to write down critical passwords and encryption keys and store them safely so they can be accessed if required. As encryption becomes ever more prevalent, an organization must set policy as to who owns the secret keys and passphrases and under what circumstances they can be used and accessed. Also, being able to react when an incident occurs is very important. Therefore having a jump bag that contains everything you will need to handle an incident will enable you to react in a more timely fashion. 4 - 11 Identification • How do you identify an incident • Be willing to alert early but do not jump to a conclusion – “Boy that cried wolf” syndrome – Look at all of the facts • Notify correct people • Utilize help desk to track trouble tickets to track the problem Incident Handling Foundations - SANS ©2001 12 Bad things can happen when an unqualified, unauthorized person makes the call on an incident and is wrong. Fifty thousand dollars later, after three days off-line, you question that individual, “What were you thinking?” The answer is usually the same, “I, uh, thought it was nothing.” After a fire alarm is pulled, qualified firefighters who know the signs to look for come to the site and investigate. Only then, does the person in charge at the scene authorize re-entry into the building. This should be the paradigm we work under, be willing to alert early, have trained people look at the situation, and be able to stand down if nothing is wrong at a minimum of expense. Either way, make sure you have mechanisms in place to identify an incident. There is nothing wrong with alerting early if you maintain situational awareness and everyone understands that it might not be an incident. You want to avoid screaming it is an incident and an hour later say, “ Oh never mind.” If you do this several times, you will be a victim of the “boy that cried wolf” syndrome, which is when there is actually an incident, no one will believe you because you were wrong so many times before. Also when it comes to identifying an incident, do what you are good at and utilize others in the organization to help you. Why should an incident handling team go through all of the trouble of tracking issues when the help desk is setup and they do this on a regular basis? Let the help desk and others help you track issues and this will help make sure, in the end, that all of the issues have been resolved. 4 - 12 Signs of an Incident • IDS tool has an alert • Unexplained entries in a log file • Failed events, such as logon • Unexplained events (new accounts) • System reboots • Poor performance Incident Handling Foundations - SANS ©2001 13 There are many different signs of an incident, but this slide lists some of the more common ones. •IDS tool has an alert •Unexplained entries in a log file •Failed events, such as logon •Unexplained events (new accounts) •System reboots •Poor performance Some additional signs of an incident are: Suspicious entries in system or network accounting (e.g. a UNIX user obtains root access without going through the normal sequence necessary to obtain this access). Accounting discrepancies (e.g. someone notices an 18-minute gap in the accounting log in which no entries whatsoever appear). Unexplained, new files or unfamiliar file names. Unexplained attempts to write to system files or make changes. 4 - 13 Identification (2) • Assign a primary handler • Determine whether an event is an incident • Maintain a provable chain of custody • Make a clean backup of the system Incident Handling Foundations - SANS ©2001 14 When it comes to handling an incident, if one person isn’t in charge, no person is in charge. For smaller incidents, often of the “would you check this out” category, there is no need to send core incident handlers. It is recommended practice to have a core team of well-trained handlers, and also have incident handling skills and training as part of the job for security officers or system administrators. An organization that does this, benefits by having multiple levels of trained “firefighters”. However, in such a case, it is important to set up assignments in a way that encourages the system administrator to succeed. A non-full time handler should be given the assignment in a way that it is clear what is expected of them: the quality of their investigation, what documentation they should produce, and when it is due. It is also very important that they know who they can call if they feel they need additional guidance or support. Once you determine whether an event is actually an incident, it is important that you maintain a provable chain of custody. In order to do that and make sure you do not destroy any evidence, always make a clean binary backup of the system before you start making any modifications. 4 - 14 Containment • An incident handler should not make things worse • Secure the area • Make a backup • Possibly pull the system off the network • Change passwords Incident Handling Foundations - SANS ©2001 15 An incident handler should not make things worse, they should make things better. In containing an incident, the first thing that should be done is to secure the area. In doing so, a backup should be made of all infected systems and if the original hard drive can not be kept for evidence, multiple copies of the backups should be made. One should be kept for evidence and the other used to analyze the incident. At some point in the containment process a decision needs to be made of whether the systems should be pulled off of the network or if the entire network should be pulled from the Internet. Also, passwords should be changed to make sure a compromised account cannot be used as a re-entry point into the system. 4 - 15 Eradication • Must fix problem before putting it back online • Determine cause and symptom • Improve defenses • Perform vulnerability analysis Incident Handling Foundations - SANS ©2001 16 Before the system goes back online, an incident handler must make sure that they fix the problem or the vulnerability that the attacker used to compromise the system. Nuking the operating system from high orbit may be considered a shortcut in the handling process. While it is certainly true that total destruction of the contents of the disk will take care of any malevolent code, the opportunity for re-infection via the same channel after you reload the operating system still exists. There are tons of cases where handlers have taken systems down, reloaded the operating system, only to have the box compromised again a couple days later. The best course of action is to determine what the cause of the incident was, to find the vector of infection, and take action to prevent this from happening again. Once your system is hacked, the word gets out and every pea-brained hacker on the planet lines up to take another shot at you. It is not enough to simply recover the system; the security of the affected system(s) needs to be upgraded. If it is a production system, you may hear arguments that the organization cannot take the risk of modifying a production system. This is an important, and to some extent, a valid argument. The counter to it is that if the system was compromised, it must have some vulnerability. If we do not remove the vulnerability, the system may become compromised again. The simple trick of changing the name and IP address of the system can solve a lot of problems. If your organization has the time and resources, this can be a wonderful opportunity to play with a “honeypot”, a system that is designed to collect information about an attacker without yielding any useful data. One of the most interesting sources of software for building a honey pot is Fred Cohen’s deception toolkit, available from: http://all.net Vulnerability scanners such as the NAI Cybercop, Internet Security Scanner, Cisco’s Net Sonar, Nessus, Nmap and Saint can identify weaknesses in your organization’s internal network. The commercial software packages listed are somewhat expensive. If money is an issue, you can pay a consultant to run the software for you on a one time, or on a reoccurring basis, and provide a report. Nmap, a free tool, is becoming one of my favorite tools and I have also had good success with Saint, another free tool. After placing a suspect system on a small hub and doing the backup, I have sometimes found it helpful to run Nmap on the target computer from another system on the hub. Several times, this has given me insight into the potential problems I may be dealing with. Running a security scanner on the neighboring systems in a compromise can help you make sure you have full and complete eradication. If one system is compromised, there is every chance the number is actually two or more. 4 - 16 Recovery • Make sure you do not restore compromised code • Validate the system • Decide when to restore operations • Monitor the systems Incident Handling Foundations - SANS ©2001 17 How do you restore from backups and ensure you are not reloading compromised code? There is no easy solution, but you can use file integrity software in reverse. Use a software package like Tripwire on the compromised system and then do a restore from backups, possibly on a clean system. Run Tripwire again and compare the results. This can help you find the compromised code. For best results mount the disk that you are running Tripwire on from a system with a known-good operating system. This way kernel modules will fail to protect the compromised code. Remember again that after you have touched the machine, everything that breaks is your fault. Be sure to get the owner of the machine to sign that it is back in full operation. Make every effort to ensure the system is working properly before leaving the scene. The decision of when to put the system back into business has to be made by the system owner. As a handler, you can give them advice and try to be helpful, but this is their call. They are the ones that depend on this system. Needless to say, if the eradication was not complete or the infection vector was not closed off, the earlier you detect re-infection the better off everyone is. It is also politically better if the handlers detect the problem and show up to fix it, than if the problem comes to light because business operations are affected. This is a serious problem. Many times handlers take some shortcut along the way, or there is something you never discovered about the trust relationship and the problem comes back. 4 - 17 Follow-up • Develop a report – Try to get consensus • Conduct lessons learned meeting • Send recommendations to management • Follow-up meeting Incident Handling Foundations - SANS ©2001 18 The only one that really can or will write the report is the on-site handler. They submit the draft, but we should allow everyone involved to review the draft. If they have a strong disagreement about the facts of the matter, they can submit that and their statement can remain a part of the incident record. It is far better to find out that you have a lack of consensus before going to court than during court! After the report has been reviewed, schedule a lessons learned meeting. In general, the main purpose of such a meeting is to get consensus on the executive summary of the report. What is the most important thing for an executive summary to cover? How much the organization saved by having an effective incident handling procedure! During every incident, mistakes occur. We learn from these, improve our process for the future, and move on. Sometimes, we run into policy or other organizational problems that hinder bringing the incident to a close. We note these and submit them to management for their consideration. Follow-up meetings are never the most popular of events. Everyone is tired; they have been under stress. The system is now back in operation and the last thing anyone wants to do is have a meeting to rehash painful memories. However, this is a valuable tool for organizational improvement. This is the hardest time not to blame people. The focus should be on process improvement however. 4 - 18 Putting the Steps Together • Steps must be customized for your environment • Every incident is different • Planning is everything • Make things simple with checklists and tested procedures Incident Handling Foundations - SANS ©2001 19 We just covered the general 6 steps for handling an incident, but since every company and every incident is different, they must be customized for your particular environment. Remember planning is everything and by utilizing written checklists and tested procedures, it helps ensure that mistakes are minimized. 4 - 19 Key Mistakes in Incident Handling • Failure to report or ask for help • Incomplete/non-existent notes • Mishandling/destroying evidence • Failure to create working backups • Failure to contain or eradicate • Failure to prevent re-infection • Failure to apply lessons learned Incident Handling Foundations - SANS ©2001 20 On this slide, we have a list in probably time order, of the mistakes that are most likely to occur in the incident handling process. The key mistakes often made are the following: •Failure to report or ask for help •Incomplete/non-existent notes •Mishandling/destroying evidence •Failure to create working backups •Failure to contain or eradicate •Failure to prevent re-infection •Failure to apply lessons learned A good handler thinks a few steps ahead and tries to avoid the problems. Of course things happen. Don’t lose your cool if a re-infection occurs or anything like that. It doesn’t mean you are not a good handler, but if you can avoid mistakes, you might well be able to get home, get that shower, and jump into bed several hours earlier. 4 - 20
DMCA.com Protection Status Copyright by webtailieu.net