It’s 2 a.m. at a major industrial facility, and about 20 yards from the rear perimeter, two figures dressed in full camouflage gear are slinking along the tree line just outside the plant fence. They’re wearing backpacks and carrying various paraphernalia, pausing occasionally to peer through night vision monoculars to scan the plant perimeter. Suddenly, a plant guard patrol vehicle rounds the corner of a building, its headlights shining in the direction of the pair. Both quickly drop, falling on their bellies in the mud and standing water from the previous night’s rain. The guard vehicle passes, and the pair remain undetected.Minutes later, the two figures reach a spot where trees and tall grass provide some cover; they pull out a laptop computer and attach an antenna, which they aim toward the plant campus. They remain in the area for an additional two hours, deploying their gear to scan for radio frequencies emanating from the plant, while observing guard patrol schedules and looking for holes in the fence or other perimeter breach points. At around 4 a.m., the pair end their surveillance and sneak away undetected. Only a few days later, the intelligence gathered during the nighttime surveillance by these two individuals—members of a four-man covert team—will be put to use, together with information from other daytime and nighttime reconnaissance visits. In broad daylight, the team will use what they’ve learned to send one of their members through a weak point in the perimeter fence and into the plant campus. Once inside, this individual, disguised as a contractor, will brazenly walk directly into the plant’s control room, where he will plug his laptop computer into the plant’s control network. Meanwhile, another of the team members will be simultaneously attempting to talk his way past the guard at the plant’s front gate. At the same time, the team’s other two members will be infiltrating a nearby plant office building. None of these covert activities will be discovered by plant security, though the second imposter will be held up by a suspicious front-gate guard.Covert operationsThese men could have been bad guys, intent on doing harm. Thankfully, however, they were only posing as bad guys—members of an industrial “Red Team” hired by the plant’s owner. The team’s mission: to covertly gain access to the plant’s critical control systems, using whatever means necessary, short of doing any harm. “A Red Team test is basically an all-out attempt to gain access to the client’s systems, whether it be completely through the network from a remote location, or by gaining physical access at one of their sites that is networked together,” explains Jonathan Pollet, one of the four Red Team members, and founder of PlantData Technologies Inc., a Houston-based industrial security consulting company that was acquired last year by Verano Inc., Mansfield, Mass. Verano recently changed its corporate name, and is now known as Industrial Defender Inc. The company specializes in cyber security for real-time control and SCADA environments (for supervisory control and data acquisition) in critical infrastructure industries. Clients include oil and gas, chemical, power, water and transportation companies. Pollet serves as vice president of professional services for Industrial Defender, and continues to head up the former
PlantData consulting operation, now known as Industrial Defender Consulting Services. Over the past six years, this organization has conducted more than 60 control system cyber security assessments for clients. These range from standard cyber vulnerability assessments to more extensive cyber penetration tests and all-out Red Team attacks. In most cases, only top personnel at a plant know when a Red Team test has been commissioned. Information technology (IT) and security staffs are not tipped off. “We carry letters from the top people in the [client] company with 24/7 phone numbers, so that if we do get caught, we don’t go to jail that night,” says Clint Bodungen, a security consultant who is a member of the Industrial Defender Red Team. A Red Team test can sometimes be mostly cyber-based. “If we can penetrate through the Internet, get through the corporate network and find the specific plant network that we’re looking for, then almost all of it is cyber,” says Ty Bodell, another of the Red Team members. But that scenario is rare, he adds; in most cases, a covert physical entry into the plant
is required. A major objective of the physical entry is to attach a wireless access device to the plant network. Once this is accomplished, the team can access the plant network wirelessly from outside the facility—parked on a nearby street or in a plant parking lot, for instance—taking all the time needed to probe the network. To be sure, the idea of a covert systems penetration attempt—even by friendly cyber security contractors—often makes process control companies nervous. Many first-time clients fear the test may adversely impact their systems. The Industrial Defender team attempts to allay those fears by pointing to the control systems expertise of many of its staff members, says Pollet, himself a former automation engineer for Chevron USA. “And we do have some rules of engagement that we sign off with the customer, which makes our process very safe.” To provide readers with a better awareness of the ways in which a determined group of motivated hackers, cyber terrorists or other criminals might attempt to gain access to their company’s critical control systems, Automation World interviewed Pollet and other Red Team members. We asked them to describe a Red Team test from start to finish, and to let us in on some of the tricks and techniques used to crack a company’s security defenses. Following is a report on what we learned.Who are they?When a client signs up for a Red Team test, the team is often provided with nothing more than the name of the client company. So the team’s first task is to discover all that it can about the customer. On the cyber side, this begins with research using publicly available Internet sources, says Bodell, who typically works with Patrick Turner, the fourth Red Team member, on most of the team’s cyber activities. “We’ll research what types of domain names the client has and the IP (Internet Protocol) address ranges they have,” says Bodell. “Typically, we’ll Google for e-mail addresses of people from the company in relevant positions like the IT and process control groups.” This information will be passed along for use in “social engineering” activities by the physical penetration team later, who may benefit by “name dropping” as they try to talk their way into a plant. Online sources of information that often prove useful include company press releases, mailing lists and Internet forums, Bodell observes. As part of their research, Bodell and Turner also typically use Google Earth software to obtain satellite pictures of the target plant, as an aid to determining the best locations for physical surveillance and eventual surreptitious entry. Also during the reconnaissance phase, the pair begin Internet-based scanning of computer ports discovered at the target company or plant, in an effort to gain information on systems and services, and to assess vulnerabilities. Taking what Turner calls a “slow and low” approach, they scan only a few ports at a time, as way to avoid detection by the target company’s IT security group.Surveillance While the cyber discovery work is going on, the team is also performing both nighttime and daytime physical surveillance of the target plant. Bodell often teams with Bodungen for this activity; it was a nighttime reconnaissance mission by this pair that is described in the opening paragraphs of this story.The team is typically able to gather as much information as it needs during four trips to a target plant—twice at night and twice during the day, says Bodungen. During the nighttime visits, the team will generally prowl the plant perimeter, looking for potential entry routes into the plant, and scanning for in-plant wireless frequencies that may be leaking from the plant, as well as for frequencies used by plant guards. “We’re also looking for key buildings that are either well-lit or high traffic, and have lots of wires running to them or maybe lots of fans on the back, indicating a data center,” says Bodungen. “We’ll pass that information on to the penetration team, because they may be locations of network access, which is what we’re targeting.” During nighttime visits, the pair typically tries not to be seen. But daytime surveillance techniques involve what Bodungen calls “hiding in plain sight.” For example, “we may get a plain white truck out there near the plant and put orange cones around us and act like we’re doing surveying work,” says Bodungen. Typically, Bodell and Bodungen will also spend time sitting in a car in a plant parking lot, “just acting like we’re supposed to be there,” he adds. The pair change locations frequently and use different vehicles each time they return. During the daytime surveillance, the pair take numerous photos, including close-ups of employee and contractor badges, as an aid to making their own fake badges later. “As people walk past our car in the parking lot, we inconspicuously snap pictures,” says Bodungen. They also take special note of the color and other details of contractor uniforms. “Typically, employees know each other. Contractors are in and out, so we can usually slip in and out of a plant a lot easier [disguised] as a contractor,” Bodungen explains.When it is time to enter the plant, the team assesses its options based on the intelligence it has gathered. In most cases, a daytime entry carries less risk. “Usually at nighttime, people know we’re not supposed to be there, so if they see us, we’re caught,” Bodungen says. “But during the daytime, it doesn’t matter if they see us, as long as we look the part.”The team typically attempts two entries simultaneously at different places by different team members, each carrying wireless access points to be planted inside. This increases the chances of success. “If one of us gets caught, the guards would go on alert, and we wouldn’t have a second chance,” Bodell explains. “So we’ll strike at the same time.”Pollet typically attempts one of those entries, most often disguised as a contractor. “Sometimes, we actually go through the front gate with a [fake] work order that tells people we’re supposed to be there,” says Pollet. But in the case described in the opening scene, the team decided to send Pollet in through a rear area of the plant where surveillance had revealed an easy entrance point through the perimeter. We’ll pick up more of that story here:PenetrationWearing a hard hat with safety glasses, and a fake contractor badge and uniform, Pollet carries a duffle bag containing a wireless access point, hubs, switches, a laptop computer and various gear to connect the computer to the plant network. Once inside the plant environment, he walks around freely, nodding to others, who typically smile and nod back. “It’s a Friday, so most of these guys have got one thing in mind—heading out for the weekend,” he surmises.In search of a control room, he tries the doors on several likely looking buildings. “If they’re open, I walk right in,” Pollet says. He eventually hits pay dirt; as he enters one building, he sees two men standing outside the door to what is obviously a control room containing SCADA terminals and other equipment. He walks up to the pair and begins making small talk: “Man, I’m glad it’s Friday…” After they chat for a few minutes, Pollet saunters into the control room, goes directly to a jack and begins plugging in his laptop. A few minutes later, the two men enter the room, pull up chairs and continue talking. They don’t ask Pollet what he is doing. “While I’m talking with them, I’m basically scanning their network,” says Pollet. “I’ve got various programs running in the background, bringing back the names of their computers, their IP addresses, operating systems and the kinds of applications they’re running.”When he is left alone, Pollet takes pictures of the control room equipment, and also snaps pictures of himself sitting at a control station, to be used in the wrap-up report to the client. He also attaches his wireless access device, hiding it in a bundle of wires in a SCADA console cabinet. Uh ohWhile this is going on, Bodungen, also dressed as a contractor, is attempting to join Pollet by entering through the plant’s front gate. He’s got a fake work order. But he runs into trouble. The front gate guard can’t find the contractor name on his list. And the more Bodungen tries to convince the guard that he is legitimate, the more suspicious the guard becomes.At this point, Bodungen goes to his exit plan; he makes a phone call to Bodell and Turner, who are in another location. Bodungen fakes a conversation on the cell phone, then tells the guard that he was mistakenly dropped at the wrong work site. Someone will come by shortly to pick him up, he says. This immediately relaxes the guard. “His suspicion goes away, because now he has a reason for me to be there,” Bodungen relates. “So I figure that I can use the opportunity to gather some more recon close up.”While “waiting for his ride,” Bodungen chats with the guard, who is now comfortable enough to leave Bodungen alone in the guard shack on several occasions. “I could have grabbed a handful of badges, because they were just hanging there, or I could have sneaked out the back and gone on into the plant,” says Bodungen. He does neither. But he is there long enough to observe the strict exit procedures practiced by the guard; Bodungen calls Pollet and advises him not to try leaving by the front gate—the original plan—but to instead exit the same way he entered, through the rear perimeter. Meanwhile, Bodell and Turner are trying a different penetration approach. During surveillance, the team had identified an office building that is not within the plant fence boundaries, but has cables running from it into the plant environment.
They suspect the building may be on the plant network. Dressed in office casual clothing, they enter through the front door, walk past an unmanned security desk, ignoring the sign-in sheet, and proceed unchallenged into the building. “We have our laptops out, with antennas sticking out, looking around as though we’re doing a wireless signals survey,” says Bodell. “But we never have to use our story, because nobody talks to us.”The pair locate a printer room, where they attach and conceal a wireless access point, then quickly leave the building. Back in their car in a nearby lot, they successfully connect to the access point, and find themselves on the plant network. They call Pollet, who is still in the plant control room, and tell him to retrieve his access point and get out. “Since we had an access point working outside, we didn’t need to risk having to do the more difficult penetration back into the plant later to retrieve an access point there,” Bodell explains. Bodell and Turner drive to the plant front gate and pick up Bodungen, then pick up Pollet exiting the plant at the designated spot, and the team goes home for the day. From here on, the physical work is done, and rest of the Red Team attack is cyber penetration testing. With an access point in place, team members are free to come back, park on a nearby street or a plant parking lot, and take their time probing the network. “We usually choose a time that’s late at night on the weekend, or maybe at 5 p.m. on a Friday so the cars are still there and we don’t look suspicious,” says Bodell. “At this point, since we have access to a production plant network, our next steps have to be really careful ones, because we don’t want to shut the plant down,” Bodell observes. While probing the network, the team may grab screen shots or evidence data to prove that they were there. Depending on the client contract rules of engagement, the team may stop the test once network administrator access is obtained, for example, or when it achieves whatever is deemed to be “the keys to the city,” as Bodell puts it. Wrap upAt the end of an engagement, Industrial Defender’s Red Team consultants provide a complete report with narrative, photos and screen shots detailing vulnerabilities uncovered and mitigation recommendations.Commonly encountered cyber vulnerabilities include uninstalled control system software patches that are not yet on vendors’ approved patch lists, says Bodell, as well as weaknesses involving unsecured legacy network hardware. The team typically stresses the importance of “layered” security defenses. On the physical side, fixes often include obvious items such as repairing holes in perimeter fences and correctly positioning motion sensors. The team often also recommends stepped-up user awareness training and testing for plant guards, control staff and other employees. In all, the Industrial Defender consultants have performed a total of five full-blown Red Team tests, in each case achieving their objective without being discovered. Based on what they’ve seen to date, the team believes that most industrial plants could benefit from better coordination between traditionally separate cyber security and physical security staffs. “One thing that is important for companies to understand is that even if they have strong cyber controls, their physical security, or
lack thereof, can also provide a huge attack vector into their process control networks,” Bodungen advises. For more information, search keywords “cyber security” and “physical security” at www.automationworld.com.
PlantData consulting operation, now known as Industrial Defender Consulting Services. Over the past six years, this organization has conducted more than 60 control system cyber security assessments for clients. These range from standard cyber vulnerability assessments to more extensive cyber penetration tests and all-out Red Team attacks. In most cases, only top personnel at a plant know when a Red Team test has been commissioned. Information technology (IT) and security staffs are not tipped off. “We carry letters from the top people in the [client] company with 24/7 phone numbers, so that if we do get caught, we don’t go to jail that night,” says Clint Bodungen, a security consultant who is a member of the Industrial Defender Red Team. A Red Team test can sometimes be mostly cyber-based. “If we can penetrate through the Internet, get through the corporate network and find the specific plant network that we’re looking for, then almost all of it is cyber,” says Ty Bodell, another of the Red Team members. But that scenario is rare, he adds; in most cases, a covert physical entry into the plant
is required. A major objective of the physical entry is to attach a wireless access device to the plant network. Once this is accomplished, the team can access the plant network wirelessly from outside the facility—parked on a nearby street or in a plant parking lot, for instance—taking all the time needed to probe the network. To be sure, the idea of a covert systems penetration attempt—even by friendly cyber security contractors—often makes process control companies nervous. Many first-time clients fear the test may adversely impact their systems. The Industrial Defender team attempts to allay those fears by pointing to the control systems expertise of many of its staff members, says Pollet, himself a former automation engineer for Chevron USA. “And we do have some rules of engagement that we sign off with the customer, which makes our process very safe.” To provide readers with a better awareness of the ways in which a determined group of motivated hackers, cyber terrorists or other criminals might attempt to gain access to their company’s critical control systems, Automation World interviewed Pollet and other Red Team members. We asked them to describe a Red Team test from start to finish, and to let us in on some of the tricks and techniques used to crack a company’s security defenses. Following is a report on what we learned.Who are they?When a client signs up for a Red Team test, the team is often provided with nothing more than the name of the client company. So the team’s first task is to discover all that it can about the customer. On the cyber side, this begins with research using publicly available Internet sources, says Bodell, who typically works with Patrick Turner, the fourth Red Team member, on most of the team’s cyber activities. “We’ll research what types of domain names the client has and the IP (Internet Protocol) address ranges they have,” says Bodell. “Typically, we’ll Google for e-mail addresses of people from the company in relevant positions like the IT and process control groups.” This information will be passed along for use in “social engineering” activities by the physical penetration team later, who may benefit by “name dropping” as they try to talk their way into a plant. Online sources of information that often prove useful include company press releases, mailing lists and Internet forums, Bodell observes. As part of their research, Bodell and Turner also typically use Google Earth software to obtain satellite pictures of the target plant, as an aid to determining the best locations for physical surveillance and eventual surreptitious entry. Also during the reconnaissance phase, the pair begin Internet-based scanning of computer ports discovered at the target company or plant, in an effort to gain information on systems and services, and to assess vulnerabilities. Taking what Turner calls a “slow and low” approach, they scan only a few ports at a time, as way to avoid detection by the target company’s IT security group.Surveillance While the cyber discovery work is going on, the team is also performing both nighttime and daytime physical surveillance of the target plant. Bodell often teams with Bodungen for this activity; it was a nighttime reconnaissance mission by this pair that is described in the opening paragraphs of this story.The team is typically able to gather as much information as it needs during four trips to a target plant—twice at night and twice during the day, says Bodungen. During the nighttime visits, the team will generally prowl the plant perimeter, looking for potential entry routes into the plant, and scanning for in-plant wireless frequencies that may be leaking from the plant, as well as for frequencies used by plant guards. “We’re also looking for key buildings that are either well-lit or high traffic, and have lots of wires running to them or maybe lots of fans on the back, indicating a data center,” says Bodungen. “We’ll pass that information on to the penetration team, because they may be locations of network access, which is what we’re targeting.” During nighttime visits, the pair typically tries not to be seen. But daytime surveillance techniques involve what Bodungen calls “hiding in plain sight.” For example, “we may get a plain white truck out there near the plant and put orange cones around us and act like we’re doing surveying work,” says Bodungen. Typically, Bodell and Bodungen will also spend time sitting in a car in a plant parking lot, “just acting like we’re supposed to be there,” he adds. The pair change locations frequently and use different vehicles each time they return. During the daytime surveillance, the pair take numerous photos, including close-ups of employee and contractor badges, as an aid to making their own fake badges later. “As people walk past our car in the parking lot, we inconspicuously snap pictures,” says Bodungen. They also take special note of the color and other details of contractor uniforms. “Typically, employees know each other. Contractors are in and out, so we can usually slip in and out of a plant a lot easier [disguised] as a contractor,” Bodungen explains.When it is time to enter the plant, the team assesses its options based on the intelligence it has gathered. In most cases, a daytime entry carries less risk. “Usually at nighttime, people know we’re not supposed to be there, so if they see us, we’re caught,” Bodungen says. “But during the daytime, it doesn’t matter if they see us, as long as we look the part.”The team typically attempts two entries simultaneously at different places by different team members, each carrying wireless access points to be planted inside. This increases the chances of success. “If one of us gets caught, the guards would go on alert, and we wouldn’t have a second chance,” Bodell explains. “So we’ll strike at the same time.”Pollet typically attempts one of those entries, most often disguised as a contractor. “Sometimes, we actually go through the front gate with a [fake] work order that tells people we’re supposed to be there,” says Pollet. But in the case described in the opening scene, the team decided to send Pollet in through a rear area of the plant where surveillance had revealed an easy entrance point through the perimeter. We’ll pick up more of that story here:PenetrationWearing a hard hat with safety glasses, and a fake contractor badge and uniform, Pollet carries a duffle bag containing a wireless access point, hubs, switches, a laptop computer and various gear to connect the computer to the plant network. Once inside the plant environment, he walks around freely, nodding to others, who typically smile and nod back. “It’s a Friday, so most of these guys have got one thing in mind—heading out for the weekend,” he surmises.In search of a control room, he tries the doors on several likely looking buildings. “If they’re open, I walk right in,” Pollet says. He eventually hits pay dirt; as he enters one building, he sees two men standing outside the door to what is obviously a control room containing SCADA terminals and other equipment. He walks up to the pair and begins making small talk: “Man, I’m glad it’s Friday…” After they chat for a few minutes, Pollet saunters into the control room, goes directly to a jack and begins plugging in his laptop. A few minutes later, the two men enter the room, pull up chairs and continue talking. They don’t ask Pollet what he is doing. “While I’m talking with them, I’m basically scanning their network,” says Pollet. “I’ve got various programs running in the background, bringing back the names of their computers, their IP addresses, operating systems and the kinds of applications they’re running.”When he is left alone, Pollet takes pictures of the control room equipment, and also snaps pictures of himself sitting at a control station, to be used in the wrap-up report to the client. He also attaches his wireless access device, hiding it in a bundle of wires in a SCADA console cabinet. Uh ohWhile this is going on, Bodungen, also dressed as a contractor, is attempting to join Pollet by entering through the plant’s front gate. He’s got a fake work order. But he runs into trouble. The front gate guard can’t find the contractor name on his list. And the more Bodungen tries to convince the guard that he is legitimate, the more suspicious the guard becomes.At this point, Bodungen goes to his exit plan; he makes a phone call to Bodell and Turner, who are in another location. Bodungen fakes a conversation on the cell phone, then tells the guard that he was mistakenly dropped at the wrong work site. Someone will come by shortly to pick him up, he says. This immediately relaxes the guard. “His suspicion goes away, because now he has a reason for me to be there,” Bodungen relates. “So I figure that I can use the opportunity to gather some more recon close up.”While “waiting for his ride,” Bodungen chats with the guard, who is now comfortable enough to leave Bodungen alone in the guard shack on several occasions. “I could have grabbed a handful of badges, because they were just hanging there, or I could have sneaked out the back and gone on into the plant,” says Bodungen. He does neither. But he is there long enough to observe the strict exit procedures practiced by the guard; Bodungen calls Pollet and advises him not to try leaving by the front gate—the original plan—but to instead exit the same way he entered, through the rear perimeter. Meanwhile, Bodell and Turner are trying a different penetration approach. During surveillance, the team had identified an office building that is not within the plant fence boundaries, but has cables running from it into the plant environment.
They suspect the building may be on the plant network. Dressed in office casual clothing, they enter through the front door, walk past an unmanned security desk, ignoring the sign-in sheet, and proceed unchallenged into the building. “We have our laptops out, with antennas sticking out, looking around as though we’re doing a wireless signals survey,” says Bodell. “But we never have to use our story, because nobody talks to us.”The pair locate a printer room, where they attach and conceal a wireless access point, then quickly leave the building. Back in their car in a nearby lot, they successfully connect to the access point, and find themselves on the plant network. They call Pollet, who is still in the plant control room, and tell him to retrieve his access point and get out. “Since we had an access point working outside, we didn’t need to risk having to do the more difficult penetration back into the plant later to retrieve an access point there,” Bodell explains. Bodell and Turner drive to the plant front gate and pick up Bodungen, then pick up Pollet exiting the plant at the designated spot, and the team goes home for the day. From here on, the physical work is done, and rest of the Red Team attack is cyber penetration testing. With an access point in place, team members are free to come back, park on a nearby street or a plant parking lot, and take their time probing the network. “We usually choose a time that’s late at night on the weekend, or maybe at 5 p.m. on a Friday so the cars are still there and we don’t look suspicious,” says Bodell. “At this point, since we have access to a production plant network, our next steps have to be really careful ones, because we don’t want to shut the plant down,” Bodell observes. While probing the network, the team may grab screen shots or evidence data to prove that they were there. Depending on the client contract rules of engagement, the team may stop the test once network administrator access is obtained, for example, or when it achieves whatever is deemed to be “the keys to the city,” as Bodell puts it. Wrap upAt the end of an engagement, Industrial Defender’s Red Team consultants provide a complete report with narrative, photos and screen shots detailing vulnerabilities uncovered and mitigation recommendations.Commonly encountered cyber vulnerabilities include uninstalled control system software patches that are not yet on vendors’ approved patch lists, says Bodell, as well as weaknesses involving unsecured legacy network hardware. The team typically stresses the importance of “layered” security defenses. On the physical side, fixes often include obvious items such as repairing holes in perimeter fences and correctly positioning motion sensors. The team often also recommends stepped-up user awareness training and testing for plant guards, control staff and other employees. In all, the Industrial Defender consultants have performed a total of five full-blown Red Team tests, in each case achieving their objective without being discovered. Based on what they’ve seen to date, the team believes that most industrial plants could benefit from better coordination between traditionally separate cyber security and physical security staffs. “One thing that is important for companies to understand is that even if they have strong cyber controls, their physical security, or
lack thereof, can also provide a huge attack vector into their process control networks,” Bodungen advises. For more information, search keywords “cyber security” and “physical security” at www.automationworld.com.
