Windows Insider

Isolation Automation Exploration: Part I

Vista makes system-to-system IPSec authentication MUCH easier, if you know how to use it.

• By Greg Shields
• 05/01/2007

Before Windows Vista, setting up IPSec for system-to-system authentication was complex, sometimes requiring hundreds of filters to secure traffic between domain controllers while at the same time not inhibiting log-ons for older operating systems. When a non-IPSec-aware client tried to connect to an IPSec-enabled server, it often resulted in no connection at all.

Thankfully, with Windows Vista's improvements to IPSec, it all gets a lot easier. It's now possible to create isolation groups that mandate machine-to-machine authentication between sets of computers on your network.

Additional Authentication
So what's an isolation group? It's a way of using network rules to further protect potentially open spots on your network. Let's say an administrator accidentally shares a sensitive folder on your file server with Full Control permissions to the Everyone group. Suddenly, all that sensitive data is immediately exposed to anyone. If the data is on a human resources or other highly sensitive server, you're really in trouble. Isolation domains leverage IPSec to ensure that any machine attempting to connect to that share must authenticate via Kerberos before it can transfer data. Think of an isolation domain as an extra access control list (ACL) -- like NTFS and share permissions-but way down at the network level. This extra computer-based ACL ensures that only the correct machines get access to sensitive data and can only transfer that data securely.

Here's how it works. When you log in to a computer, your user account goes through a Kerberos authentication process that ensures you are who you say you are. Adding in an isolation group with IPSec means that any time your computer tries to access another computer, the computer itself goes through an additional authentication. If your computer successfully authenticates, then you can access the data. This assumes of course that you then have the correct share and NTFS rights. If your computer can't authenticate, the server either rejects the request or allows a fallback to clear text communication.

All this was possible in Windows 2003, but IPSec was notoriously difficult to set up. In Windows Vista, IPSec configuration has been merged with the Windows Firewall and is now called Windows Firewall with Advanced Security.

In setting up an isolation group, four types of canned rules are available or a custom rule can be created:

• Isolation: This will create a group of machines that are isolated from other computers. This group can be for all machines in the Active Directory Kerberos boundary, or can be an identified list of machines by IP address. Authentication can occur for either inbound or outbound traffic, or both.

• Authentication Exemption: This will create a group of machines exempt from any authentication requirements.

• Server to Server: This will create an authenticated connection between two specific groups of computers. Think of this as the "one-to-one" connection where the Isolation group would be the "many-to-many" connection.

• Tunnel: Like Server to Server, but usually used for bridging traffic across the Internet, this will create an authenticated connection between two computers utilizing an Internet-facing gateway server.

• Custom Connection: A connection that can be created using a combination of the four different rules.

Next month I'll give step-by-step instructions for setting up an isolation group on your network and go over some other tips on how to protect your network from the inside out.