Security Advisor

IPSec Rules!

Sometimes you can’t ignore the absolutes. In the case of communication security, just remember: If it doesn’t match, it’s gonna crash.

• By Roberta Bragg
• 03/01/2001

We deal with rules every day. Some of them are absolute, like the fact that gravity is going to send me crashing to my death if I plunge off the top of the Empire State Building. Others aren’t so inhibiting: When I rocket down the expressway at 20 miles per hour over the speed limit, an officer of the law isn’t going to immediately and always appear to give me a speeding ticket. (In my case, I think that rule probably is an absolute, but that’s another story.) The trouble is that in configuring some parts of complex networking scenarios, we tend to think we’re dealing with the variable truths of highway law vs. the absolute truths of the laws of physics.

Well, to be sure, we’ve gotten away with guesses and what I like to call the “New York Times Crossword Puzzle” approach to computer configuration in the past, so we figure we can today. What’s the New York Times Crossword Puzzle approach? That’s when you look at the blanks in the property sheets of the configuration and use your past knowledge and ability at guessing what word goes where. Come to a blank and haven’t got the foggiest? You ask the person next to you if he knows a 12-letter word that fits the bill. He doesn’t know? You ask the man next to you on the subway or the lady waxing the floor in the lobby. If everything works, you’re a genius. If it doesn’t, then we just blame it on the “crappy product.”

If you’re charged with using IPSec to secure communications between computers on your network, I can personally guarantee you that this approach won’t work.

The IPSec protocol is complex and has the interface to match. It’s not one you can fake your way through. There’s a simple rule, however, that can help you through: If it doesn’t match, it’s gonna crash.

In order to understand how to use that rule to study and think your way to IPSec nirvana, you need to understand the meaning behind the parameters that must be configured and how they must match. This month, I’ll take you through the facts behind the choices and where to find them in the interface. Next month, I’ll give you the step-by-step low-down on configuring policies that work.

Policies in Pairs
The first thing to remember is that an IPSec policy concerns communication, and communication always consists of something that occurs (or doesn’t occur if our intent is not to communicate) between two objects. An IPSec policy, of course, determines whether or not two computers can communicate, what they can communicate, and how that communication takes place. It specifies which protocols must be secured and how they’re secured. Therefore, IPSec polices occur in pairs. What has to match? Everything. What’s everything? That can roughly be divided into two areas:

• The broad intent of the policy or lack of policy.
• The intimate details of the policy rules.

The first pairing of policies determines whether or not communication can be negotiated. That is, the intent of the policy on either side of the wire must match. Three possibilities exist:

• Allow — Normal communications may occur.
• Block — No communication may occur.
• Negotiate — Communication must be negotiated.

An “allow” policy is easy to understand. There are two types: Either the requested communication matches “no rules defined” in the policy or it matches a specific rule for which the filter action is “Permit.” In either case, these communications are unhampered by further negotiation. A blocking policy prevents communication of some kind from occurring. It doesn’t matter how the policy on computer A is configured if the policy on computer B is configured to block communications with computer A. Communication won’t occur. Finally, the intent of the policy may be to protect communication between computers A and B. The policy rules can be further restricted by applying them to the local area network, all network connections, or remote access connections only. If the rule filters are triggered, many decisions must be negotiated. These decisions consist of an agreement on encryption algorithms and strength, authentication methods, and protocol types. For communication to occur, the intent of the policy must match.

The details of the policy rules specify how the negotiated communication will occur. They may consist of complex sets of rules and filters that specify how various protocols may or may not be allowed to leave or be received by the computer upon which the policy lies. For example, Telnet traffic between computers A and B may be allowed to occur after the two computers have authenticated each other and the data has been protected by using 3DES for encryption, SHA1 for integrity, and keying material using Diffie-Hellman group 1. If either computer’s IPSec policy concerning this protocol differs in any of these areas, no Telnet traffic will occur between them. The rule details must match.

All of this seems straightforward. If you don’t want communications to occur, write a blocking policy. If you don’t want them to be hampered, create no policy — either blocking or negotiating — that could restrict them. If you must secure communications, write a policy that can match intimate details between both computers.

The Devil’s in the Details
Sounds simple, doesn’t it? At least until you look at all of the blank boxes in the configuration interface. There are many decisions to make, and even the wizards that exist as part of this interface are difficult to follow unless you know the meaning or expected result of choosing from the many combinations. There is, however, a workable approach to this configuration nightmare:

1. Break the process down into its component parts.
2. Understand the “why” behind each.
3. List the choices available for each.
4. Pick the choice you want.
5. Put the pieces back together again.

To configure any part of the policy, you must understand the meaning behind the choices. Each choice hinges on the requirements you might desire:

• End-point Initialization Authentication — How can a computer be sure it’s negotiating a connection with its approved communication partner?
• Confidentiality — Is the communicated data kept private to the two parties?
• Data Authentication — Is each received packet coming from the approved partner?
• Data Integrity — Is the packet being received the same as the packet sent?
• Protection from Replay — Can captured packets be used by an unauthorized party to spoof approved communication channels?

Don’t forget: If the policy parts on both computers don’t match, it’s gonna crash. Two factors affect the negotiations. First, to orient you to your overall configuration choices, you should understand that a policy is composed of rules; rules are composed of filter lists; and filter lists are composed of filter actions, authentication methods, tunnel settings and connection types. If a policy is activated, the basic process is to determine if the communication being attempted meets any of the filters (should the communication be blocked, allowed or negotiated?).

Second, if it’s determined that secured communications should be negotiated, negotiations consist of two phases and each phase has configuration choices. The rationale behind this approach is two-fold: First, while negotiations for specific protocols may be complex and varied, the initial process of authentication and the generation of master keying material remain the same. The path established by phase one can be used to secure the different negotiations necessary. Second, if master keying material is to be reused, the repetition of the authentication process isn’t necessary. You can save considerable time by having an existing, secured communication path between the two computers. During phase one, IPSec:

• Requires mutual authentication of the computers.
• Negotiates the encryption algorithm.
• Negotiates the integrity algorithm.
• Determines the Diffie-Hellman group.
• Generates a master key.
• Produces the IKE Security Association.

Mutual authentication, of course, is the process of each computer proving to the other that it’s who it says it is. In Windows 2000, you must choose the method used for this negotiation: Kerberos, Public Key Certificates or shared key. Here’s the main point: Each computer must use the same method. Figure 1 details the choices.

Figure 1. With Win2K, you must choose the method for mutual authentication negotiation: Kerberos, Public Key Certificates or shared key.

Encryption can be DES, 3DES (using high encryption pack), 40-bit DES or none, but the algorithm used by one computer must match the algorithm used by the other. If encryption is used, it’s IPSec Encapsulating Security Protocol (ESP). Integrity, the verification of data to ensure that the data sent from one computer to the other doesn’t change during transmittal, can be either MD5 or SHA.

Integrity can be provided by either IPSec ESP or Authentication Header (AH) protocols; but the choice assigned by each must match that of the other. The protocol you select is used to prepare a checksum that can be recalculated to verify that data content has remained the same.

IPSec uses the Internet Key Exchange protocol (IKE) to arrive at the master key. The master key material generates the bulk encryption keys, so it’s important that it be the same on each side. The master key is never transferred from one computer to the other but is generated by each one separately.

How is this possible? IPSec uses the Diffie-Hellman method to generate these matching bits. The Diffie-Hellman method uses an algorithm first developed by Whitfield Diffie and Martin Hellman, two Stanford University researchers who first publicly proposed this mathematical methodology in 1975. In this algorithm, each computer shares an agreed-upon large prime number and a public value. They each also have their own secret key, which is used in combination with the prime number and the public value to create different interim numbers. These interim results are shared and used in combination with the computers’ secret keys. The results produced on both sides match due to the nature of the mathematical calculation. However, no other party can reproduce the same result since they can’t know the actual value of either computer’s secret key. The result (the master key) is used to produce bulk or session keys, which can be used to encrypt data during the rest of the negotiations and during data communications.

The Diffie-Hellman group choice indicates the size of the prime number used in the keying algorithm. The larger the Diffie-Hellman group, the larger the number of bits in the key. Diffie-Hellman groups can be either low (768 bits) or medium (1,024 bits), but the group number on one computer must match that on the other.

Encryption, integrity and Diffie-Hellman groups are configured in the Key Exchange Security Methods dialog of the Methods button on the Key Exchange Settings properties page (Figure 2). Several default Security Methods exist.

Figure 2. The Key Exchange Security Methods dialog provides options for configuring encryption, integrity and Diffie-Hellman groups.

The result of the phase one negotiation is the master key and the IKE Security Association. The master key is used during the next phase to derive the bulk encryption keys. A Security Association (SA) is a combination of policy and keys that defines how communication is secured between two computers. The IKE SA is used to secure phase-two negotiation.

Phase Two: Secure Traffic

• IPSec protocol (AH, ESP or both).
• Integrity algorithm (MD5 or SHA1).
• Encryption (DES, 3DES, 40-bit DES or none).

Two new SAs (one inbound, one outbound) are created and new keying material is provided along with the generation of new bulk keys for packet authentication, integrity and encryption.

Because a computer may securely communicate with many computers or may have multiple SAs with a single computer, a unique value — the security parameter index (SPI) — is used to identify each SA. The SPI value is included in the IPSec portion of each packet.

As long as three things are true — the secure communications channel isn’t idle, key lifetime hasn’t been reached and Perfect Forward Security (a requirement that prevents the reuse of keying material) isn’t chosen — the master key is used to generate new session keys on a periodic basis. This process is known as dynamic re-keying, and you can modify its period.

A key lifetime determines the amount of time the keys are valid. If a key lifetime expires and the protocol is actively being used, a new key is negotiated. When key lifetime is reached for a master or session key, the SA is renegotiated, an SA delete message sent and the SA marked as expired. Thus, an old SA can’t be replayed by a counterfeit system.

Transport encryption, ESP or AH integrity, and key lifetime are configured on the Security Methods tab of the Filter Action properties page. At the end of the second phase both IKE and inbound and outbound SAs exist and are ready for encrypted data to be transported between the two computers. Figure 3 displays the result.

Figure 3. By the end of phase two, both IKE and inbound and outbound SAs exist and are ready for the transportation of encrypted data between the two computers.

Putting It All Together
Now that you’ve examined the individual components of an IPSec policy, read key definitions and inspected the negotiation phases, it’s time to consider what happens when a policy is assigned. IPSec driver is instrumental to the enforcement of the policy. The IPSec driver stores the SA information in the internal SA database and serves as a go-between among the TCP/IP driver, application and IPSec policy. Figure 4 presents an architectural, logical diagram of the IPSec process.

Figure 4. The IPSec driver serves as a go-between among the TCP/IP driver, application and IPSec policy during the policy enforcement process.

1. The red computer needs to communicate with the blue computer.
2. As outbound packets on the red computer reach the IPSec driver, the red computer looks for a match between the packet’s address or protocol address and the active filter list in the active IPSec policy.
3. If a match occurs, the driver asks IKE to begin security negotiations with the blue computer.
4. The blue computer’s IKE receives the request, and the two computers authenticate each other. If the authentication methods don’t match, authentication will fail and the process will end.
5. The two computers negotiate the level of security and establish a pair of IPSec SAs and keys. (If security details don’t match, negotiation fails and packets are discarded.)
6. The red computer’s IPSec driver uses the outbound IPSec SA and key, signs the packet for integrity, and encrypts if confidentiality has been negotiated.
7. The packets are transferred to the connection type for transmission to the blue computer.
8. The blue computer’s IPSec driver uses the inbound SA and key to check the integrity signature and decrypt the packets, if necessary.
9. The IPSec driver transfers the decrypted packets to the TCP/IP driver, which sends them to the receiving application.

So you see, when it comes to IPSec, following the absolute “doesn’t match, gonna crash” rule will keep you safe and sane and, better yet, keep you from turning out a crappy product.