Sometimes you can’t ignore the absolutes. In the case of communication
security, just remember: If it doesn’t match, it’s gonna crash.
We deal with rules every day. Some of them are absolute, like the fact
that gravity is going to send me crashing to my death if I plunge off
the top of the Empire State Building. Others aren’t so inhibiting: When
I rocket down the expressway at 20 miles per hour over the speed limit,
an officer of the law isn’t going to immediately and always appear to
give me a speeding ticket. (In my case, I think that rule probably is
an absolute, but that’s another story.) The trouble is that in configuring
some parts of complex networking scenarios, we tend to think we’re dealing
with the variable truths of highway law vs. the absolute truths of the
laws of physics.
Well, to be sure, we’ve gotten away with guesses and what I like to call
the “New York Times Crossword Puzzle” approach to computer configuration
in the past, so we figure we can today. What’s the New York Times
Crossword Puzzle approach? That’s when you look at the blanks in the property
sheets of the configuration and use your past knowledge and ability at
guessing what word goes where. Come to a blank and haven’t got the foggiest?
You ask the person next to you if he knows a 12-letter word that fits
the bill. He doesn’t know? You ask the man next to you on the subway or
the lady waxing the floor in the lobby. If everything works, you’re a
genius. If it doesn’t, then we just blame it on the “crappy product.”
If you’re charged with using IPSec to secure communications between computers
on your network, I can personally guarantee you that this approach won’t
The IPSec protocol is complex and has the interface to match. It’s not
one you can fake your way through. There’s a simple rule, however, that
can help you through: If it doesn’t match, it’s gonna crash.
In order to understand how to use that rule to study and think your way
to IPSec nirvana, you need to understand the meaning behind the parameters
that must be configured and how they must match. This month, I’ll take
you through the facts behind the choices and where to find them in the
interface. Next month, I’ll give you the step-by-step low-down on configuring
policies that work.
3DES — Triple-DES
AH — Authentication Header
DES — Data Encryption Standard
ESP — Encapsulating Security Protocol
IETF — Internet Engineering Task Force
IKE — Internet Key Exchange
IPSec — Internet Protocol Security
MD — Message-Digest Algorithm
RFC — Request for Comments
SA — Security Association
SHA — Secure Hash Algorithm
SPI — Security Parameter Index
Policies in Pairs
The first thing to remember is that an IPSec policy concerns communication,
and communication always consists of something that occurs (or doesn’t
occur if our intent is not to communicate) between two objects. An IPSec
policy, of course, determines whether or not two computers can communicate,
what they can communicate, and how that communication takes place. It
specifies which protocols must be secured and how they’re secured. Therefore,
IPSec polices occur in pairs. What has to match? Everything. What’s everything?
That can roughly be divided into two areas:
- The broad intent of the policy or lack of policy.
- The intimate details of the policy rules.
The first pairing of policies determines whether or not communication
can be negotiated. That is, the intent of the policy on either side of
the wire must match. Three possibilities exist:
- Allow — Normal communications may occur.
- Block — No communication may occur.
- Negotiate — Communication must be negotiated.
An “allow” policy is easy to understand. There are two types: Either
the requested communication matches “no rules defined” in the policy or
it matches a specific rule for which the filter action is “Permit.” In
either case, these communications are unhampered by further negotiation.
A blocking policy prevents communication of some kind from occurring.
It doesn’t matter how the policy on computer A is configured if the policy
on computer B is configured to block communications with computer A. Communication
won’t occur. Finally, the intent of the policy may be to protect communication
between computers A and B. The policy rules can be further restricted
by applying them to the local area network, all network connections, or
remote access connections only. If the rule filters are triggered, many
decisions must be negotiated. These decisions consist of an agreement
on encryption algorithms and strength, authentication methods, and protocol
types. For communication to occur, the intent of the policy must match.
The details of the policy rules specify how the negotiated communication
will occur. They may consist of complex sets of rules and filters that
specify how various protocols may or may not be allowed to leave or be
received by the computer upon which the policy lies. For example, Telnet
traffic between computers A and B may be allowed to occur after the two
computers have authenticated each other and the data has been protected
by using 3DES for encryption, SHA1 for integrity, and keying material
using Diffie-Hellman group 1. If either computer’s IPSec policy concerning
this protocol differs in any of these areas, no Telnet traffic will occur
between them. The rule details must match.
All of this seems straightforward. If you don’t want
communications to occur, write a blocking policy. If you
don’t want them to be hampered, create no policy — either
blocking or negotiating — that could restrict them. If
you must secure communications, write a policy
that can match intimate details between both computers.
Filters and Filter Actions
Each IPSec Policy consists of rules that consist
of filters. Each filter can be configured to affect
all or individual protocols, but they don’t filter
certain types of packets such as WebTV. When a request
to transmit a protocol matches a filter of an assigned
rule, a filter action determines what happens. Filter
- Permit — A
permit or passthrough filter action means that
negotiations aren’t allowed for secure communication.
This traffic is ignored by IPSec. If a filter
is configured to act on Telnet traffic and the
filter action is “Permit,” Telnet traffic will
always be allowed to pass. It won’t be encrypted,
authenticated or checked for integrity by IPSec.
This type of filter list should be limited to
avoid letting through traffic that should be secured.
- Block — Any traffic filtered
with a filter action of block will be stopped.
If outbound, it isn’t transmitted. If inbound,
it’ll be stopped.
- Request Security — Filtered
traffic should be secured, but communication with
a non-IPSec-capable computer will be allowed to
pass. This filter action should be used sparingly.
If the policy negotiation fails for any reason,
normal communication will be used and no security
- Require Security — Traffic
filtered here must be secured according to the
policy configuration. If negotiations fail, no
The choice of an IPSec Filter Action is made on
the rule property pages in the Filter Action tab.
Important IETF RFCs that describe parts of the IPSec protocol and
IKE are listed here for further information. You’ll find those at
- 2401, “Security Architecture for the Internet Protocol”
- 2402, “IP Authentication Header”
- 2406, “IP Encapsulating Security Payload (ESP)”
- 2409, “The Internet Key Exchange (IKE)”
The Devil’s in the Details
Sounds simple, doesn’t it? At least until you look at all of the blank
boxes in the configuration interface. There are many decisions to make,
and even the wizards that exist as part of this interface are difficult
to follow unless you know the meaning or expected result of choosing from
the many combinations. There is, however, a workable approach to this
- Break the process down into its component parts.
- Understand the “why” behind each.
- List the choices available for each.
- Pick the choice you want.
- Put the pieces back together again.
To configure any part of the policy, you must understand the meaning
behind the choices. Each choice hinges on the requirements you might desire:
- End-point Initialization Authentication
— How can a computer be sure it’s negotiating a connection with its
approved communication partner?
- Confidentiality — Is the communicated data
kept private to the two parties?
- Data Authentication — Is each received
packet coming from the approved partner?
- Data Integrity — Is the packet being received
the same as the packet sent?
- Protection from Replay — Can captured packets
be used by an unauthorized party to spoof approved communication channels?
Don’t forget: If the policy parts on both computers don’t match,
it’s gonna crash. Two factors affect the negotiations. First, to orient
you to your overall configuration choices, you should understand that
a policy is composed of rules; rules are composed of filter lists; and
filter lists are composed of filter actions, authentication methods, tunnel
settings and connection types. If a policy is activated, the basic process
is to determine if the communication being attempted meets any of the
filters (should the communication be blocked, allowed or negotiated?).
Second, if it’s determined that secured communications should be negotiated,
negotiations consist of two phases and each phase has configuration choices.
The rationale behind this approach is two-fold: First, while negotiations
for specific protocols may be complex and varied, the initial process
of authentication and the generation of master keying material remain
the same. The path established by phase one can be used to secure the
different negotiations necessary. Second, if master keying material is
to be reused, the repetition of the authentication process isn’t necessary.
You can save considerable time by having an existing, secured communication
path between the two computers. During phase one, IPSec:
- Requires mutual authentication of the computers.
- Negotiates the encryption algorithm.
- Negotiates the integrity algorithm.
- Determines the Diffie-Hellman group.
- Generates a master key.
- Produces the IKE Security Association.
Mutual authentication, of course, is the process of each computer proving
to the other that it’s who it says it is. In Windows 2000, you must choose
the method used for this negotiation: Kerberos, Public Key Certificates
or shared key. Here’s the main point: Each computer must use the same
method. Figure 1 details the choices.
|Figure 1. With Win2K, you must choose the method
for mutual authentication negotiation: Kerberos, Public Key Certificates
or shared key.
Encryption can be DES, 3DES (using high encryption pack), 40-bit DES
or none, but the algorithm used by one computer must match the algorithm
used by the other. If encryption is used, it’s IPSec Encapsulating Security
Protocol (ESP). Integrity, the verification of data to ensure that the
data sent from one computer to the other doesn’t change during transmittal,
can be either MD5 or SHA.
Integrity can be provided by either IPSec ESP or Authentication Header
(AH) protocols; but the choice assigned by each must match that of the
other. The protocol you select is used to prepare a checksum that can
be recalculated to verify that data content has remained the same.
IPSec uses the Internet Key Exchange protocol (IKE) to arrive at the
master key. The master key material generates the bulk encryption keys,
so it’s important that it be the same on each side. The master key is
never transferred from one computer to the other but is generated by each
How is this possible? IPSec uses the Diffie-Hellman method to generate
these matching bits. The Diffie-Hellman method uses an algorithm first
developed by Whitfield Diffie and Martin Hellman, two Stanford University
researchers who first publicly proposed this mathematical methodology
in 1975. In this algorithm, each computer shares an agreed-upon large
prime number and a public value. They each also have their own secret
key, which is used in combination with the prime number and the public
value to create different interim numbers. These interim results are shared
and used in combination with the computers’ secret keys. The results produced
on both sides match due to the nature of the mathematical calculation.
However, no other party can reproduce the same result since they can’t
know the actual value of either computer’s secret key. The result (the
master key) is used to produce bulk or session keys, which can be used
to encrypt data during the rest of the negotiations and during data communications.
The Diffie-Hellman group choice indicates the size of the prime number
used in the keying algorithm. The larger the Diffie-Hellman group, the
larger the number of bits in the key. Diffie-Hellman groups can be either
low (768 bits) or medium (1,024 bits), but the group number on one computer
must match that on the other.
Encryption, integrity and Diffie-Hellman groups are configured in the
Key Exchange Security Methods dialog of the Methods button on the Key
Exchange Settings properties page (Figure 2). Several default Security
|Figure 2. The Key Exchange Security Methods dialog
provides options for configuring encryption, integrity and Diffie-Hellman
The result of the phase one negotiation is the master key and the IKE
Security Association. The master key is used during the next phase to
derive the bulk encryption keys. A Security Association (SA) is a combination
of policy and keys that defines how communication is secured between two
computers. The IKE SA is used to secure phase-two negotiation.
Phase Two: Secure Traffic
- IPSec protocol (AH, ESP or both).
- Integrity algorithm (MD5 or SHA1).
- Encryption (DES, 3DES, 40-bit DES or none).
Two new SAs (one inbound, one outbound) are created and new keying material
is provided along with the generation of new bulk keys for packet authentication,
integrity and encryption.
Because a computer may securely communicate with many computers or may
have multiple SAs with a single computer, a unique value — the security
parameter index (SPI) — is used to identify each SA. The SPI value is
included in the IPSec portion of each packet.
As long as three things are true — the secure communications channel
isn’t idle, key lifetime hasn’t been reached and Perfect Forward Security
(a requirement that prevents the reuse of keying material) isn’t chosen
— the master key is used to generate new session keys on a periodic basis.
This process is known as dynamic re-keying, and you can modify its period.
A key lifetime determines the amount of time the keys are valid. If a
key lifetime expires and the protocol is actively being used, a new key
is negotiated. When key lifetime is reached for a master or session key,
the SA is renegotiated, an SA delete message sent and the SA marked as
expired. Thus, an old SA can’t be replayed by a counterfeit system.
Transport encryption, ESP or AH integrity, and key lifetime are configured
on the Security Methods tab of the Filter Action properties page. At the
end of the second phase both IKE and inbound and outbound SAs exist and
are ready for encrypted data to be transported between the two computers.
Figure 3 displays the result.
|Figure 3. By the end of phase two, both IKE and
inbound and outbound SAs exist and are ready for the transportation
of encrypted data between the two computers.
Putting It All Together
Now that you’ve examined the individual components of
an IPSec policy, read key definitions and inspected the
negotiation phases, it’s time to consider what happens
when a policy is assigned. IPSec driver is instrumental
to the enforcement of the policy. The IPSec driver stores
the SA information in the internal SA database and serves
as a go-between among the TCP/IP driver, application and
IPSec policy. Figure 4 presents an architectural, logical
diagram of the IPSec process.
|Figure 4. The IPSec driver serves as a go-between
among the TCP/IP driver, application and IPSec policy during the policy
- The red computer needs to communicate with the blue computer.
- As outbound packets on the red computer reach the IPSec driver, the
red computer looks for a match between the packet’s address or protocol
address and the active filter list in the active IPSec policy.
- If a match occurs, the driver asks IKE to begin security negotiations
with the blue computer.
- The blue computer’s IKE receives the request, and the two computers
authenticate each other. If the authentication methods don’t match,
authentication will fail and the process will end.
- The two computers negotiate the level of security and establish a
pair of IPSec SAs and keys. (If security details don’t match, negotiation
fails and packets are discarded.)
- The red computer’s IPSec driver uses the outbound IPSec SA and key,
signs the packet for integrity, and encrypts if confidentiality has
- The packets are transferred to the connection type for transmission
to the blue computer.
- The blue computer’s IPSec driver uses the inbound SA and key to check
the integrity signature and decrypt the packets, if necessary.
- The IPSec driver transfers the decrypted packets to the TCP/IP driver,
which sends them to the receiving application.
So you see, when it comes to IPSec, following the absolute “doesn’t match,
gonna crash” rule will keep you safe and sane and, better yet, keep you
from turning out a crappy product.