Practical App

How To Protect Your Network with Lync 2010 Edge Server

DoS and password brute-force attacks can be particularly nasty, but you can protect your network against these types of disruptions.

• By Rui Maximo
• 08/22/2011

In this extraordinarily interconnected world, companies want to allow the utmost flexibility and mobility for their employees, many of whom may work remotely. Consequently, almost every organization exposes services to the Internet. However, there's always the threat of attacks. Some companies are particularly concerned withDenial of Service (DoS) and password brute-force attacks. These are legitimate concerns. These types of attacks can be disruptive to users and consume internal server resources.

The primary trouble with DoS attacks is that they're nearly indistinguishable from legitimate sign-in requests.

The Microsoft Lync Server 2010 Edge Server protects against unauthorized access using industry-standard security measures. It monitors sign-in requests and enforces account lockout at the network perimeter. All communications are encrypted and authenticated.

Edge Server does not protect against DoS attacks. However, Lync Server provides a flexible programming platform you can use to create server applications to intercept Session Initiation Protocol (SIP) messages on the server and perform specialized logic using the Microsoft SIP Processing Language (MSPL). This is how the security filter operates.

It inspects all incoming sign-in requests on the Edge Server. The remote user is not authenticated at the Edge Server, so the sign-in request is passed to the Director or directly to the internal pool, which then performs the authentication process. The response is then passed back to the Edge Server. The security filter inspects both the request and the response. If the sign-in fails, the security filter tracks the number of failed attempts for each user account.

The next time a client attempts to sign in to the same user account, and the number of failed attempts exceeds the maximum number of allowed sign-in attempts, the security filter immediately rejects the request without passing the request to the Director or internal pool for authentication. By enforcing account lockout at the Edge Server, the security filter blocks DoS attacks at the edge of the network perimeter. As a result, the security filter protects the internal Lync Server resources.

Using the security filter to prevent Windows NT LAN Manager (NTLM) version 2 authentication, companies can force their users to only sign in from authorized company-issued laptops.

If at First You Don't Succeed...
To prevent brute-force attacks on user accounts, many organizations enforce an Active Directory Group Policy to lock out the account after a certain number of failed attempts. The side effect of this countermeasure is that the attacker can lock out a user's account by simply launching multiple attempts. This is what amounts to a DoS attack.

If the account isn't protected by an Active Directory Group Policy, the attacker can use this type of brute-force attack on the user's password. These attacks use up valuable internal server resources and deny users access to their account.

Uniquely identifying the user can prevent attacks on user accounts. There are several options with which to do this. You could use the source IP address, the sign-in name (that is, the SIP URI), the account name or even a combination of any of these options. After investigating each option, it seems that rogue clients mounting a DoS attack could spoof the source IP address, eliminating this choice as a way to uniquely identify the user.

The sign-in name, although required to successfully sign in to Lync Server, does not authenticate the user. A sign-in name can be varied during sign-in requests, yet still lock out the same user account. Therefore, neither the source IP address nor the sign-in name are good sources with which to identify the user. Only the account name uniquely identifies the user account.

You can only extract the account name, which consists of the user name and domain name, from the authentication protocol. Remote users trying to sign in and authenticate use the NTLM v2 protocol, not Kerberos. The NTLM protocol uses a three-stage handshake authentication process. The client passes the user's credentials in the third stage of the NTLM handshake.

The security filter runs as a trusted server application on the Edge Server, so it's allowed to intercept this sign-in request. The security filter decodes the user name and domain name from the NTLM authentication message. Because the account name isn't available in the response, the security filter maps the response to the request using the message ID.

When either the internal pool or the Director sends the authentication response to the Edge Server, the security filter captures the Register response. If the sign-in failed, the security filter counts the failed attempts. If the sign-in succeeds, the security filter resets the count of failed attempts to zero.

Filtering at the Edge
Every time the Edge Server receives a sign-in request, it's passed to the security filter. It checks whether the sign-in request has exceeded the maximum number allowed for that particular user account. If the request has not exceeded the maximum lockout count permitted, the security filter allows the request to continue to either the internal pool or the Director.

If the request exceeds the maximum lockout count permitted, the security filter blocks the request and returns a 403 response. This summarily rejects the request. Any further sign-in attempts are rejected for the duration of the lockout period. After the lockout period expires, it's reset to permit new sign-in requests.

One problem can occur when users sign in from a computer not joined to the corporate Active Directory domain. Lync 2010 can automatically attempt to sign in using the user's local computer credentials. Because those credentials aren't corporate domain credentials, the authentication will fail. The user will eventually be blocked from signing in to Lync Server. To prevent the security filter from locking out valid users, it doesn't count these attempts against the user.

Lync Server 2010 introduces support for an additional authentication protocol called TLS-DSK. This requires users to supply a client certificate for authentication. The Lync client requests certificates from Lync Server. This is an automatic process that happens the first time the user signs in to Lync Server from within the corporate network where the user is authenticated using Kerberos.

This client certificate is used for authentication with any subsequent log-in attempts. This is a self-signed certificate issued by Lync Server, not a Certificate Authority. If that same user tries to sign in to Lync from a different computer, he's authenticated using Kerberos (if inside the corporate network) or using NTLM v2 (if outside the corporate network). The process of obtaining another client certificate starts all over.

TLS-DSK provides a level of security that's very close to two-factor authentication. When combined with Windows BitLocker, the computer or laptop acts as the equivalent of a smartcard (something you have). The password that BitLocker requires to boot your computer is equivalent to the pin required to authorize the use of the smartcard (something you know).

There's the remote possibility someone could steal the client certificate from the user's computer, but you can mitigate this risk. Make sure corporate-issued computers are locked down to prevent users from downloading unauthorized applications.

You can force the Edge Server to negotiate the authentication protocol down from TLS-DSK to NTLM v2. In this case, the attacker can still target the user's account, as discussed earlier. To prevent this scenario, the security filter provides an option to reject all NTLM v2 authentication requests, forcing TLS-DSK-only authentication. This doesn't affect federated partner connections or PIC connections.