Crossing the Great Divide
Keep your hair from turning gray by understanding how Active Directory access works between domains.
In spite of our civilized veneer, we are tribal animals at heart. Elementary
school kids join clubs. High school students form cliques. Military units
have officers and enlisted persons. There are Democrats and Republicans,
Eastern and Western football conferences, Orthodox and Reformed sects,
lovers of haiku and, well, you get the picture. Everywhere you look, deep
divisions form between folks who might otherwise be friends and colleagues.
Take Active Directory as a case in point. Whether you work in a global
conglomerate or a 20-node organization with two branch offices, it’s nearly
impossible to get different IT groups to trust each other enough to form
a single domain. Once you decide to have separate domains, though, arranging
user access to shared resources can get a little tricky. Consider, for
example, a situation encountered recently by two system administrators,
Ann and Frank, who work at a large university.
The AD forest for this university roots at a domain called ad.mortarboard.edu.
Ann works for the university hospital, whose domain is hospital.ad.mortarboard.edu.
Frank works for the College of Math and Science in the mathsci. ad.mortarboard.edu
domain. Figure 1 shows the trust relationships in AD Domains and Trusts.
|Figure 1. The domain trust relationships of Mortarboard
University. (Click image to view larger version.)
At Mortarboard University, staff and faculty in the hospital often require
access to research documents stored on servers in the College of Math
and Science. By the same token, pre-med students in the College need access
to medical records to do their research. Ann and Frank became frustrated
with the university’s convoluted cross-departmental work order system
and decided to use the transitive trusts between AD domains to control
access to resources in each other’s domains. Here’s how their arrangement
Let’s say grad students in the MathSci domain need access to files stored
on Ann’s servers in the Hospital domain. Ann protects the shared folders
containing these files with NTFS permissions that give read/write access
to certain Domain Local groups in her domain. Ann chose to use Domain
Local groups because they can have members from other domains. This is
also true for Universal groups; but the Hospital domain has not yet been
shifted to Native mode, which is necessary for Universal groups to be
functional. Ann gave Frank permission to modify the membership of those
selected groups in her domain, and Frank uses this privilege to add and
remove his users to the groups in response to access requests. Ann and
Frank have worked together for years and they trust each other.
The Mystery of the Gray-haired Lady
One day, though, Ann noticed something strange and gave Frank a call.
“I’m looking at the membership list for one of my groups and I see a user
named Charlotte Johnson from your domain. It’s weird, though, because
the icon next to her name has gray hair. All the other icons from your
domain and mine have black hair.” She pasted a screenshot into an e-mail
message and sent it to Frank.
Frank scanned a listing in AD Users and Computers and told Ann, “I don’t
have any users named Charlotte Johnson. Hold on.” He riffled through some
paperwork. “I do have a Charlotte Hennessy and it looks like she got married
over the summer. There’s a completed work order here for changing her
name. Let’s see. Yup. Her maiden name was Johnson. Let me check her group
memberships.” He opened the properties page for Charlotte’s object in
AD Users and Computers and selected the Member Of tab. “Sorry,” he said,
“Must be someone else. Charlotte Hennessey doesn’t show up as a member
of any groups in your domain.”
Ann and Frank called a somewhat bewildered Charlotte Hennessey and verified
that she, indeed, used to be Charlotte Johnson. She also verified that
she had access to her files in the Hospital domain even though the group
showed “Charlotte Johnson,” not “Charlotte Hennessy,” as a member.
Because there didn’t appear to be any outright problem other than the
difference in the name and the funny icon, Ann and Frank decided to shelve
the mystery for a while. Later that day, just before she left for home,
Ann checked the membership list for the group again and discovered that
Charlotte’s name now displayed as Charlotte Hennessey and the user icon
once again had black hair, not gray. She left thinking her mother had
been right about studying dentistry rather than Information Systems Management.
To understand what happened to Charlotte’s name and user icon, and to
make Ann feel better about selecting a professional career that doesn’t
involve getting saliva on her hands, let’s look at the way AD displays
and replicates group members.
Forward and Backward
The Group properties displayed in the AD Users and Computers console show
the contents of various attributes associated with the group object in
AD. Group membership is contained in an attribute called Member. To ensure
internal consistency for Group members, AD uses a trick taken from double
entry accounting. It designates the Member attribute of a group as a forward
link attribute and associates it with a back link attribute called MemberOf
on each object that is a member of the group. AD doesn’t actually store
entries in a back link attribute. Instead, as shown in Figure 2, it calculates
the back link contents on the fly by querying the associated forward links.
For example, when you open the properties of a user object in AD Users
and Computers and select the Member Of tab, the domain controller searches
AD for any groups to which the user belongs and returns that list as if
it were the contents of the MemberOf attribute.
|Figure 2. Contents of the Member Of attribute
are calculated on the fly by querying back links to the Member attribute
in groups. (Click image to view larger version.)
If you look at the contents of the Member attribute for a group using
an LDAP browser or the ADSI Editor, you would see a list of LDAP distinguished
names in the format cn=user_name, ou=some_container,dc=domain_name,dc=domain_root.
Actually, though, AD doesn’t store distinguished names in the forward
link of a forward/back link pair. Instead, it stores the internal database
identifier of each object assigned to the forward link. This maintains
referential integrity of the object list.
A Rose by any Other Naming Context…
All this talk of forward and back links might seem like programming exotica,
but it goes to the root of the odd behavior of Charlotte’s user icon because
of the way AD stores objects from different domains. AD is an LDAP directory
service. In LDAP, it’s possible to divide the database into separate partitions,
called naming contexts. Each naming context forms a unique replication
unit. This reduces the size of the overall database hosted by any individual
DC and avoids replicating changes between geographically or politically
diverse portions of an organization. If an LDAP server gets a search request
for an object in a naming context it doesn’t host, it returns a referral
listing the LDAP servers where the client can find the correct naming
AD takes advantage of this LDAP feature by placing each domain into its
own discrete naming context. Furthermore, an AD DC can only host a read-write
replica of the naming context for its own domain. For example, a DC in
the Hospital domain would have a full, read-write replica of the Hospital
naming context and no replica at all of the MathSci naming context.
This creates a challenge for forward/back link attributes like Member
and MemberOf. Here’s why. When Frank originally added the Charlotte Johnson
account from the MathSci domain to the group in the Hospital domain, the
Hospital DC was forced to account for the entry by adding a database identifier
for the “Charlotte Johnson” object into the Member attribute for the group.
But the Hospital DC has no replica of the MathSci naming context and therefore
has no database identifier for the “Charlotte Johnson” object.
To overcome this challenge, a DC uses another accounting trick. It creates
a balancing entry called a phantom record in its naming context to represent
the object in the outside naming context. The phantom record consists
of the outside object’s Distinguished Name, the object’s SID, and its
Globally Unique Identifier, or GUID. Essentially, a phantom record acts
like a placeholder at a fancy dinner party that reads, “This place setting
belongs to the Tralfamadorian ambassador. Leave any messages under the
One more bit of information is necessary to explain what happened to
Charlotte’s name and icon. Although a standard DC can only host its own
domain naming context, AD has a feature called a Global Catalog (GC) that
permits selected DCs to host a partial, read-only replica of the naming
contexts for all domains in a forest. This permits a GC server to fulfill
LDAP search requests for any object in the forest so long as the search
involves attributes in the partial attribute set assigned to the Global
Catalog. For example, the partial attribute set contains the Distinguished
Name and SID attributes for user and group objects but doesn’t contain
the Member attribute for Domain Local groups such as those used by Ann
to control access to her shared resources.
So, armed with this information, let’s work out an explanation for Ann
and Frank as to why Charlotte’s name and user icon behaved so strangely.
When Charlotte’s name in the MathSci domain was changed from Charlotte
Johnson to Charlotte Hennessey, this resulted in a change to the Distinguished
Name attribute for Charlotte’s AD object in the MathSci naming context.
This modified Distinguished Name attribute was then replicated to every
DC hosting a replica of the MathSci naming context, including any Global
Charlotte’s name change had no impact on her security permissions because
her Security ID (SID) remained unchanged. This is why Charlotte was still
able to access resources in the Hospital domain even though her name didn’t
match the listing in the Member attribute of the Domain Local group protecting
As for the accelerated aging of Charlotte’s icon, this was a result of
the way that AD Users and Computers handles the contents of the Member
attribute for a group. When Ann opened the properties window for the group
and selected the Members tab, AD Users and Computers submitted an LDAP
search to a DC in the Hospital domain asking for the contents of the group’s
Member attribute. This search returned the Distinguished Names of the
group’s members, names that were derived from the database identifiers
that constitute the true contents of a forward link attribute such as
Member. One of those database identifiers pointed at a phantom record
for “Charlotte Johnson” in the Hospital naming context.
AD Users and Computers always verifies the membership list for a group
by querying a Global Catalog server for matches to the Distinguished Names
on the list. If it doesn’t find a match for a particular Distinguished
Name, it changes the associated user icon to an icon with gray hair rather
than one with black hair. This is why Charlotte’s icon changed. The name
“Charlotte Johnson” in the phantom record no longer existed in the MathSci
naming context stored on the GC server. (If AD Users and Computers cannot
locate a GC server at all, it displays a warning then shows all the icons
with gray hair.)
The Infrastructure Master Did It
That’s not the end of the story, though. Recall that when Ann checked
the group members later that day, she discovered that Charlotte’s name
had changed from Charlotte Johnson to Charlotte Hennessey and the user
icon once again had black hair. This happened thanks to the action of
a Flexible Single Master Operations (FSMO) role master called the Infrastructure
Master. (A FSMO is a DC that’s been assigned this unique duty, either
in a domain or in a forest.)
The Infrastructure Master periodically paws through the phantom records
in its domain and verifies that the Distinguished Names of the phantom
records match the Distinguished Names of the objects in the source domains.
It does this by querying a Global Catalog server. If the Distinguished
Name of a phantom record doesn’t exist in the GC, the Infrastructure Master
checks to see if the GUID still exists. If so, it creates a new phantom
record with a new Distinguished Name and deletes the old one. If the GUID
doesn’t exist, it deletes the phantom record entirely. These phantom records
then replicate from the Infrastructure Master to the other DCs in that
You may already know that it’s important not to assign the Infrastructure
Master role to a Global Catalog server. Here’s the reason. A DC creates
phantom records because it needs internal database identifiers for linked
attributes that reference objects from outside naming contexts. Because
a GC server hosts a replica of all naming contexts in a forest, it has
no need for phantom records and won’t create them. Without phantom records,
the Infrastructure Master has nothing to check, so the other DCs that
aren’t GC servers will not obtain updates to their phantom records. Always
be sure to transfer the Infrastructure Master role to a non-GC DC in each
If you have multiple domains in your production forest, you may have
already encountered the mysterious gray-haired icons seen by Ann and Frank.
The Infrastructure Master checks the phantom records every eight hours,
so it might take an icon and the associated user name a long time to change.
As we’ve seen, having the old name on the member list doesn’t interfere
with operations and requires no additional action on your part. That leaves
you free to attend the planning meetings for consolidating your domains
once Windows .NET is released. The fun never stops.