Posey's Tips & Tricks
When AI Goes Wrong: The Hidden Risks of Conditional Access Controls, Part 2
AI-driven conditional access systems can unintentionally lock trustworthy employees into a cycle of escalating mistrust, turning minor incidents into lasting reputational damage through bias and negative feedback loops.
In my first article in this series, I talked about the way that AI based conditional access controls and zero trust are supposed to work. Even so, things can go very wrong as a result of negative feedback loop amplification, bias reinforcement and cross domain AI. I want to begin by focusing specifically on AI based conditional access. In Part 3, I will get into a discussion of how these problems can spread to other systems with potentially devastating consequences.
So with that said, imagine for a moment that a particular company uses AI based conditional access. Let's also pretend that there is a user at that company who is a good employee and never causes any problems. For the sake of this discussion, let's assume that this employee has no animosity toward the company and would never intentionally engage in malicious activity.
Like so many other people, our fictitious user has a hybrid work style, coming into the office some days and working from home on a personal device on other days. The conditional access system is aware of this hybrid work and treats it as normal.
Now, just to make things interesting, let's pretend that the employee's kid borrows their device and visits some sketchy Web sites. The next day, the employee decides to work from home and logs on from the same device that they always use. This time however, the AI detects that the device has been used in a potentially unsafe manner. At a minimum, this would increase the device risk score. Realistically however, since the device is owned by that user, the AI would probably attribute anything that has been done to the device as having been done by the user who owns it. As such, the AI might assign a higher risk score to the end user while also lowering the user's overall trust rating. Once again however, this is how conditional access is supposed to work.
The reason why this functionality can become problematic stems from the fact that AI never forgets. In other words, our fictitious user no longer has a stellar reputation, at least not as far as the AI is concerned. Now, there is a historical marker indicating that the user has engaged in risky behavior at least once in the past. In all fairness, some AI engines contain a "decay algorithm" which will eventually cause old history items to be forgotten, but based on my own observations, it seems that most do not.
The problem with allowing AI to remember what has happened in the past is that future actions will likely be judged through a historical lens. In other words, past behavioral history becomes one of the signals that AI looks at when making access control decisions.
If a user has committed a single "infraction" at some point in the past, that action should probably be regarded as an outlier and not taken into account with regard to access control decisions. After all, a single bad decision is not the same as a long history of persistent risky behavior. However, the AI engines used in security products are often tuned to avoid false negatives, which by definition increases the risk of false positives. The end result is likely to be that the AI regards the past incident as being way more serious and significant than it actually was. This could mean that if in the future, the user were to do something that is only slightly risky and that would typically be ignored, the system would likely judge the user more harshly than it would someone else, simply because of the user's past history. And because this extremely minor risk was judged so strictly, it may count as a second "incident" for that user.
Now that there are two incidents associated with the user, there is (as far as the AI is concerned) a pattern of behavior. This means that the AI may begin judging the user's actions even more harshly than before, thereby creating a negative feedback loop in which a trusted employee with no ill intent is increasingly viewed as a threat.
To the best of my knowledge, the AI engines that are built into conditional access systems do not contain an “enemies list” feature. Even so, an otherwise trusted user could be treated as a “potential enemy” or at the very least, as someone who is less trustworthy than their coworkers. Such a user might be punished with frequent MFA challenges or even outright denials pertaining to high trust workflows.
So far I have only described AI related problems as they pertain to conditional access. However, a conditional access system's bias against a user can adversely affect the user in ways that have little to do with access control decisions and that may ultimately ruin a user's career. I will talk about how this works in Part 3.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.