Posey's Tips & Tricks
How Microsoft Protects Copilot Against ASCII Smuggling Attacks
With new tech comes the rise of new attack vectors.
I have long suggested that given enough time, AI input prompts (like the ones used by ChatGPT and Microsoft Copilot) would be exploited in ways that we have not yet imagined.
I have experimented with various ideas for exploiting AI prompts, and with mixed success. Most of these experiments have involved phrasing questions in creative ways so as to trick the AI into giving up information that it would not normally disclose. Some of my results have been interesting, to say the least.
Recently however, I learned about a new technique that has been used as a way of manipulating AI prompts. There are a few different versions of this technique, but it centers around something called ASCII smuggling.
The basic idea behind the ASCII smuggling exploit is that there are a few different ways to get a computer to display characters on the screen. Early PCs and other computers of the time used a standard called ASCIII. ASCII is an acronym for American Symbolic Code for Information Interchange. It works by assigning a numeric value to each character. There are also some ASCII values that produce control codes and there are some that display various symbols. WhenI was a kid for example, I wrote a short BASIC program that created a loop that counted from 0 to 255. For each number, I made the machine display the ASCII character associated with that number, as a way of familiarizing myself with the way that ASCII works.
Herein lies the problem though. The reason why I used the numbers 0 to 255 is because the computer that I was using (an old Radio Shack Color Computer) could only produce 256 unique characters. Some of the other computers of the time could only display 128 characters. That’s not a big deal if you only need to be able to display basic English text, but having such a limited character set is hugely problematic if you need support for characters used in foreign languages.
This is where Unicode comes into play. Unicode is similar to ASCII in that it is used to display characters on the screen. Whereas ASCII is limited to 128 or 256 characters however, Unicode supports 17 individual planes, each of which can map 65,536 characters, for a grand total of 1,114,112 possible characters.
So why does any of this matter and what does it have to do with AI prompt exploits? Well, because Unicode is so flexible, its creators have designed it to be able to do more than just display characters on the screen, though that is its primary purpose. Unlike ASCII, Unicode natively supports the use of tagging. Tagging is essentially just a method for embedding metadata within a string of text, but in a non-disruptive way. In other words, if you were to display a Unicode text string, any tags would be hidden from view.
The important thing to keep in mind about these tags is that even though they are hidden from view, that doesn’t mean that they do not exist. More importantly, when you enter text into a Large Language Model based AI prompt, the prompt looks at the full text string, including both visible and invisible characters. I don’t know that this holds true for all Large Language Models (though it might), but at least some will interpret all input text, whether that text is visible to humans or not!
This is where the aforementioned ASCII Smuggling attack comes into play. The idea is that an attacker can silently embed strings of invisible Unicode characters into an AI prompt. According to The Hacker News, Microsoft has fixed the vulnerability within Copilot that would allow an ASCII smuggling attack to succeed. However, it is possible that similar attacks could be made against other, non-Microsoft Large Language Models.
So with this in mind, just imagine what an ASCII smuggling attack against Copilot might have looked like. An attacker could conceivably use the invisible characters to provide instructions to Copilot. For example, if a user were to initiate a Copilot query against an email, hidden instructions within a compromised system might instruct Copilot to check to see if the email contains passwords and if so, then silently forward the email to a particular address.
Similarly, the article on The Hacker News mentions an attack in which Copilot was used for spear phishing attacks. Though not a lot of information was given, the article mentions an attacker using a compromised machine using Copilot to create an email written in the style of the user whose machine had been compromised.
Although Microsoft has indeed fixed the problem, thereby preventing Copilot from being exploited in this way, the techniques discussed by The Hacker News underscore the idea that cyber criminals are constantly looking for new, and often low tech, ways to exploit emerging technologies for illicit gain.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.