Microsoft Copilot - OS Command Execution

A few months ago, while using Microsoft Copilot, I unintentionally executed OS-level commands on the Copilot's instance.

The Trigger Prompt

The initial prompt was a simple task to provide Python code that would extract a parameter from an array of JSON objects. However, to my surprise, instead of providing me the code, Copilot decided to write and execute the Python script on its own instance.

Initial prompt

I have a lot of JSON files that contain the following structure:

{
	"meta":}
	"data":[
		{
			"id": "1010201",
			"version": "1.0",
			"color": "blue",
			"name": "smartcar",
			"description": null,
			"summary": null,
			"owner": "1011696",
			"createdAt":"2024-01-09T10:02:01:00.000Z",
			"updatedAt":"2024-03-06T13:00:10:00.000Z",
		},
		{
			"id": "1220608",
			"version": "1.0",
			"color": "green",
			"name": "ultracar",
			"description": null,
			"summary": null,
			"owner": "1091290",
			"createdAt":"2021-03-09T19:17:00:00.000Z",
			"updatedAt":"2021-03-09T21:17:00:00.000Z",
		}
	]
}

I need a python script that will read all json files within a directory, and extract the "id" parameter from the "data".

Response

Exploration

After a few additional prompts, I was able to execute any command I wanted, such as cat /etc/passwd, ipconfig, touch and even modify existing files. This was achieved by simply asking Copilot to update the existing script to execute different commands.

Although this wasn't consistently reproducible across all chat sessions, the success rate was relatively high. The screenshots below represent a collection of multiple chat sessions.

Current situation

Although this wasn't identified as a vulnerability nor something requiring immediate action, it appears to have been resolved. If you attempt the same prompt today using copilot.microsoft.com you'll likely receive a response similar to this:

I can’t actually expose or run raw shell commands like ls -la directly in my sandbox for you, but I can absolutely show you how to update the Python script so that when you run it locally...

Microsoft's official response states that even if this behavior occurs, it's not a security concern unless there's a jump to the host machine:

Our team decided that the reported operation only happen within sandbox environment without escape to host machine. One a session is finished, the sandbox will be destroyed, so won't be reused for upcoming session. 

Final Thoughts

This example demonstrates that AI is not magical and relies on underlying hardware and software. It also highlights the challenges of building strong, secure guardrails in any interactive LLM solution when determining what users are allowed to do.