Press Esc to close

Three AI providers were leaking customer prompts, canary test finds

Three AI providers were leaking customer prompts, canary test finds
Dragos

Image: Dragos

HD Moore, chief research officer at the security firm Dragos, spent five months sending nearly two million prompts through AI services. Each one carried a unique URL, DNS name and sets of AWS keys that had never been used before. If anything touched them, he would know the prompt had leaked. In a write-up published this week, he says the results exposed three compromised providers, resellers stealing prompts and privacy settings that were ignored.

The most serious finding was at three hosting providers. Attackers used the planted AWS keys for tens of thousands of API calls, listing S3 buckets, searching Secrets Manager and Parameter Store, and trying to create Bedrock keys and IAM users. Some of the stolen requests had been rejected before a model ever ran, which Moore says shows the attacker was copying traffic at the provider's front door rather than deeper in the stack. None of the three knew about the breach until Dragos told them. Ars Technica reported the findings.

A second group did not need an exploit. Sellers on an inference marketplace listed popular models at low prices, switched on request logging and waited. In one case, someone on a home broadband connection opened two secret URLs about a day and a half later and began guessing usernames at a login page. In another, a Mac in France used passwords from the prompts to try root logins over SSH, then tested several of the AWS keys in one session.

The third finding is the one customers would most like to believe is impossible. A Chinese lab's systems fetched the secret URLs from more than half the prompts sent to its models, including about two-thirds of the requests marked for zero data retention, or ZDR. One US provider kept crawling a URL more than two months after a single search request. In the strangest case, a link from a prompt turned up in a Feishu company chat 24 days later and was opened from three desktops and three phones in Beijing and Shanghai. Dragos had shared only the first eight characters with the provider, so whoever posted it was reading a stored copy of the full prompt, sent to a US provider with ZDR on.

Moore also found routers that hid parts of the path. One forwarded traffic to a marketplace without disclosing it. Another seller quietly adds about a thousand hidden tokens to every prompt. Tracing roughly 2,000 routes, Dragos found several hundred that passed through a system it could not identify.

The lesson is not that AI services are unusable. It is that a privacy flag binds only the first hop, and that every router and reseller in the chain can read the whole prompt. Moore's advice is to host sensitive work yourself, buy directly from a first-tier provider, pick the upstream providers yourself instead of accepting a default pool, and plant your own canary tokens. Dragos did not name the providers; one has published a preliminary security notice, and about a dozen are working with the firm.

TechcityAuthority has covered the retention side of this before. Google keeps Gemini chats for two months even when history is off, and Meta's own staff warned that its Muse assistant read messages. The canary test shows the same problem one layer down, in the infrastructure companies use to reach models they do not run themselves.

Comments

Reader support

No ads. Just independent reporting.

Techcity Authority runs without advertising. Hosting, tools, and the hours behind each piece are supported by readers instead. Even a small contribution goes a long way — and every supporter is credited by name below for the month they give.

Support the site

Monthly via PayPal · cancel anytime

Secure checkout on PayPal. Your name appears here for the month you support.

This month’s supporters

    No supporters yet this month — yours could be the first name here.