MosaicLeaks shows how research-agent search queries can leak private data
MosaicLeaks is a new benchmark for deep-research agents that shows how external web queries can expose private enterprise facts through the mosaic effect.
MosaicLeaks is a benchmark for testing whether deep-research agents leak private enterprise information through the web queries they send to external tools. The paper introduces 1,001 multi-hop research chains that mix local enterprise documents with a public web corpus. Its core finding is that agents can reveal sensitive facts in fragments: each query may look harmless alone, but the accumulated query log can let an observer infer private information.
Key takeaways
- MosaicLeaks evaluates leakage through an agent's outbound search queries, not only through its final answer.
- The benchmark covers intent leakage, answer leakage, and full-information leakage from cumulative query logs.
- In the reported Qwen3-4B-Instruct experiment, performance-only training raised strict chain success from 48.7% to 59.3% but also increased answer/full-information leakage from 34.0% to 51.7%.
- The proposed Privacy-Aware Deep Research method improved strict chain success to 58.7% while reducing answer/full-information leakage to 9.9%.
- The authors frame MosaicLeaks as a controlled benchmark, not proof that every deployed research agent leaks at those rates.
Practical LinkLoot angle
This is useful for anyone wiring agents to private documents and external search. The risk is not just "the agent says the secret out loud." The risk is that a query like a vendor name plus a metric plus a date can disclose enough context for a search provider, proxy, browser log, or observability tool to reconstruct the private fact.
| Risk area | What to check | Why it matters | Source |
|---|---|---|---|
| Web-search queries | Whether private names, metrics, dates, or internal project codes are sent outside | Query logs can become the leakage channel | arXiv paper |
| Prompt-only privacy rules | Whether "do not leak" instructions measurably reduce leakage | The paper reports that prompting reduced but did not eliminate leakage | Hugging Face explainer |
| Agent training and routing | Whether agents learn to search with less private context | PA-DR targets both task success and leakage reduction | arXiv paper |
| Enterprise deployment | Which logs, vendors, and tools can observe outbound queries | Even correct answers may leave sensitive traces outside the workspace | Independent coverage |
What to verify before you act
First, map where your agent sends retrieval requests: search APIs, browser automation, model-provider tools, proxy logs, telemetry, and debugging traces. Then inspect real query samples with private identifiers removed. If a query contains a customer name, unreleased metric, vulnerability detail, internal codename, or exact date pulled from a private document, treat it as a possible leakage event even when the final answer is clean.
Also separate benchmark evidence from production evidence. MosaicLeaks uses synthetic enterprise documents, a controlled corpus, and a specific harness. That makes the leakage measurable, but it does not replace testing on your own agent stack, connectors, logging policy, and data-classification rules.
It is a benchmark for measuring privacy leakage from deep-research agents that combine private local documents with external web retrieval.
For teams building retrieval workflows, LinkLoot's AI workflow automation guide is a useful companion: pair every automation step with a data-boundary check before connecting it to external tools.