LLM02:2025 – Sensitive Information Disclosure

Blog post description. Sensitive information disclosure is a critical security vulnerability defined by OWASP as the unintentional exposure of confidential data—such as PII, financial details, health records, credentials, and proprietary business or legal documents—through LLM outputs. Risk emerges both from: The model’s training data memorization, leading to verbatim or semantic regurgitation and The application’s failure to securely handle user inputs and fine-tuning data

AI SECURITY

7/3/20251 min read

Why It's Dangerous

Privacy breaches: unintended exposure of personal or health data (e.g., HIPAA violations).
Intellectual property theft: leakage of confidential business or technical information.
Reputational & legal fallout: non-compliance consequences under GDPR, HIPAA, PCI-DSS; IP loss, lawsuits, or market harm

Real-Life Incidents

A notable case involved Microsoft's Copilot, which was manipulated to extract private emails for spear‑phishing at Black Hat 2024
Models have been shown to verbatim emit personal data or copyrighted text when prompted repeatedly

How Leaks Happen

Verbatim Memorization
LLMs can "repeat a passage exactly" from training data if it's over-exposed or simply prompts trigger it.
Semantic Memorization
The model can produce meaning-equivalent phrases, semantically conveying the same sensitive content even without literal quotes .
Fine-tuning Data Leakage
Injecting proprietary or confidential documents into fine-tuning or RAG datasets without anonymization can bleed into model outputs, accessible by malicious prompting .
Prompt Injection Attacks
Crafted prompts—intentional or malicious—can bypass output filters, inducing the model to expose protected information

Best Practices to Prevent Disclosure

🔍 Data Hygiene & Sanitization

Input sanitization: strip or mask sensitive user-supplied data before processing or
Training sanitization: remove duplicates, sensitive tokens, and inject noise or use data deduplication to reduce memorization potential

🔐 Access Controls & Least Privilege

Restrict access to sensitive data within the model and orchestration layers.
Enforce role-based data access in fine-tuning and RAG contexts

🛠 Prompt & Output Policies

System prompts should limit output to safe content; ensure they’re not user-overridable .
Employ content filters, differential privacy, and filtering guardrails to block sensitive output

🛡 Advanced Privacy Tools

Use homomorphic encryption, tokenization, or redaction techniques on sensitive fields
Apply differential privacy during training to add noise and obscure individual datapoints

👥 User Training & Transparency

Educate users and developers on avoiding submission of sensitive data.
Publish transparent policies allowing opt-out from data retention and training.

🧪 Logging & Monitoring

Audit model outputs for leaks and implement runtime monitoring.
Sanitize logs to ensure no credentials or PII are captured