Start Free Security Scan

RAG Security Benchmarks

This page describes RAGSec’s approach to benchmarking RAG security posture — what we measure, how measurements are defined, and what we will publish as our dataset grows. No numeric benchmarks are published here yet; this page is our reference for how they will be constructed and reported.

Hero image: RAG Security Benchmarks — Benchmarks | RAGSec

What RAGSec benchmarks will measure

What RAGSec benchmarks will measure — Benchmarks | RAGSec

RAGSec benchmark reports will cover three areas: configuration posture across scanned environments, retrieval isolation test outcomes, and remediation effectiveness. All measurements will be reported in aggregate and anonymized. Individual organization data will never be included in public benchmarks without explicit opt-in.

Configuration posture metrics

  • Credential scope failure rate: Fraction of scanned environments where production retrieval paths use write-capable or admin credentials
  • Tenant field coverage rate: Fraction of scanned records with a tenant or owner field present
  • Missing payload index rate: Fraction of tenant fields used for isolation that lack a payload index (Qdrant) or equivalent enforcement mechanism
  • Sensitive content indicator rate: Fraction of sampled records containing high-confidence secrets, PII indicators, or prompt-injection patterns
  • Metadata completeness score: Composite score of owner, source, and classification field coverage

Retrieval isolation test outcomes

  • Cross-tenant isolation pass rate: Fraction of active isolation tests that return no cross-tenant records across all tested scenarios
  • Scenario-specific pass rates: Direct, paraphrased, multi-turn, namespace manipulation, and fallback scenarios reported separately
  • Confirmation rate: Fraction of posture findings that, when actively tested, are confirmed as exploitable
  • Time to confirmation: Elapsed time from scan start to confirmed evidence package

Remediation effectiveness

  • Mean time to remediation: Time from confirmed finding to verified fix
  • Patch acceptance rate: Fraction of RAGSec-generated patch candidates that were accepted by engineering teams without modification
  • Regression rate: Fraction of confirmed findings that reappear in subsequent scans after a reported fix

Benchmark methodology commitments

Benchmark methodology commitments — Benchmarks | RAGSec
  • All numeric claims will include sample size, collection period, and confidence interval
  • Methodology will be published alongside any benchmark figures
  • We will clearly distinguish observed rates from norms or recommendations
  • We will not publish findings attributed to any organization without explicit consent
  • Benchmark reports will be versioned and previous versions will remain accessible

Current status

RAGSec is in early access. We are collecting posture and test outcome data. We will publish the first benchmark report when our dataset reaches a size sufficient to produce statistically meaningful aggregates. Interim findings will be published on the blog as case studies and methodology notes — without organization-identifying detail.

If you are a researcher working on RAG security evaluation methodology or interested in collaborating on benchmark design, reach out through the contact page.

Testing methodology