Benchmarking OpenAI’s Privacy Filter: What it gets right, and where PII detection still needs real data

**Executive summary:** This report benchmarks OpenAI Privacy Filter (OPF) against Tonic Textual for token-level PII detection across web crawl, EHR notes, legal documents, and ASR transcripts, concluding that OPF is a strong, permissively-licensed base encoder with conservative default thresholds (low recall) and that domain-specific labeled training data—not the model itself—is the primary determinant of production performance; experiments (Viterbi calibration, hidden-state UMAP, and fine-tuning with 100–10k documents) show fine-tuning can make OPF competitive in some domains but struggles on web-scrape long-tail cases.