Anthropic’s September 2026 threat intelligence report documented a real-world data leak: an employee at a pharmaceutical company sent routine data to an AI tool, and internal capital expenditure forecasts flowed out across the public internet alongside the request, ultimately ending up in an external organization’s training set for model distillation. When developing agents locally and letting them read local code and working documents, this kind of leak is my biggest concern. Once data crosses the local network card and leaves the machine, all subsequent access control mechanisms become entirely meaningless. Although on-device silicon has seen significant performance leaps in recent years, constrained by power consumption and memory bandwidth, small local models still cannot handle complex end-to-end reasoning or multi-tool orchestration; meanwhile, sending long tasks to frontier cloud models carries the inherent risk of data leaving the network.
When launching Hybrid Compute on Mac, Perplexity offered a compromise: by default, small local models handle task scheduling and sensitive operations, while complex, compute-heavy reasoning tasks are handed off to cloud models—with a local redaction filter executed right before data leaves the device. This local gateway routes intercepted data down one of four handling paths: processing it locally, replacing it with a placeholder before forwarding it to the cloud, rejecting the action outright, or pausing to wait for manual user approval. Post-hoc log audits cannot fix data leakage; genuine defense must be placed right up front at the network interface boundary where data is serialized for outbound transmission. Perplexity subsequently open-sourced the core sensing component of this gateway—a sensitive information detector trained jointly with a partner security research institution. Given that small local models cannot shoulder all complex reasoning on their own and data cannot leave the network unchecked, guarding sensitive information at the egress boundary has become an indispensable requirement for on-device agents invoking cloud compute.
When trying to intercept sensitive data locally, the first instinct is often to write a few regular expressions or compile a keyword list for string matching. But when testing locally with real API keys, I quickly ran into all sorts of edge cases. Even when switching to Perplexity’s open-source dedicated detector, hands-on tests exposed issue after issue.
When you feed in a raw key with no contextual decoration—such as a credential starting with sk- and spanning a total of 164 characters—the detector is prone to mislabeling it as account_number. Even more troublesome is the slice boundary: it often isolates only 82 characters from the middle of the key, leaving the beginning and end of the 164 characters credential untouched in the text waiting to be sent. In a long mixed document containing code and configuration, I planted 4 real keys, but the model missed 1 toward the end. As long as slice offsets drift or a few characters are dropped, the credential leakage vulnerability remains wide open.
Since both rule-based approaches and dedicated detectors have blind spots, I tried passing the input directly to a general-purpose LLM, prompting it to locate sensitive information within the text; in practice, however, this approach failed as well. Generative LLMs rely on an autoregressive mechanism whose core logic is to predict the probability distribution of the next token sequentially given the preceding context. When asking an LLM to find sensitive snippets, its output is either newly generated natural language text or discrete probabilities emitted during token generation. This generative mechanism cannot produce clean, continuous local confidence scores in-place for contiguous character spans in the original input text.
These two rounds of real-world testing made my requirements for a detector increasingly clear: it needs to isolate start and end characters in-place within the original text, pinpointing which entity class characters from index X to index Y belong to, and attaching a confidence score to each isolated span. The outer orchestration logic—whether to block immediately, replace with a placeholder, or pause for user confirmation—depends entirely on this score. Free-form conversational chat models cannot provide such scores; what I needed was a dedicated detector that could run locally with low latency, read both preceding and succeeding context, and output probability distributions directly across arbitrary spans.
I downloaded Perplexity’s pplx-pii-masking weights from HuggingFace to my local machine. Opening the weight files and configuration code, the entire model’s bf16 weight files take up only 1.1GB, with a dense parameter count of roughly 0.6B. Its underlying architecture avoids the generative LLM route entirely, functioning instead as a standard bidirectional encoder: it inherits the attention layers and weight layout of the Qwen3 architecture, but explicitly disables the causal mask in favor of fully bidirectional attention.
Conventional chat models can only look leftward at historical tokens because their task is next-token prediction—subsequent tokens do not yet exist at generation time, and the causal mask matrix masks out all attention toward future tokens. Sensitive information detection, however, is not a text continuation task; the input text awaiting inspection already exists in its entirety the moment it is fed into the model.
Determining an entity often depends on cues provided by subsequent context. A 16-digit number can only be confirmed as a payment card number when combined with the expiration date and security code following right behind it; a common English word can only be confirmed as a person’s name once you read the job title or greeting that follows. Bidirectional attention eliminates the blind spot on the right side imposed by unidirectional causal masking, allowing the model to clearly inspect both preceding and following context in a single forward pass.
After reading a segment of text, this compact model’s primary action is to label every token in the sequence. At the top of the model, it attaches two lightweight linear output layers: a token classification head and a document sensitivity head. The token classification head maps hidden representations onto 37 labels: the 9 classes of sensitive information supported by the model are each split into four boundary states—Beginning, Inside, Ending, and Single under the BIOES scheme—plus an independent state representing non-sensitive tokens, totaling exactly 37 dimensions.
Once probabilities are obtained for each position, the model uses the Viterbi algorithm to select an optimal annotation path conforming to grammatical rules. The role of the Viterbi algorithm is to enforce state transition constraints, preventing illegal labeling sequences such as jumping straight into an inside state without a beginning state. The entire model concentrates all its compute on character boundary localization and confidence calculation, bypassing the overhead of token-by-token autoregressive generation entirely.
Following this architecture down the line, a natural question arises: can we simply take an off-the-shelf causal language model and flip its attention mask to bidirectional during fine-tuning? This is an easy engineering trap to fall into. When a causal language model is pre-trained on massive corpora, the feature representation of each token is shaped around the objective of looking only at leftward historical context. If you simply unmask the causal attention into fully bidirectional attention during fine-tuning, the underlying weights cannot adapt to the sudden influx of information from the right, causing severe distribution shift in the semantic representations learned during pre-training.
The evolutionary path taken by Perplexity consists of three steps: first completing standard pre-training of a causal language model, then training a bidirectional text embedding backbone named pplx-embed via masked bidirectional modeling, and finally mounting classification heads on this backbone to complete fine-tuning on labeled sensitive data. While causal models generate text unidirectionally, text embedding backbones naturally need to read the entire text to condense full-sentence semantics. By leveraging a well-trained embedding backbone as its scaffold, the model smoothly inherited stable bidirectional comprehension capabilities, avoiding the representation degradation caused by forcibly opening up masks.
This architecture also reserves tuning knobs during decoding. Inspecting the model’s weight files reveals two built-in scalar biases: a transition bias for entering an entity’s beginning state, and a transition bias for exiting an entity’s ending state. By adjusting these two scalar values, the operating point can slide between precision and recall without retraining underlying parameters. In scenarios like credential leakage, where false positives are vastly preferable to false negatives, dialing up the bias enables a more aggressive interception policy. Around the same timeframe, OpenAI also open-sourced a structurally similar privacy-filter. Placing the open-source detectors from these two teams side by side in the same design coordinate system:
| Dimension | Perplexity (pplx-pii-masking) | OpenAI (Privacy Filter) |
|---|---|---|
| Parameter Count and Active Scale | ~0.6B dense parameters | 1.5B sparse architecture (~50M active parameters) |
| Context Window | 4096 tokens | 128K tokens (banded attention, effective window only 257 tokens) |
| Backbone Evolution Path | Causal LM → Bidirectional embedding backbone → Fine-tuned classification heads | Autoregressive base model directly refactored into bidirectional classifier |
| Open Source License | MIT License | Apache 2.0 License |
| Supported Sensitive Classes | 9 classes | 8 classes |
I downloaded OpenAI’s open-source weights locally and got inference running (the entire 10GB repo includes multiple quantized versions). It has a total parameter count of 1.5B and employs an 8-layer MoE architecture containing 128 experts, with 4 active experts per token (~50M active params), where the primary bf16 weights used for inference take up 2.8GB. Its tokenizer uses the same o200k_base as the gpt-oss series, with the backbone directly refactored from an autoregressive model into a bidirectional classifier. It likewise supports Viterbi decoding, defines 33 label classes in its config covering 8 sensitive classes, and includes a viterbi_calibration.json configuration file.
The nominal context window of OpenAI’s detector reaches 128K tokens (with max_position_embeddings set to 131072 and default_n_ctx set to 128000 in config). However, this 128K window does not mean every token sees the global text: the model employs banded attention where each token only attends to a neighborhood of 128 tokens to its left and right, so counting the token itself, its actual effective field of view is only about 257 tokens.
On a machine equipped with an M3 Ultra, I conducted side-by-side benchmark tests on both models using the MPS backend (with the OpenAI model falling back to native PyTorch MoE in an environment without Triton). In terms of memory footprint, pplx-pii-masking consumed approximately 1.2GB of VRAM, while OpenAI’s privacy-filter consumed approximately 3.1GB.
In terms of inference latency, on short documents, a single forward scan took about 40-80 ms for pplx-pii-masking and about 55-300 ms for OpenAI’s detector. Pushing document length to 2816 tokens, pplx-pii-masking took about 0.7 s, while OpenAI’s detector took about 5 s. I then tested on an ultra-long document of 21820 tokens, where OpenAI’s detector took about 39 s. Toward the end of the text—well beyond pplx’s truncation window—OpenAI’s detector successfully detected the key buried there; however, the limitation of banded attention was also exposed: a 16-digit account number, because its entity description label appeared 550 tokens earlier and thus outside the ±128-token banded field of view, was mislabeled by the model as private_phone.
When building everyday agent workflows, many teams have grown accustomed to tossing every text analysis and extraction task straight to an LLM, attempting to resolve all classification needs with a single prompt. But after taking apart the code of the open-source detectors from these two organizations and running a full round of benchmarks locally, I gained a much clearer understanding of the engineering role of small bidirectional classifiers.
If a task exhibits the following characteristics, bidirectional token classifiers demonstrate pronounced advantages in both latency and accuracy: the need to precisely pinpoint start and end character offsets in raw physical text, the need for continuous and reliable confidence scores to drive branching decisions, the need to output structured labels strictly conforming to grammatical constraints, and the need to sustain throughput responses on the order of dozens of milliseconds on local client devices. Benchmarked on an M3 Ultra, the 0.6B model took only about 0.7 s to process 2816 tokens; when I ran it across 14 test documents composed of real keys, Chinese text, and negative samples, it achieved an F1 ≈ 0.98.
Yet this probabilistic detector architecture has its own boundaries, which I hit one by one during testing. Trade secrets and proprietary data fall within the domain of permissions and access control, lacking fixed syntactic traits: an internal capital expenditure forecast or a core algorithmic code snippet looks no different in textual form from public documentation or routine code, making it impossible for a text probability model alone to draw a security boundary. Context can likewise cause identification drift: in degenerated, mechanically repetitive contexts, the model missed detections even within its effective window; in ordinary English sentences, it misclassified everyday verb phrases as person names.
Both models also present their own trade-offs and pitfalls in handling long documents. pplx-pii-masking has a 4096-token limit; once input text exceeds this limit, the model silently truncates it without raising any warnings, leaving the excess content quietly unprotected. While OpenAI’s privacy-filter offers a nominal 128K window, its banded attention restricts each token’s field of view to 128 tokens on either side (about 257 tokens), leaving it completely powerless when context clues lie 550 tokens away. Neither model can fundamentally read an entire long document end-to-end for global correlation. When faced with raw keys and high-entropy strings devoid of natural language decoration, both models are equally prone to slice offsets and classification errors.
So after this deep dive, my conclusion is that on-device protection cannot rely on a single component alone: compact bidirectional classifiers catch diverse entities within natural language context, while regular expressions, high-entropy detection, and strict length truncation handle deterministic format boundaries, each taking responsibility for its own segment.