Model Fingerprinting in LLM-Generated Text

Model Fingerprinting in LLM-Generated Text.

Why every LLM leaves a stylometric signature in what it writes

Every large language model writes with an accent, and that accent survives translation across prompts, topics, and even languages. This is not a defect anyone is going to patch out. It comes from how the model was trained and how it samples the next token, which means the fingerprint is baked in at a structural level, not a bug sitting in a changelog waiting for a fix.

The strange part is that this shouldn't work, at least not on first glance. Nearly every frontier model runs on some version of the transformer architecture, sharing the same basic mechanism for attention and next-token prediction. The shared substrate doesn't erase the model-specific fingerprint. If anything, it makes the fingerprint more interesting, because the differentiation is happening upstream of architecture, in the choices made during training and deployment.

A study led by Zaitsu and published in Frontiers in AI put this to the test directly FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting. Researchers generated 300 texts using six LLMs (ChatGPT running GPT5, Claude 3.5, Gemini, Microsoft Copilot, Llama 3.1, and Perplexity) and distinguished them using stylometric features: function-word unigrams, part-of-speech bigrams, and phrase patterns; Random Forest achieved macro F1 scores exceeding 0.95, reaching 1.00 for several individual models FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting. That's not a coin flip with a thumb on the scale. That's near-total separability.

The one model that gave the classifier any trouble was Llama 3.1, which showed real overlap with other models in the study's UMAP visualization. But overlap isn't invisibility. Even the hardest case in the dataset was still detectable, which tells you something important going into the rest of this piece: there is no model on the market today that fully escapes stylometric identification, not even the one built to be more permissive and unstructured in tone.

Humans, for what it's worth, are bad at this particular game. Machines don't share that limitation, and that asymmetry is why fingerprinting belongs in the same conversation as infrastructure, security, and compliance, not proofreading.

Where does the accent actually come from? Training data composition sets the base vocabulary and syntactic habits a model defaults to. Sampling strategy (temperature, top-p, nucleus cutoffs) shapes how predictable or varied the token choices are moment to moment. RLHF tuning imprints the preferences of whoever did the reinforcement learning, often nudging a model toward hedging, disclaimers, or a particular register of politeness. Tokenization choices affect how words get split and reassembled, and this is visible statistically even when a reader would never notice it directly.

What follows here is really three separate but related problems: what the fingerprint is, how someone finds it (passively, by reading finished text, or actively, by interrogating a live system), and how a fingerprint gets planted on purpose through watermarking. Treating these as one blurry topic is where a lot of practitioner writing on this subject goes wrong.

Model fingerprinting versus AI detection

Three questions get asked interchangeably in casual conversation, and they should not be. The first is whether a machine produced a given piece of text. The second, model fingerprinting, asks something narrower and more useful: which specific model or version produced this output. The third, watermarking, is a design choice made before the text ever leaves the model, embedding a marker that can be checked later.

It's a post-hoc estimate, run on text that already exists, with no cooperation required from whatever produced it. It answers a binary question, AI or not AI, and it attributes nothing to a specific system. Fingerprinting is the finer instrument. It doesn't ask whether a machine wrote something, it asks which machine, and that distinction matters enormously to a publisher trying to verify a vendor's claims or a legal team trying to establish where a derivative model came from.

Fingerprinting itself splits into two operating modes. Passive fingerprinting works on text that already exists, pulling latent statistical signatures out of finished output. Active fingerprinting works on a live system, sending it crafted queries and reading the pattern of responses.

Because the forensics literature treats them as distinct tasks, attribution and fingerprinting form a related pair of sub-problems that need to be named precisely. Attribution asks which model, from a defined candidate set, most likely produced a given output. Fingerprinting, in the forensic sense researchers at UC San Diego use, asks whether two separate endpoints are running the same underlying configuration, a subtly different question with different evidentiary standards. Both get addressed in the black-box forensics work from White, Jafari, and Berg-Kirkpatrick at UC San Diego, which treats attribution (mapping output to a base model from a candidate set) and fingerprinting (asking whether two endpoints share the same configuration) as sub-problems.

There's also a commercial angle that gives all this technical apparatus its urgency. A survey out of Zhejiang University, led by Xu and colleagues, frames fingerprinting explicitly as an intellectual property question LLM Fingerprinting via Semantically Conditioned Watermarks. Training a frontier model costs an enormous amount of compute and data curation, and if a competitor fine-tunes on top of a leaked or stolen base model, fingerprinting is the mechanism by which the original developer proves the derivative violates a license LLM Fingerprinting via Semantically Conditioned Watermarks. This is not an academic curiosity. It's the same logic that underlies watermarking on currency or provenance tracking in fine art, applied to weights and token distributions instead of ink and paint.

One clarification before going further: the word "fingerprint" in this literature sometimes means the latent stylistic trace, and sometimes a deliberately planted ownership marker, and both meanings appear throughout this piece. Keep those two senses separate as the rest of this piece moves from detection into embedding. Active: sending crafted queries to a live system and analyzing responses, the approach used by LLMmap (Pasquini, Kornaropoulos, Ateniese; USENIX Security 2025) (FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting, LLM Fingerprinting via Semantically Conditioned Watermarks).

How passive stylometric fingerprinting works in practice

Passive fingerprinting reads finished text the way a document examiner reads a ransom note, looking for the tics an author can't help but repeat. Function-word unigrams (the distribution of small, high-frequency words like "the," "of," "which") shift depending on what a model was trained on and how it samples. Part-of-speech bigrams capture grammatical transition habits that differ measurably across model families. Phrase patterns pick up longer structural regularities specific to how a given model tends to build sentences. Punctuation patterns matter too. The black-box forensics literature (White et al.) flags character-level punctuation patterns, run through a TF-IDF weighting, as a genuinely useful discriminating signal.

Not every stylometric feature earns its keep, though. That matters for anyone building a detection pipeline, because throwing every conceivable stylometric feature at a classifier wastes compute on signal that isn't there.

A different approach skips surface style altogether and goes after the probability dynamics of the text. The DWT-Fusion framework, developed by Özdaş and Osmanoğlu at Ankara University, treats the sequence of token-level log-probabilities as a one-dimensional waveform and runs it through discrete wavelet analysis, training-free. Instead of averaging probability across an entire document, the way older detection methods do, it looks at local and multiscale variation, catching the small fluctuations in predictability that a global average would smooth away. On benchmark datasets, the best single wavelet configuration reached an AUROC of 0.9872 on HC3, 0.8185 on M4, and 0.7138 on MAGE pmc.ncbi.nlm.nih.gov DWT-Fusion: A Signal-Based Framework for Training-Free LLM-Generated Text Detection. A calibration-weighted voting ensemble pushed those numbers higher still, to 0.9919, 0.8477, and 0.7471 respectively DWT-Fusion: A Signal-Based Framework for Training-Free LLM-Generated Text Detection.

Zero-shot detection methods deserve a mention here too, because they're widely used and widely misunderstood. DetectGPT works off probability curvature, and Binoculars works off cross-model scoring, and both need no training data to run. That's their appeal. It's also their weakness: both degrade meaningfully once the text has been paraphrased, a structural vulnerability that resurfaces later in this piece when the conversation turns to evasion.

A more unusual angle comes from Stanford, where Kuditipudi and colleagues built a method around what they call palimpsestic memorization Blackbox Model Provenance via Palimpsestic Membership Inference. Instead of asking what a piece of text looks like stylistically, the method asks whether a derivative model, or text produced by it, correlates statistically with the order in which the original training data was presented. Run across fine-tunes of Pythia and OLMo base models ranging from 1B to 12B parameters, the method produced p-values on the order of 1 × 10⁻⁸, and in the observational setting, it could distinguish provenance from as few as a few hundred tokens Blackbox Model Provenance via Palimpsestic Membership Inference. That's a strikingly small sample size for a signal that strong, and it points to something important: the fingerprint isn't only legible in surface style or token probability. It's legible three layers down, in the deep statistical residue of training itself, which means an adversary who scrubs surface style clean can still leave a trace at a level they never thought to touch. (Frontiers in AI, June 2026) and related work. SHAP analysis (Zaitsu et al.) identified which specific features drove separation between models (not all stylometric features are equally informative, some are model-discriminative, others are not).

Active fingerprinting: probing live systems to identify the model behind an endpoint

Passive methods work on text someone hands you. Active methods go looking for the model themselves, and they do it by talking to it. LLMmap, again from Pasquini, Kornaropoulos, and Ateniese at USENIX Security 2025, sends carefully constructed queries at a live system and reads the pattern of responses to identify the exact model version behind the endpoint FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting LLM Fingerprinting via Semantically Conditioned Watermarks. It's the first-generation tool built specifically for LLM-integrated applications, the kind of product where a company builds a chat interface on top of somebody else's API and never discloses which model is actually doing the work FDLLM: A Dedicated Detector for Black-Box LLMs Fingerprinting LLM Fingerprinting via Semantically Conditioned Watermarks.

The UC San Diego forensics team took this further with conversational agents specifically. Their attribution classifiers identified the base model behind a deployed agent with 98% accuracy, using only stylistic and n-gram features pulled from a few turns of ordinary, non-adversarial conversation. On harder cases, cross-encoder fingerprinting against a system prompt it had never seen before scored an AUC of 0.768 and an F1 of 0.703 Black-Box Forensics for Conversational LLM Agents. Aggregating across 50 interaction conversations for a single target agent pushed AUC up to 0.943 Black-Box Forensics for Conversational LLM Agents. Notably, the method could tell apart models from the same family at different sizes: the fingerprint survives even when the comparison is within a single lineage, not across competing labs. And the probing itself stays covert by design: it uses ordinary, non-adversarial dialogue, so a target system has no way of knowing it's being investigated.

Behavioral signatures appear in other places too, not just style and phrasing. A 2026 paper (arXiv:2602.09434) builds an attribution method around refusal patterns specifically, the way a model declines a request. Refusal behavior gets shaped heavily by RLHF, and different labs tune that behavior differently, which means the shape of a "no" carries as much signal as the shape of a "yes."

DuFFin, published at EACL 2026 by Yan, Tang, Yan, and Dai, works both levels at once, model-level and output-level attribution, built specifically for IP protection cases. There's also a very practical use case tied to all of this that any team paying for API access should care about. A 2025 paper (arXiv:2504.04715) audits the question of model substitution directly, that is, whether an API provider is quietly swapping the backend model serving a request without telling the customer. Active fingerprinting is the mechanism by which a buyer can check whether they're getting the model they're paying for. FDLLM, presented at IEEE TrustCom 2025, is another operational tool built for exactly this kind of black-box detection work.

Watermarking: embedding the fingerprint before the text leaves the model

Watermarking flips the entire premise. Instead of hunting for a signature that happens to exist, it puts one there on purpose, before the text ever reaches a reader. It's a deliberate act, not a residue. Every watermarking method since gets measured against that baseline.

CurveMark represents where the field has pushed that idea since. It embeds a watermark through entropy-aware, semantically informed token selection, hitting 95.4% detection accuracy while raising perplexity by less than 1.3, a remarkably small quality cost for that level of detectability. Its channel capacity utilization runs between 85% and 89%, and under adversarial perturbation, its information retention ranges from 72% to 94% pmc.ncbi.nlm.nih.gov. The gap between CurveMark's best case and its worst case is wide enough that a production deployment can't simply assume the average will hold under attack.

Historically, watermarking has forced a trade-off between staying invisible and staying robust under manipulation, better at one usually meant worse at the other. A 2026 ICLR paper pushes back on that assumption directly, introducing domain-specific watermarks designed to hold both properties at once across different deployment scenarios, suggesting the trade-off was never as fixed as the field assumed.

A real vulnerability sits in most current systems. Detection typically requires a secret key generated at the same time as the watermark, and a 2025 paper in the ACL Computational Linguistics track points out that this creates a forgery risk anywhere the detection pipeline is exposed publicly. If the key leaks or gets reverse-engineered, an adversary doesn't just evade the watermark, they can forge it, planting a false attribution on text the original model never touched.

A second gap appears in mixed-source documents. Most watermark detection tools label an entire document as watermarked or not, a binary call that breaks down when only sections of a document are LLM-generated, a common real-world scenario such as AI-assisted news manipulation.

For teams trying to evaluate any of this independently, MarkLLM, an open-source toolkit from Pan and colleagues presented at EMNLP 2024, gives a shared evaluation harness rather than forcing every lab to build test infrastructure from scratch. And the training side keeps moving too: a 2025 ICLR workshop paper (Xu, Yao, and Liu) explores using reinforcement learning as the training signal for watermark quality itself, an emerging direction rather than a settled method. The PAN 2026 shared task sets the adversarial bar the whole field should be building toward, benchmarking watermark robustness against unknown, automated obfuscation attacks and scoring systems on inconspicuousness and robustness against modifications nobody designed the watermark to anticipate. (ICML 2023) baseline: foundational watermark for LLMs, the reference point against which all subsequent methods are measured (established the token-partition approach to embedding).

Detector failure modes and evasion in practice

Supervised detectors, the kind trained as a straightforward classifier on labeled human-versus-AI datasets, share a common failure mode: they get brittle the moment real-world text drifts from whatever distribution they were trained on. That's precisely the failure that led OpenAI to shut down its own deployed detector rather than keep shipping something unreliable. Ghostbuster, from Verma, Fleisig, Tomlin, and Klein at NAACL 2024, represents the supervised approach pushed about as far as it goes, and it's still bound by the same generalization ceiling.

A 2025 paper (arXiv:2510.08602) reframes the whole problem in a way worth taking seriously. As AI-generated text becomes the majority of what's on the internet, human writing becomes the statistical outlier, the rare signal a detector needs to catch, rather than the default it needs to confirm departure from. That's a genuine inversion of how detection has been framed since the field started, and it changes what "success" should even mean for a detector going forward.

Evasion, meanwhile, isn't hypothetical anymore. Work presented at UTS for the ELOQUENT 2026 Voight-Kampff shared task shows that structural shifts in AI writing, sentence architecture, paragraph organization, bypass detectors that otherwise perform well. That's a meaningfully different threat than word-swap evasion: detection is already documented as beatable at the level of syntax, not just vocabulary. On a controlled, balanced dataset, GPT-4 hit 98% classification accuracy in a 2026 Frontiers in AI study, but controlled corpora aren't production conditions Black-Box Forensics for Conversational LLM Agents. Real-world text that's been mixed, edited, and partially rewritten performs far worse, and that gap between lab numbers and deployed performance is the one that actually matters to anyone shipping a detection product Black-Box Forensics for Conversational LLM Agents. Paraphrasing remains the cheapest attack available: DetectGPT and Binoculars both degrade against it, and an adversary needs no training data or technical sophistication to run a paraphrase pass.

The structural evasion that matters most for the rest of this piece, though, is routing. When content passes through several models in sequence rather than one, no single model's fingerprint dominates the output. The AdaptPrint paper (arXiv:2608.22213, 2026) documents that production API stacks are already built this way, routing across multiple backend models, which makes attribution to any single model close to impossible for a detector built around single-model assumptions. What defeats a fingerprinting detector, though, turns out to be the same architecture that improves writing quality, which is where this piece turns next.

Multi-model architectures as both an evasion mechanism and a quality mechanism

Routing content across multiple models does two things at once, and they pull in the same direction rather than against each other. It complicates attribution, since no single fingerprint dominates the output, and it improves quality, since no single model's specific failure modes get to dominate either.

There's empirical support for both halves of that claim. Some of the more interesting architecture in this space builds the adversarial relationship directly into the system. In the LLM-GAN architecture, a single LLM instantiated as both Generator and Detector runs an adversarial loop, generation against detection, with feedback from each round informing the next, and a Reflector agent's self-reflection cycles forcing the system to refine its own explanations rather than just its raw output.

This kind of layered, multi-agent setup is becoming less of a novelty and more of a baseline. Reporting from Lyzr describes agentic generation as the direction production content operations are heading: agents coordinate planning, generation, distribution, and monitoring as a connected pipeline, while the human role shifts toward strategy and creative direction rather than line-level drafting. Every added step in that pipeline is a place where a different model might do the work, and fingerprint complexity compounds with each one. AdaptPrint's account of API layering makes the practical consequence explicit: multi-LLM routing on cloud infrastructure makes it close to impossible for a downstream fingerprinting system to pin output on one model, which is really the point where single-model detection stops being a viable premise for any production-scale system.

The same logic scales down to the sentence level in council-style writing architectures, where drafts get generated, critiqued, and voted on line by line across many models until the output converges, sometimes over several rounds of iteration built into the design on purpose rather than treated as inefficiency. That kind of recursive, multi-model convergence mirrors what the research on ensembles keeps finding: more voices in the room, properly structured, produce output that is both better written and harder to trace to a single source. Whether a given team is optimizing for evading detection or simply for better prose, the underlying mechanism turns out to be the same one. MLSDet (multi-LLM statistical deep ensemble for Chinese AI-generated text detection, ICASSP 2025) and multi-LLM text summarization research (Fang et al., arXiv:2412.15487, 2024): empirical confirmation that ensemble outputs are harder to attribute and better in quality.