One-sentence Summary

DeepMind, ETH Zurich, University of Maryland, and collaborators present the first model-stealing attack that recovers the embedding projection layer (up to symmetries) of transformer models from black-box production language models via standard API access, confirming hidden dimensions of OpenAI's ada, babbage, and gpt-3.5-turbo for as little as $20.

Key Contributions

• The paper presents a model-stealing attack that recovers the embedding projection layer of a transformer black-box production language model from typical API outputs, exploiting features such as logit bias and log probabilities.
• The attack extracts the full embedding projection matrix of OpenAI ada and babbage for under $20 USD, confirming hidden dimensions of 1024 and 2048; it also recovers the exact hidden dimension of gpt-3.5-turbo and estimates full matrix recovery would cost under$20USD,confirminghiddendimensionsof1024and2048;italsorecoverstheexacthiddendimensionofgpt−3.5−turboandestimatesfullmatrixrecoverywouldcostunder2,000 in queries.
• The method reconstructs logits from limited API response information, works under realistic API constraints, and discusses defenses, mitigations, and security implications of API design choices for production language-model services.

Introduction

Major large language models are offered through closed APIs, while their architecture, training details, and weights remain hidden. This creates an important application context: if model weights and internal parameters cannot be inspected directly, an adversary may try to infer them through repeated API queries, a problem studied under model stealing.

Prior high-fidelity model stealing work relied on assumptions that do not hold for production language models, such as ReLU activations, continuous inputs suitable for finite differences, small networks, and access to high-precision logits. These assumptions fail for modern LLMs because they use tokenized text, transformer components, non-ReLU activations, very large scales, and limited output interfaces.

The authors introduce a black-box top-down attack that directly targets the final embedding projection layer of a transformer language model. By exploiting the low-rank structure of the final logit projection and making targeted queries through APIs that expose log probabilities or logit bias, the attack can recover the last weight matrix or infer the hidden embedding dimension. This contribution is meaningful because it shows that partial but precise model information can be extracted from production LLMs, narrowing their black-box security posture and motivating stronger API defenses.

Method

The authors formulate the extraction problem as a sequence of queries to an oracle exposed by a production language model. In the strongest threat model, the adversary can observe the raw logit vector before softmax for an arbitrary prefix $p$p:

where $g_{\theta}(p)$gθ​(p) is the final hidden representation and $\mathbf{W}$W is the output embedding projection matrix. This oracle reveals an $l$l-dimensional logit vector, where $l$l is the vocabulary size, even though the hidden representation is only $h$h-dimensional. The extraction pipeline is organized around this dimensionality gap.

The first stage is hidden dimension recovery. The authors query the model with many distinct prefixes and stack the responses into a matrix

Since each column satisfies $\mathcal{O}(p_i)=\mathbf{W}g_{\theta}(p_i)$O(pi​)=Wgθ​(pi​), the response matrix factorizes as

where $\mathbf{H}$H has hidden dimension $h$h. Thus, under generic full-rank assumptions on $\mathbf{W}$W and the hidden-state matrix, the rank of $\mathbf{Q}$Q equals $h$h. In practice, because production logits may be low precision, the authors do not use exact rank. They instead estimate a numerical rank by sorting the singular values

and selecting the index where the ratio $\lambda_i/\lambda_{i+1}$λi​/λi+1​ is maximal. This identifies the transition from large meaningful singular values to small numerical-noise singular values. As shown in the figure below, the recovered hidden dimension corresponds to a sharp gap in the singular-value spectrum.

The same phenomenon can also be observed by increasing the number of queries and tracking how many numerically significant singular values appear. When the number of queries is below the hidden dimension, the response matrix appears to add independent directions, but once the number of queries exceeds $h$h, the number of large singular values stops growing. As shown in the figure below, this saturation makes the effective hidden dimension identifiable.

Once the hidden dimension $h$h is known, the authors move to full output-layer extraction. They apply a singular value decomposition to the response matrix:

Under the same rank assumptions, the leading singular structure directly encodes the final projection matrix up to an invertible hidden-space transformation. Specifically, the matrix

satisfies

for some $h\times h$h×h matrix $\mathbf{G}$G. This gives a high-fidelity reconstruction of the final layer, but only up to a hidden-space symmetry. The authors show that exact recovery of $\mathbf{W}$W is impossible under additional transformer-like residual assumptions, because the transformation $\mathbf{G}$G cannot be uniquely identified from logits alone. To evaluate the quality of extraction, they solve a least-squares problem

and then compare the transformed reconstruction with the true output matrix.

The authors then relax the full-logit oracle and address the APIs exposed by production models. Many such APIs return only the top $K$K log probabilities after adding user-controlled logit biases:

Because softmax is invariant to additive shifts in logits, the goal becomes recovering logits relative to a reference token rather than absolute logits. For a top-$K$K API, the authors select a reference token that is naturally high probability and then bias several target tokens into the top $K$K in a single query. If token $R$R is used as the reference and token $i$i is boosted by a large bias $B$B, then the observed top-token log probabilities reveal the relative logit difference:

By cycling through token blocks, the adversary can recover the full relative logit vector. The baseline cost is approximately $1/(K-1)$1/(K−1) queries per extracted logit, since one of the $K$K returned tokens serves as the reference. The authors also describe an extension in which each returned log probability is treated as a linear constraint on the original logits, reducing the query cost toward $1/K$1/K.

For APIs that generate multiple tokens or expose log probabilities for prompt tokens, the authors exploit repeated token generation. They force the model to emit a repeated token $x$x by giving it a large logit bias, while giving slightly smaller biases to other tokens that remain within the top outputs. This allows the adversary to inspect the top-$K$K log probabilities for each generated position and recover logit information for many positions from a single query. The resulting query cost depends on the expansion length $m$m, and the token cost increases with both expansion and API overhead.

The authors further consider stricter APIs that expose only the top log probability and restrict each logit bias to a binary value, such as $0$0 or $-1$−1. Even in this setting, they show that information can be extracted by comparing the top token’s log probability under no bias and under a small negative bias on a target token. The difference between these two observations depends on the target token’s original log probability, so repeated queries can recover logit information one token at a time. This method is less numerically stable than the top-$K$K biased-logit attacks, but it demonstrates that extremely restricted APIs can still leak structural model information.

When log probabilities are not available at all, the authors describe a logprob-free approach based on binary search over logit biases. For each token, the adversary increases or decreases its bias until the model’s argmax output changes. The bias value at which this transition occurs provides a relative estimate of that token’s logit. Although straightforward token-by-token binary search is expensive, the authors note that multiple token biases can be modified in parallel, and the observed argmax token provides partial information about which token has the largest biased logit. This parallelization improves query efficiency, though it remains more costly than attacks that observe log probabilities.

The method also leads naturally to defense considerations. The extraction pipeline depends on exposing information that reveals the geometry of the final hidden dimension. The authors discuss removing logit bias entirely, replacing it with a block list, preventing simultaneous use of logit bias and log probabilities, adding noise to output logits, rate limiting biased queries for a given prompt, or modifying the final layer architecture so that the observable hidden dimension does not directly reveal the original $h$h. One architectural approach splits the final projection through an intermediate expansion with a nonlinearity, while another expands the output matrix with extra low-magnitude orthogonal directions and injects noise into the hidden vector, causing the observed logit geometry to appear wider than the true hidden dimension.

Experiment

The experiments evaluate whether model weights and full logit vectors can be recovered through API queries, using cost metrics that distinguish monetary token use from query latency and numerical agreement between reconstructed and true parameters. They show that theoretical extraction attacks can reconstruct model components accurately under appropriate assumptions, though some stronger variants are impractical due to numerical instability, while more robust attacks remain efficient. In real production models, the attacks successfully recover hidden dimensions and full layers, yielding parameters nearly identical to the original weights and confirming accurate model sizes where permitted.

The table summarizes different API access settings for model extraction, ranging from full logit access to restricted interfaces with logit bias. These settings are presented with distinct motivations, spanning teaching, practical deployment, and stricter API design. Full logit access is described as a basis for subsequent attacks. Top logprob access with logit bias is associated with current LLM APIs. A no-logprob interface with logit bias is framed as a possible future constrained API.

The experiment shows that the attack recovers hidden dimensionality closely across open-source models of different sizes and precisions. It also reconstructs the output projection matrix with very low error after affine alignment, indicating that the stolen representation is close to the true weights. The reconstruction error is much smaller than the error of a randomly initialized matrix. Stolen size estimates are very close to the models known hidden dimensions. Recovered projection matrices have low RMS error and are far more accurate than random initialization.

The attack successfully recovered hidden model dimensions for all five tested production models, and the reported sizes were confirmed correct even when withheld. Weight extraction was highly accurate for the models where full layer recovery was completed, though the effort and cost grew substantially for larger GPT-3.5 variants. Hidden dimension extraction worked for all five models and matched the true sizes. Extracted weight matrices aligned very closely with actual weights, as shown by low RMS error. Large GPT-3.5 extraction attempts required much higher query counts and cost, and some weight extraction was not run for security reasons.

The results show that attacks with stronger logprob access generally recover more bits of the true logit vector. The four logprob top five method gives the best precision with the fewest queries per logit, while other methods approach similar precision only at higher query cost. Among logprob free attacks, the one of n method achieves strong precision efficiently, whereas binary search needs many more queries for much lower precision. The four logprob top five attack is the most precise and query efficient method. The five logprob top five attack performs worse in practice despite learning more logprobs per query, showing numerical instability can reduce fidelity and efficiency. The strongest logprob free attack recovers most of the logit vector with relatively few queries and is the best alternative when logprobs are unavailable.

The experiments evaluate model extraction under several API access settings, from full logit access to constrained no-logprob interfaces with logit bias. They first validate the attack on open-source models by accurately estimating hidden dimensions and reconstructing projection matrices, and then extend the evaluation to production models, where hidden sizes are consistently recovered and extracted weights remain close to true parameters despite higher cost for larger systems. A separate comparison of extraction strategies shows that access to more logprob information generally improves fidelity, with a top-five method providing the best query efficiency and an efficient alternative remaining viable when logprobs are unavailable.