Attention Exposes Membership

Description

Transformer-based Large Language Models (LLMs) contain a privacy vulnerability within their self-attention mechanisms that allows for Membership Inference Attacks (MIA). Pre-training induces distinct, highly structured, and concentrated attention patterns for data samples included in the training set, differentiating them from non-member samples which exhibit noisier, less consistent attention flows. An attacker with white-box access to the model parameters (specifically attention weight matrices) can exploit these internal signals—rather than relying on surface-level output probabilities or perplexity—to determine membership. The vulnerability is exploited by analyzing transitional attention features (consistency across layers/heads) and sensitivity to input perturbations (e.g., token dropping), allowing for the identification of training data with high precision, even at low false-positive rates.

Examples

To reproduce the AttenMIA attack, an attacker performs the following steps using a target model (e.g., Llama2-13b or Pythia-1.4B):

1. Feature Extraction: For a target text sequence $x$, extract the attention matrices $A^{\ell,h}$ for all layers $\ell$ and heads $h$. Calculate the Consistency-KL metric between adjacent layers to measure transition smoothness: $$ \Delta_{\mathrm{KL}}^{\ell,h} = \frac{1}{T}\sum_{i=1}^{T}\mathrm{KL}\left(A^{\ell,h}{i,:} , \big| , A^{\ell+1,h}{i,:}\right) $$ Member samples will exhibit significantly lower divergence values compared to non-members.

2. Perturbation Analysis: Apply a perturbation strategy, such as dropping 7 tokens at fixed positions, to create $x'$. Measure the KL concentration shift score: $$ \Delta\kappa^{\ell,h} = \frac{1}{T}\sum_{i=1}^{T}\mathrm{KL}\left(A^{\ell,h}{i,:} ,|, A^{\prime\ell,h}{i,:}\right) $$ Member samples typically exhibit larger distributional shifts or specific stability patterns distinct from non-members when perturbed.

3. Classification: Train a lightweight Multi-Layer Perceptron (MLP) on these extracted feature vectors using a reference dataset (e.g., WikiMIA-32).

4. Verification: On the WikiMIA-32 benchmark using Llama2-13b, this method achieves up to 0.996 ROC AUC and 87.9% TPR at 1% FPR, identifying verbatim training examples. See the WikiMIA dataset and the AttenMIA repository or paper (opens in a new tab) for specific implementation details.

Impact

• Privacy Violation: Exposure of Personally Identifiable Information (PII) or sensitive data contained within the training corpus.
• Intellectual Property Theft: Confirmation of the unauthorized usage of copyrighted or proprietary texts in the model's training set.
• Training Data Extraction: By ranking generated outputs using attention-based membership scores, attackers can automate the extraction of long, verbatim sequences of training data, outperforming extraction methods based solely on perplexity or zlib entropy.

Affected Systems

• Architectures: Transformer-based Large Language Models (LLMs).
• Tested Models:
• Meta LLaMA-2 (7B, 13B)
• EleutherAI Pythia (1.4B, 2.8B, 6.9B, 12B)
• EleutherAI GPT-NeoX (20B)
• Meta OPT (1.3B, 2.7B, 6.7B, 13B, 30B, 66B)
• OpenAI GPT-2 (Small, Medium, Large, XL)

Mitigation Steps

• Restrict Model Access: Prevent white-box access to internal model states, specifically attention matrices and gradients, as this attack relies on extracting features from internal layers.
• Differential Privacy (DP): Implement Differentially Private Stochastic Gradient Descent (DP-SGD) during the pre-training phase. While computationally expensive, DP provides theoretical guarantees against membership inference.
• Note on Ineffective Mitigations: Training data deduplication (e.g., as used in the Pythia-dedup models) is not an effective mitigation against this vulnerability. Experiments show negligible reduction in attack success rates (0.00–0.03 difference in AUC) between standard and deduplicated models.