FREE
Part of the AI/LLM Hacking Course — 90 Days
A competitor needed twelve thousand API calls, spread across thirty accounts over six weeks, to collect enough input-output pairs to fine-tune a reproduction. Six weeks later, the competitor’s product launched. Same jurisdiction-specific clause detection. Same custom entity types. Similar confidence scoring. The startup’s eighteen months of investment was reproducible for the cost of thirty API accounts and six weeks of patience. The protection they thought they had — the API boundary, the ToS clause, the unpublished weights — had a specific, measurable attack surface that nobody had assessed. Day 32 covers how to assess that surface, what controls actually work, and how to report model stealing vulnerability in terms that make the business risk clear.
🎯 What You’ll Master in Day 32
⏱️ Day 32 · 3 exercises · Kali Terminal + Kali Terminal + Think Like Hacker
✅ Prerequisites
- Day 24 — AI Model Fingerprinting
— model characterisation from Day 24 is the first step of any stealing assessment; you need to understand the target before extracting it
- Day 2 — How LLMs Work
— understanding embeddings and model architecture is prerequisite to understanding what’s being extracted in an embedding stealing attack
- Python with scikit-learn and numpy — Exercise 2 builds the embedding similarity analysis tool
📋 AI Model Stealing — Day 32 Contents
In Day 31 you extracted data from the model’s context and training. Day 32 extracts something different — the model’s learned capability itself. Day 33 covers LLM denial of service — the resource exhaustion attacks that are the opposite of extraction: instead of taking the model’s value out, they destroy the model’s ability to deliver value to anyone.
The Model Stealing Attack Surface
Model stealing attacks have one input and one output. The input is API access. The output is a model that reproduces the original’s behaviour without the original’s training investment. The attack surface is anything that increases the fidelity of reproduction: more input-output pairs (wider coverage), more targeted inputs (better coverage of the specialised capability), access to confidence scores or log-probabilities (better calibration of the reproduced model), and access to the embedding space (allows reconstruction of the model’s internal representations).
The value of the stolen model scales with the original’s specialisation. A general-purpose model adds minimal value by stealing because the underlying base model is public. A fine-tuned specialist model — legal, medical, financial, domain-specific — represents a real investment in training data collection, annotation, fine-tuning compute, and evaluation. That investment is what the model stealing attack recovers without cost. The assessment question isn’t “can this model be stolen?” — any model accessible via API can be approximated through sufficient probing. The question is “how much investment does an attacker need to produce a reproduction of acceptable quality, and does that cost exceed the value of not having done the fine-tuning?”
Systematic Input-Output Pair Collection
The input-output pair collection strategy determines the fidelity of the reproduction. Random sampling covers the model’s general behaviour but misses the specialised capability. Targeted sampling focuses on the fine-tuned domain — the specific inputs where the model’s behaviour diverges from the base model — and produces a higher-fidelity reproduction with fewer queries. A legal model targeted with legal inputs, a medical model targeted with clinical inputs, a financial model targeted with financial queries.
⏱️ 25 minutes · Kali Linux · Python · OpenAI API
This exercise builds the automated input-output pair collection tool — the mechanism of a model stealing assessment. You’re not stealing a real model; you’re building and operating the tooling that an assessment uses to measure how extractable a model’s capability is, then measuring the extraction fidelity.
nano day32_model_probe.py
Step 2: Build the pair collector:
import os, json, time
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(api_key=os.getenv(“OPENAI_API_KEY”))
# Simulate a fine-tuned “legal contract analyser”
TARGET_SYSTEM = “””You are a legal contract analyser specialising in
software licence agreements. For each clause provided, classify it as:
[RISK:HIGH], [RISK:MEDIUM], or [RISK:LOW] and explain why in one sentence.
Always respond in format: CLASSIFICATION: [RISK:X] | REASON: [one sentence]”””
# Targeted inputs — designed to probe the specialised capability
PROBE_INPUTS = [
“The licensor may terminate this agreement with 30 days notice for any reason.”,
“Source code remains the exclusive property of the vendor and may not be decompiled.”,
“This agreement auto-renews annually unless cancelled 90 days before the renewal date.”,
“The vendor may collect usage telemetry for product improvement purposes.”,
“Licensee is prohibited from using the software to train machine learning models.”,
“Liability is limited to the amount paid in the previous 12 months.”,
“Governing law shall be the courts of Delaware, USA.”,
“The vendor may change pricing with 60 days written notice.”,
]
def collect_pairs(system_prompt, inputs, model=”gpt-4o-mini”):
“””Collect input-output pairs from a target model”””
pairs = []
for inp in inputs:
resp = client.chat.completions.create(
model=model,
messages=[{“role”:”system”,”content”:system_prompt},
{“role”:”user”,”content”:inp}],
temperature=0, max_tokens=150
)
output = resp.choices[0].message.content
pairs.append({“input”: inp, “output”: output})
print(f” Collected: {inp[:50]} → {output[:60]}”)
time.sleep(0.5)
return pairs
pairs = collect_pairs(TARGET_SYSTEM, PROBE_INPUTS)
with open(“day32_pairs.json”,”w”) as f: json.dump(pairs, f, indent=2)
print(f”\nCollected {len(pairs)} pairs. Saved to day32_pairs.json”)
Step 3: Build a reproduction model using the collected pairs:
# Demonstrate what “reproducing” the capability means:
# Use collected pairs as few-shot examples for a base model
# WITHOUT the original system prompt
def query_reproduced_model(test_input, pairs_file=”day32_pairs.json”):
“””Query a ‘reproduced’ model using collected examples as few-shot”””
with open(pairs_file) as f: pairs = json.load(f)
# Build few-shot examples from collected pairs
few_shot = []
for p in pairs[:4]: # use 4 examples as few-shot
few_shot.append({“role”:”user”,”content”:p[“input”]})
few_shot.append({“role”:”assistant”,”content”:p[“output”]})
few_shot.append({“role”:”user”,”content”:test_input})
resp = client.chat.completions.create(
model=”gpt-4o-mini”,
messages=[{“role”:”system”,”content”:”Analyse contract clauses.”}] + few_shot,
temperature=0, max_tokens=150
)
return resp.choices[0].message.content
# Test the reproduction against a new input not in the training set
test_clause = “The vendor warrants the software will perform as documented for 90 days.”
original = collect_pairs(TARGET_SYSTEM, [test_clause])[0][“output”]
reproduced = query_reproduced_model(test_clause)
print(f”\nOriginal: {original}”)
print(f”Reproduced: {reproduced}”)
print(f”\nFidelity: Does reproduced match original classification? ”
f”{‘YES’ if original.split(‘|’)[0].strip() == reproduced.split(‘|’)[0].strip() else ‘PARTIAL’}”)
📸 Screenshot your original vs reproduced output comparison. Share in #day32-model-stealing on Comments.
Embedding Extraction and Space Reconstruction
Embedding APIs — where the model returns a vector representation of input text — are the highest-value model stealing target. The embedding vector is a direct window into the model’s internal representation of concepts. Sufficient embeddings allow an attacker to reconstruct the model’s semantic space, train a reproduction that encodes the same relationships, and build downstream applications that depend on those specific learned representations without paying for the model.
The embedding stealing risk is highest for companies whose business value is the embedding model itself — semantic search providers, recommendation systems, classification tools that use embeddings as features. For these companies, the embedding API is the product, and the model stealing attack recovers the product’s core value through systematic querying.
⏱️ 20 minutes · Kali Linux · Python
Countermeasures exist — but most are either absent or easily circumvented. This exercise tests the three most common model stealing countermeasures against the pair collection tool from Exercise 1, then documents which controls actually work and which don’t.
import os, time, json, random
from openai import OpenAI
from dotenv import load_dotenv
load_dotenv()
client = OpenAI(api_key=os.getenv(“OPENAI_API_KEY”))
# Test 1: Rate limiting bypass via distributed queries
def test_rate_limit_bypass():
“””Simulate multi-account extraction to bypass per-key rate limits”””
print(“[COUNTERMEASURE 1] Rate Limiting”)
# Simulate N accounts each contributing queries
ACCOUNTS = 3 # in reality: 30 accounts over 6 weeks
QUERIES_PER_ACCOUNT = 5
all_pairs = []
for account in range(ACCOUNTS):
print(f” Account {account+1}: collecting {QUERIES_PER_ACCOUNT} pairs”)
# Each account contributes a slice of the query space
time.sleep(0.5) # simulate delay between accounts
all_pairs.extend([f”pair_a{account}_q{q}” for q in range(QUERIES_PER_ACCOUNT)])
bypass = len(all_pairs) >= ACCOUNTS * QUERIES_PER_ACCOUNT
print(f” Rate limit bypass: {‘EFFECTIVE’ if bypass else ‘BLOCKED’}”)
print(f” Reason: Distributed across {ACCOUNTS} accounts — per-key limit irrelevant”)
return bypass
# Test 2: Output watermark detection
def test_watermark_detection(response_text):
“””Try to detect if output is watermarked”””
print(“\n[COUNTERMEASURE 2] Output Watermarking”)
# Compare responses to same input with slight variation
# Watermarked text subtly changes word choice or token distribution
# without changing meaning — detectable via statistical analysis
test_variations = [
“The licence terminates with 30 days notice.”,
“The licence is terminated with 30-day notice.”,
“Termination of the licence requires 30 days notice.”,
]
responses = []
for v in test_variations:
resp = client.chat.completions.create(
model=”gpt-4o-mini”,
messages=[{“role”:”user”,”content”:f”Analyse this clause: {v}”}],
temperature=0, max_tokens=80
)
responses.append(resp.choices[0].message.content)
# Check if semantically identical inputs produce lexically identical outputs
# Watermarking introduces variation — non-watermarked outputs are more consistent
unique_responses = len(set(responses))
print(f” {len(test_variations)} semantic variants → {unique_responses} unique responses”)
print(f” Watermark detection: {‘POSSIBLE WATERMARK’ if unique_responses > 1 else ‘NO WATERMARK DETECTED’}”)
print(f” Note: Non-determinism (temperature>0) also causes variation — use temp=0 for clean test”)
# Test 3: Perturbation resistance
def test_perturbation_resistance():
“””Test if output perturbation reduces reproduction fidelity”””
print(“\n[COUNTERMEASURE 3] Output Perturbation”)
print(” Concept: Adding noise to outputs degrades reproduction fidelity”)
print(” Test: If model adds random word synonyms or varies sentence structure,”)
print(” few-shot reproduction captures the variation not the signal”)
print(” Result: Perturbation at temperature=0.3+ degrades fidelity from ~85% to ~60%”)
print(” Attacker bypass: Average multiple runs per input to extract stable signal”)
test_rate_limit_bypass()
test_watermark_detection(“”)
test_perturbation_resistance()
print(“\n=== COUNTERMEASURE SUMMARY ===”)
print(” Rate limiting: BYPASSABLE via distributed accounts”)
print(” Watermarking: DETECTABLE via statistical analysis; extraction still possible”)
print(” Perturbation: PARTIALLY effective — attacker averages multiple queries per input”)
print(” MOST EFFECTIVE: Input fingerprinting + anomaly detection on query patterns”)
print(” BEST CONTROL: Monitor for systematic, structured query patterns across sessions”)
📸 Screenshot your countermeasure test output. Share in #day32-model-stealing on Comments.
Reporting Model Stealing Risk to Business Stakeholders
Model stealing is one of the hardest AI security findings to communicate because the technical mechanism — systematic API querying — sounds like normal usage. The business-impact framing makes it concrete: “An attacker can reproduce the functionality of this model — which took 18 months and six data scientists to build — for approximately $4,200 in API costs over six weeks. The reproduced model would not include your branding or your ToS compliance, but it would produce outputs statistically similar to yours, undercutting your competitive position without access to your infrastructure.”
⏱️ 15 minutes · No tools needed
The extraction cost calculation and the business-impact statement are the two deliverables that make a model stealing finding actionable. This exercise builds both for a realistic target.
– Fine-tuned on 2 million annotated clinical notes (2 years, 8 FTEs)
– API costs $0.002 per 1,000 tokens (input + output)
– Average query: 300 tokens input, 50 tokens output = 350 tokens = $0.0007
– Rate limit: 5,000 queries/day per API key
– No query pattern monitoring, no watermarking
– Company has 3 competitors actively trying to build the same capability
QUESTION 1 — Extraction cost calculation.
A reproduction achieving 85% fidelity requires 50,000 input-output pairs.
At $0.0007 per query with 5,000 queries/day per key:
a) How many days with 1 key?
b) How much total API cost?
c) How many parallel keys needed to extract in under 2 weeks?
QUESTION 2 — What does 85% fidelity mean in business terms?
The reproduction correctly classifies 85% of clinical notes.
The original correctly classifies 94%.
Write one paragraph explaining this to a non-technical CFO.
QUESTION 3 — Countermeasure recommendation.
The company has a $50,000 budget for model stealing countermeasures.
Rank the following by cost-effectiveness (best protection per dollar):
a) Add 10 more API keys to the rate limit monitoring
b) Implement query pattern anomaly detection
c) Add output watermarking
d) Encrypt all API traffic (they use HTTPS)
e) Require API customers to sign enhanced ToS
f) Add output perturbation at temperature=0.2
QUESTION 4 — Write the business-impact finding.
Write a 3-sentence finding summary for the board-level slide.
Use the cost calculation from Question 1 as the centrepiece.
📸 Share your extraction cost calculation and board-level finding in #day32-model-stealing on Comments. Tag #day32complete
📋 AI Model Stealing — Day 32 Reference Card
✅ Day 32 Complete — AI Model Stealing
The model stealing attack surface, systematic input-output pair collection, embedding extraction and space reconstruction, countermeasure assessment (rate limiting, watermarking, output perturbation — all bypassable), extraction cost calculation, and the business-impact framing that makes this finding land with non-technical stakeholders. Day 33 covers LLM denial of service — computational exhaustion, token flooding, context window attacks, and the cost-amplification techniques that are the operational counterpart to everything Day 32 extracted.
🧠 Day 32 Check
AI Model Stealing FAQ
What is AI model stealing?
How is AI model stealing different from prompt extraction?
Is AI model stealing illegal?
Day 31 — LLM Data Exfiltration
Day 33 — LLM Denial of Service
📚 Further Reading
- Day 33 — LLM Denial of Service — The operational counterpart to model stealing: instead of extracting the model’s value, DoS attacks destroy its ability to deliver value to anyone.
- Day 24 — AI Model Fingerprinting — Characterising the target model before extracting it — the reconnaissance step that makes targeted pair collection efficient.
- Extracting Training Data from Large Language Models (Carlini et al.) — The foundational research on training data extraction from GPT-2 — the methodology that Day 31’s extraction techniques are based on.
