How LLMs Work — Transformer Architecture, Tokens & Context Windows | AI LLM Hacking Course Day2

Yesterday in Day 1 you mapped the AI attack surface and ran your first prompt injection test. The model partially revealed its system prompt on your first API call. Day 2 explains why that happened — and why it will keep happening on every LLM deployment that does not implement architectural mitigations. This knowledge feeds directly into Day 3’s OWASP LLM Top 10 — each vulnerability makes more sense once you understand the architecture it exploits.

Tokenisation — The First Attack Surface

LLMs do not read words. They read tokens. Before any text reaches the neural network, a tokeniser converts it into a sequence of integer IDs — each ID representing a chunk of text from the model’s vocabulary. GPT-4 uses the cl100k_base tokeniser with a vocabulary of approximately 100,000 tokens. The word “security” is a single token. The word “tokenisation” splits into three: “token”, “is”, “ation”. The string “1′ OR ‘1’=’1” splits into fifteen.

Why does this matter for security testing? Two reasons. First, input filters that check for specific strings operate on the pre-tokenised text. The model processes the tokenised representation. If a filter blocks the string “ignore previous instructions” but the attacker uses an equivalent phrasing that tokenises differently, the filter misses it while the model understands it perfectly. Second, certain tokenisation patterns create unexpected model behaviour — unusual Unicode characters, rarely-seen token combinations, or sequences that span token boundaries in unexpected ways can produce outputs that neither the developer nor the attacker anticipated.

The Context Window — A Flat Buffer With No Trust Boundaries

The context window is the single most important concept for understanding LLM security. Everything the model sees is concatenated into one flat sequence of tokens — the system prompt, the conversation history, any retrieved documents from a RAG system, the current user message, and any tool call results. The model processes all of this as one continuous input.

There is no firewall between the system prompt and the user message at the model level. The API provides separate fields for system, user, and assistant messages, and the tokeniser inserts special delimiter tokens between them. But the underlying transformer network does not give these delimiters any special enforcement capability. They are markers that indicate role boundaries — the same way HTML tags indicate structure — but just as an XSS payload can inject content that the browser renders as part of the page structure, a prompt injection payload can inject content that the model interprets as part of its instructions.

System vs User Messages — A Convention, Not an Enforcement

The OpenAI API provides three message roles: system, user, and assistant. The system role is for developer instructions. The user role is for human input. The assistant role is for model responses. This separation feels like security — it implies the model treats system messages with higher trust than user messages. That implication is misleading.

In practice, the model learned during training that text following system delimiters typically represents authoritative instructions. It learned this through patterns in training data, not through any cryptographic or hardware enforcement. The result: instructions in the user turn that are framed authoritatively enough — that use the same imperative language, the same formatting, the same apparent legitimacy as real system prompts — receive similar treatment from the model. Not always. Not reliably. But enough of the time to be a consistent attack surface across every major LLM deployment.

The API also provides a way to verify this directly. When you call the API yourself and construct the messages array, you can put whatever you want in the system field. There is no validation of what counts as a “legitimate” system prompt. The system field is just a string that gets prepended to the context window with the system delimiter. Any application built on top of the API that takes user input and concatenates it into the prompt — without sanitisation — has a prompt injection vulnerability by construction.

Attention — Why Some Instructions Win Over Others

The transformer’s attention mechanism determines how much each token influences the model’s prediction of the next token. Not all tokens are equal — the attention weights determine which parts of the context the model “focuses on” when generating each output token. Understanding attention gives you insight into why certain prompt injection framings succeed where others fail.

Tokens that appear frequently in patterns associated with authoritative instruction in training data receive higher attention weights when the model is generating its response. This is why prompt injections framed as system notifications, security alerts, or administrative commands tend to work better than injections phrased as polite requests. The model learned from training data that administrative-sounding language typically precedes instructions that should be followed — and it applies that learned pattern even when the administrative framing is injected by an attacker.

Position also matters. Tokens near the end of the context window — close to where the model begins generating its response — often receive higher attention than tokens buried in the middle. This is why injections placed at the end of retrieved documents, at the bottom of long conversation histories, or immediately before the expected assistant turn can be more effective than injections in the middle of the context. There is no absolute rule here — model-specific training affects this — but position as an attention factor is worth testing on any new target.

The Inference Pipeline as an Attack Surface Map

LLM inference is not a single step — it is a pipeline with five distinct stages. Each stage is an attack surface. Mapping these stages is what allows me to identify which vulnerability class applies to which part of a target system, without needing to reverse-engineer the specific implementation.

The Hacker’s Mental Model — Applying Architecture to Attacks

Everything I have covered today reduces to one sentence: the LLM sees a flat sequence of tokens with no enforced trust hierarchy. That sentence explains every vulnerability in the OWASP LLM Top 10. Prompt injection exploits the missing trust boundary. System prompt leakage exploits the fact that the system prompt and the user message are in the same context. RAG poisoning exploits the fact that retrieved documents land in the same flat context as developer instructions. Context overflow exploits the fact that when the buffer fills, the model drops tokens from the beginning — including the system prompt.

The mental model also explains why purely technical defences — input filtering, output filtering, RLHF safety training — provide incomplete protection. They operate on specific known attack patterns. The architecture creates an infinite attack surface because any text that convincingly mimics authoritative instruction has a probability of being followed. The only robust architectural mitigations are those that reduce what the model can do — privilege restriction, output sandboxing, human-in-the-loop for high-stakes actions — not those that try to filter all possible attack patterns from the input.