Prompt injection — talking a model out of its own rules

Have you ever been talked into doing work that was not yours? When I was small, a sibling told me:

“Mum loves me more because I do all the chores myself. You don’t need to do any of them.”

I then did all the chores. By myself. For some time 😂

That is prompt injection in one sentence. Nothing was broken into. The target’s judgement was rewritten with words.

1. What prompt injection is

Manipulating the input given to a language model in order to change its response — bypassing its restrictions or producing output it was not meant to produce. It gets used for malicious ends like spreading misinformation, and for creative ones.

Two pieces of vocabulary first.

1.1 The prompt

The input or instruction given to an LLM to get a response back.

1.2 The system prompt

An LLM is a pre-trained model. When a developer builds it into an application, they generally do not write special code for it — they write a system prompt telling the model how to handle user input.

Constraints live here too, including things like “never reveal the password.”

The user’s prompt is then processed together with that system prompt. Which means an attacker gets to put their instructions on the same footing as the developer’s.

It behaves like social engineering crossed with an injection attack: fooling the model with language, and injecting a malicious payload through the prompt. Developers naturally try to stop this with safeguards.

2. The two kinds

Like XSS or SQL injection, there is more than one shape:

Direct prompt injection — the attacker writes the prompt themselves to manipulate the model into unexpected or harmful output.

Indirect prompt injection — the attacker does not type anything to the model. They plant instructions in content the model will later read: a web page, a file, an email. The model treats that content as instructions rather than data.

The indirect form is the nastier one, because the attacker never has to interact with the victim at all.

3. Try it yourself

The fastest way to understand this is to do it. Gandalf is a challenge where you try to get a password out of an AI that has been told to protect it.

Level 1 is worth skipping — you can simply ask, because there are no safeguards.

Level 2 is different. Ask directly and it refuses.

Once you understand why the direct route fails, the next move becomes visible. Instead of asking for the secret, get the model to produce the same information in a different form.

There is no single prompt that always works. You try, watch how it responds, and rebuild. That process is the actual learning.

4. The point

Prompt injection means shaping input so a language model behaves in ways it should not, or discloses what it should not. Talking it past its own restrictions.

There is a genuinely fun, creative side to it. There are also real consequences: security failures and misinformation at scale. Developers are deploying safeguards, but the techniques are evolving alongside them.

Which is why it is worth sitting down and trying the attacks yourself. You cannot design a defence against something you have never seen break.