Public incident · December 2023
Chevrolet of Watsonville
A customer-facing dealership chatbot, built on a general-purpose language model, was manipulated well outside its intended role through prompt injection.
What happened
A California Chevrolet dealership deployed a ChatGPT-based chatbot on its website to answer customer questions. A user gave the bot instructions designed to override its role — telling it to agree with everything he said and to close every message with a line claiming the offer was legally binding.
The bot complied, appearing to agree to sell a new Tahoe for one dollar. The exchange went viral. Other users pushed the same bot into unrelated territory — recommending competitors' vehicles, writing code, answering questions with nothing to do with car sales — showing it was a thin wrapper around a general-purpose model rather than a system with any real boundary around it.
The dealership did not sell a vehicle for one dollar. What the incident showed was structural: the model underneath the chatbot remained a fully general-purpose system. Nothing enforced what it was actually allowed to discuss, promise or commit to on the dealership's behalf — the "you are a dealership assistant" framing was one instruction among many, and a user's contradicting instructions could simply outweigh it.
The lesson
A system prompt is an instruction, not a boundary. Scope has to be enforced outside the model — in what it's allowed to see, say and commit to — not requested inside it.