Craft

Turn audit

Note

Written for AI agents. Orbit and coding agents follow this page when they work on your world. You don't need it to use Craft, but it shows how an agent will approach the job.

A turn audit explains one turn: what the GM did wrong, why, and where the fix belongs. It reads the turn's record (what the GM was sent and everything it did) the way debugging a GM turn shows, and changes nothing until the creator picks a fix.

Work through the causes below in order. Each is cheaper to rule out than the next, and an earlier cause often explains a later symptom: a model that "ignored" a rule it was never given has a trigger problem, not a model problem. Stop at the first cause the record proves, but say so if a second one also played a part.

Before you start

  • Know what went wrong. Take the user's words, and find the moment in the reply. If they only gave an id, read the reply and ask what they expected only if nothing in it is plainly off.
  • Note the turn's facts from the overview: the model, the player's move, the rules the turn was given, and the tools the GM called.
  • Check how it ended, in the overview's "How it ended" lines, before anything else. See a turn that failed.
  • Search for the exact words of what went wrong (a name, a number, a rule) before reading in order.

A turn that failed

A turn that ended on an error isn't something the GM chose. If "How it ended" says error, or lists an error, read what the error means before looking for a cause in the world:

  • The provider failed (network, stall, timeout, upstream_capacity, rate_limited, provider_error): the connection to the model's provider dropped, it went quiet, or it was overloaded. A reply cut off mid-thought by one of these is not the model stopping, and nothing in the world caused it. Say so, and suggest Try again or Rewrite. Don't suggest another model to test it: one failure says nothing about the model.
  • The model or its limits (context_length_exceeded, model_unavailable, price_limit, a call that finished on length): a different model, or a smaller context, fixes it.
  • The provider refused the content (content_filtered): that model's provider has its own moderation; another model may accept it.
  • The model's mistake (invalid_tool_input, unavailable_tool): this is the model's limits.
  • Craft's (provider_auth, invalid_request, unknown): report it, as in Craft's harness.

Whatever the turn's calls recorded before the error (moves, changes to files, rolls) has landed, and stays if the player carries on from there. Try again or Rewrite deletes the failed reply, and for the game's owner that undoes its file changes too: files it created go to the trash and files it changed go back. Say which changes landed, so the player can choose between carrying on from them and playing the turn again.

A turn Craft's own content moderation blocked never reached the GM, and has no record to read: reading it says so. Explain that it was Craft's content policy, not the world or the model.

"Craft stopped the model repeating itself" means the model looped and Craft cut it off: a model limit. Refused update() calls "never sent again fixed" mean the files fell behind the story; read the refusal to see why.

If the turn ended normally, go on to the causes below.

1. Instructions or context

Most turns that go wrong were told to, or were missing what they needed. Look for:

  • An instruction that said to do it. A GM instruction, a picked option or a tracker's rules that, read literally, asks for what happened.
  • Instructions that disagree. Two rules that fire on the same moment and say different things. The GM had to pick one.
  • Context that misled it. A file, tool result or earlier reply that held the wrong fact, an old value, or two versions of one thing.
  • Missing context. What the GM needed wasn't in the prompt or in anything it read: a file it didn't open, or a fact the world doesn't have.

Weigh the instructions against the model. A weaker model with many long, tangled or contradictory rules will drop some of them; the fix is fewer, plainer rules, with one owner for each moment (writing good instructions, refactoring GM instructions). If the instructions are plain and the context was right, move on: the cause is further down.

The fix: quote the rule or result, say where it is, and give the rewording, the merge, or the fact to add (working with AI).

2. A rule that didn't fire

A situational instruction is only given on turns its trigger matches, in the <rules> block beside the player's move. If the moment came up and the rule isn't in that block, its trigger didn't match. A turn gets at most 5 rules, the likeliest first, so a rule can also lose its place to broader ones.

The check judges each trigger against the player's move and the end of the GM's last reply, plus one line on the scene. It can't see sheets, trackers or hidden values (triggers). So:

  1. Read the player's move and the end of the reply before it. What words were there?
  2. Compare them with the trigger. A trigger about a sheet value ("When Heat is above 5"), or one in the world's jargon the player didn't use, won't match.
  3. Write a trigger for a moment someone could see in those words. A player writes "I swing at the guard" or "I try to talk him down"; a GM ends on "the guard draws his blade". "When someone attacks or is attacked" matches both.

If broader rules took the five places, narrow their triggers instead. Suggest the user try the new trigger on a few moves in the Instruction tester.

3. The model's limits

Models differ, and some habits no prompt removes. One writes short replies and another long ones; one keeps to a format, another drifts from it. Instructions help, but they don't make a weaker model as good as a stronger one: if they did, nobody would need the stronger ones.

How a model writes (length, voice, style, keeping to prose rules) is the hardest to prompt away, so a turn that broke a prose rule while the rule was plain and given is most likely this.

Look the turn's model up in the text models table, by the id the turn record gives. Core models, and the others not tagged Expensive, are smaller models, weaker than the frontier ones (how strong a model is): a world with many or strict rules asks more of them than they can reliably give. When the turn ran on one, say so plainly. A high Intelligence rating doesn't change that; don't soften it into "it depends".

Then give the creator a way to tell:

  • Try the turn on a frontier model. Name two or three of the frontier models, with what a turn costs on each: one turn to compare is worth it even at their price. Switch the game's text model (picking a game's model), then Rewrite the turn from its ⋯ menu (the message menu). If the stronger model gets it right, the instructions are fine and this model can't do it reliably.
  • Or adjust what to expect from the model they play on, and keep the instructions as they are.

Don't pile more rules on to push a model toward something it can't do: they crowd out the rules it can follow.

4. A script or computed field

When a number, a roll or bookkeeping came out wrong, check whether code made it.

  • A script. Find the run call in the turn and read its input and result. If the result was wrong, open the script and find the line that made it. If the result was right and the GM narrated something else, that is a cause above.
  • A computed field. If a value on a sheet was wrong, read the field's expression against the values it reads.

The fix is in the code: say which line is wrong and what it should be (scripts, testing scripts, computed fields).

5. Craft's harness

Some of what the GM is sent is Craft's and can't be changed by the creator: the system prompt outside the world's instructions, the reminders Craft adds to messages, and how tool calls and their results are handled and cut short. If the record shows one of these caused it, nothing in the world fixes it. Say so, and ask the user to report it in the bug reports channel of Craft's Discord, with the turn's request id.

A result that reads [pruned] was shown in full during its own turn; it isn't a cause on that turn.

The reply

Write it plainly, for the person who played the turn:

  1. What happened, in a sentence.
  2. Why: the cause, with the exact instruction, result or step quoted, and where it is.
  3. The fix, and where to make it. For a model limit, the test with a stronger model; for a harness problem, the report.

End by offering the fix: "Would you like me to reword the trigger?" Make no change until the user says.

Don't give away plot the player hasn't reached unless they ask.