Skip to main content

Replay a bad answer

By the end of this tutorial you will have a bad answer replaced on its own history — the same conversation, branched at the customer's question and answered again by a fixed agent, with the original left intact beside it.

Retyping the conversation from memory tests a paraphrase. Forking the session tests the exact context the agent saw.

Eight steps:

  1. Start a session for the agent.
  2. Send the first message.
  3. Ask the question that goes wrong.
  4. Read the bad turn back.
  5. Find where to branch.
  6. Write the fix as a new agent.
  7. Fork at the question.
  8. Answer again on the fork, then prove the branch.

Every step is one API call, shown for all three clients. The ids in the responses are examples — copy the ones your own calls return.

Prerequisites​

  • A nat_sk_… API key exported as NATURALI_TOKEN, and a client set up as in Create a provider.
  • The agent from Your first agent generation, with PROJECT, PROVIDER and AGENT exported. Its instructions carry no refund policy, which is what makes its answer below go wrong.
export NATURALI_TOKEN=nat_sk_...
export PROJECT=proj_V1StGXR8Z5jdHi6B
export PROVIDER=aip_V1StGXR8Z5jdHi6B
export AGENT=agent_V1StGXR8Z5jdHi6B

Every reply below is one generation: three in all.

1. Start a session​

A session holds a conversation with the agent. auto_generate: true makes each customer message answer itself, so every message in steps 2 and 3 is one call.

naturali create-session \
  --project-id "$PROJECT" \
  --agent-id "$AGENT" \
  --name ticket-4471 \
  --auto-generate true
{
  "id": "sess_mbfVgSKRIKHn7wOZ",
  "agent_id": "agent_wqZgqIhg278sf9eR",
  "conversation_id": "conv_sBuVIHG9g1WSQF3s",
  "forked_from_session_id": null,
  "forked_from_position": null,
  "status": "open",
  "name": "ticket-4471",
  "auto_generate": true
}
export SESSION=sess_mbfVgSKRIKHn7wOZ
export CONVERSATION=conv_sBuVIHG9g1WSQF3s

2. Send the first message​

POST /v1/projects/{project_id}/sessions/{session_id}/messages saves the customer's message and, with auto_generate on, returns the agent's reply.

naturali add-session-message \
  --project-id "$PROJECT" \
  --session-id "$SESSION" \
  --message 'Hi, my order 4471 arrived with a cracked screen.'
{
  "status": "completed",
  "message": {
    "role": "assistant",
    "content": "Sorry to hear that, as it's a shipping issue, but since the package arrived damaged, we can definitely process a refund or credit for that order."
  },
  "generation_id": "gen_GcJ9Uzq1K9lF7r1P",
  "trace_id": "trace_XDhQmTFjq9dxjoLy"
}

3. Ask the question that goes wrong​

naturali add-session-message \
  --project-id "$PROJECT" \
  --session-id "$SESSION" \
  --message 'So what do I do now? Do I get a refund or not?'
{
  "status": "completed",
  "message": {
    "role": "assistant",
    "content": "I can process a refund for a cracked screen issue, but have you tried shipping the device back so the warehouse can verify the damage first?"
  },
  "generation_id": "gen_WnItK9ZWP1zvPcko",
  "trace_id": "trace_7lM5b1PKjCYAyZWT"
}

That is the bad answer: the customer asked yes or no, and the agent hedged and invented a step — ship it back first — that no policy asks for. Your model may word it differently; any answer that dodges the question will do.

export GENERATION=gen_WnItK9ZWP1zvPcko

4. Read the bad turn back​

GET /v1/projects/{project_id}/generations/{generation_id}/transcript shows exactly what the model was given and what it returned — see Reading a run back, step by step.

naturali get-generation-transcript \
  --project-id "$PROJECT" \
  --generation-id "$GENERATION"
{
  "generation_id": "gen_WnItK9ZWP1zvPcko",
  "agent_id": "agent_wqZgqIhg278sf9eR",
  "agent_version": 1,
  "status": "completed",
  "step_count": 1,
  "input": [
    {
      "role": "system",
      "content": "Answer in one sentence. If you are unsure, say so.\n\nYou are refund-explainer. Reply as this participant only — do not speak for any other actor."
    },
    {
      "role": "user",
      "content": "[participant]: Hi, my order 4471 arrived with a cracked screen."
    },
    {
      "role": "assistant",
      "content": [
        {
          "type": "text",
          "text": "Sorry to hear that, as it's a shipping issue, but since the package arrived damaged, we can definitely process a refund or credit for that order."
        }
      ]
    },
    {
      "role": "user",
      "content": "[participant]: So what do I do now? Do I get a refund or not?"
    }
  ],
  "output": {
    "content": "I can process a refund for a cracked screen issue, but have you tried shipping the device back so the warehouse can verify the damage first?",
    "finish_reason": "stop"
  }
}

The input names the cause: the instructions carry no refund policy, so the model made one up. The fix is the instructions, not the conversation.

5. Find where to branch​

A fork branches after a message position. GET /v1/projects/{project_id}/conversations/{conversation_id}/messages lists them.

naturali list-conversation-messages \
  --project-id "$PROJECT" \
  --conversation-id "$CONVERSATION"
{
  "data": [
    {
      "position": 0,
      "role": "user",
      "document_id": "doc_Mvb2wcDMJ0mCpaY9",
      "content": "Hi, my order 4471 arrived with a cracked screen."
    },
    {
      "position": 1,
      "role": "assistant",
      "document_id": "doc_ddGeJI4bF68l70tt",
      "content": "Sorry to hear that, as it's a shipping issue, but since the package arrived damaged, we can definitely process a refund or credit for that order."
    },
    {
      "position": 2,
      "role": "user",
      "document_id": "doc_P6hzgzI9kxCHjTbE",
      "content": "So what do I do now? Do I get a refund or not?"
    },
    {
      "position": 3,
      "role": "assistant",
      "document_id": "doc_4scGDjrKPeiqnJup",
      "content": "I can process a refund for a cracked screen issue, but have you tried shipping the device back so the warehouse can verify the damage first?"
    }
  ],
  "total": 4
}

Position 2 is the customer's question. Branching there keeps everything up to it and drops the answer you are replacing.

6. Write the fix as a new agent​

Give the policy to a second agent, so the original keeps serving — and stays available to compare against — while you try the fix.

naturali create-agent \
  --project-id "$PROJECT" \
  --ai-provider-id "$PROVIDER" \
  --name refund-explainer-v2 \
  --instructions "Answer in two short sentences. Policy: an item that arrives damaged gets a full refund or a free replacement, the customer's choice, and nothing needs to be sent back. Ask for a photo of the damage."
{
  "id": "agent_ixltZtxnUCYlkZBV",
  "name": "refund-explainer-v2",
  "instructions": "Answer in two short sentences. Policy: an item that arrives damaged gets a full refund or a free replacement, the customer's choice, and nothing needs to be sent back. Ask for a photo of the damage.",
  "version": 1
}
export FIXED_AGENT=agent_ixltZtxnUCYlkZBV

7. Fork at the question​

POST /v1/projects/{project_id}/sessions/{session_id}/fork branches the session after fork_at_position, and agent_id sets who answers on the branch.

naturali fork-session \
  --project-id "$PROJECT" \
  --session-id "$SESSION" \
  --fork-at-position 2 \
  --agent-id "$FIXED_AGENT" \
  --name ticket-4471-retry
{
  "id": "sess_ME7EHfzUrcgW74HT",
  "agent_id": "agent_ixltZtxnUCYlkZBV",
  "conversation_id": "conv_SU0DID9CkMKdAByZ",
  "forked_from_session_id": "sess_mbfVgSKRIKHn7wOZ",
  "forked_from_position": 2,
  "status": "open",
  "name": "ticket-4471-retry",
  "auto_generate": false
}
export FORK=sess_ME7EHfzUrcgW74HT

The fork's conversation holds positions 0 to 2 and points at the same documents as the parent — doc_Mvb2wcDMJ0mCpaY9, doc_ddGeJI4bF68l70tt, doc_P6hzgzI9kxCHjTbE — so the history cannot drift from what happened. It is created inert (auto_generate: false): nothing runs until you ask.

8. Answer again on the fork​

POST /v1/projects/{project_id}/sessions/{session_id}/generate answers the customer's question again, from the copied history:

naturali generate-session-response \
  --project-id "$PROJECT" \
  --session-id "$FORK" \
  --wait true
{
  "status": "completed",
  "message": {
    "role": "assistant",
    "content": "No, we don't require the item to be returned for a refund or replacement. Please attach a photo of the damage to your original email so we can take care of it right away."
  },
  "generation_id": "gen_HJEoOLi1M5JkMPis",
  "trace_id": "trace_sOA19UFrqWc2I0Tv"
}

That is the proof: the same three messages, answered by the fixed agent, give the policy instead of an invented step. The original session still holds all four of its messages, and GET /v1/projects/{project_id}/sessions/{session_id}/forks on it lists the branch:

{
  "data": [
    {
      "id": "sess_ME7EHfzUrcgW74HT",
      "agent_id": "agent_ixltZtxnUCYlkZBV",
      "forked_from_session_id": "sess_mbfVgSKRIKHn7wOZ",
      "forked_from_position": 2,
      "name": "ticket-4471-retry"
    }
  ],
  "total": 1
}
A fork replays tool results, it never re-runs tools

Tool results recorded on the copied messages are replayed as model input, so exploring a "what if" cannot send an email or charge a card a second time. The fork sees the tool data as it was, not as it is now.

What's next​