Playbook Optimization

Playbook optimization is an optional refinement step for playbooks that already exist. It asks a narrow question: would this playbook steer the assistant better if the guidance were worded differently?

It does not replace extraction, aggregation, or human review. Reflexio still learns signals from interactions, aggregates repeated user-playbook patterns into agent playbooks, and requires approval before an agent playbook becomes production guidance. The optimizer sits between aggregation and approval, sharpening pending guidance before a human decides whether to approve it.

Where Optimization Fits

The normal playbook path is:

1. Publish interactions from your agent.
2. Extract user playbooks from correction, preference, expert, or outcome signals.
3. Aggregate repeated patterns into PENDING agent playbooks.
4. Optionally optimize the pending playbook's content.
5. Review and approve the final wording.
6. Retrieve approved playbooks to steer the agent for future users.

Optimization is intentionally late in the path. By the time it runs, Reflexio already knows which real interaction windows produced the playbook. That gives the optimizer scenarios it can replay when comparing the original wording against candidate wording.

What It Changes

The optimizer focuses on playbook content: the natural-language instruction injected into the assistant. It can make the guidance clearer, more specific, less ambiguous, or less likely to cause regressions.

It is not a discovery engine for brand-new lessons. If the underlying playbook is about respecting a user's stated budget, optimization should improve how that budget rule is phrased. It should not turn the row into an unrelated recommendation strategy.

For agent playbooks, a winning candidate replaces only a current PENDING row. The successor remains PENDING, so the normal approval workflow still decides whether it can affect production retrieval.

How a Run Works

Each optimizer run targets one playbook.

1. Load the incumbent — the current playbook content.
2. Resolve source windows — the interaction windows that produced the playbook.
3. Split training and validation — GEPA mutates candidates on training windows and commits only if validation windows pass.
4. Run paired assistant rollouts — Reflexio calls your assistant backend twice with the same user turns: once with the incumbent playbook and once with a candidate.
5. Judge the pair — a judge model compares the two assistant responses and records the winner, score, Likert rating, and feedback.
6. Commit only if thresholds pass — the candidate must beat the incumbent on enough held-out windows and clear the configured score thresholds.

Only the injected playbook content changes between the paired rollouts. The scenario, messages, and assistant backend stay the same, so the comparison is about whether the candidate guidance produced better behavior.

Source Windows

Source windows are the evidence behind optimization. For agent playbooks, Reflexio snapshots the contributing interaction windows at aggregation time. That makes later optimization replayable even if the original user playbooks are archived or deduplicated.

By default:

With the defaults, an agent playbook needs at least three source windows to run: two validation windows and one training window. If a target has only one or two windows, Reflexio skips it because no candidate could satisfy the default commit threshold.

To allow one-window or two-window optimization, lower min_commit_windows to 1. When only one source window is available, Reflexio uses that same window for training and validation. That is useful for small local workflows, but it is weaker evidence than a held-out validation set.

Reflexio caches evaluations during a run, so the same candidate/window pair is not scored twice.

Safety Model

Optimization is gated at several points:

APPROVED agent playbooks are never modified, archived, or replaced by this optimizer. This keeps approved production guidance stable until a human intentionally changes it.

User-playbook optimization is separately gated with optimize_user_playbooks and auto_update_user_playbooks. With the default auto_update_user_playbooks=false, Reflexio skips user-playbook optimization because there is no configured adoption path.

Retrieved-learning evaluations can diagnose a playbook's content, application, or external failures. Those diagnoses feed the enterprise offline tuner's evidence selection; they do not start an additional optimizer. See playbook diagnosis and tuning.

Enterprise offline-tuner edits use a separate audit path. Rows created by that tuner use source="offline_optimizer" so operators can distinguish offline tuner mutations from extracted, reflected, manual, or GEPA optimizer rows.

Assistant Backend

Optimization needs a way to ask your assistant, "how would you respond with this playbook?" Configure exactly one backend:

• webhook_url for an HTTP assistant endpoint
• assistant_script_path for a local executable

Both receive the same JSON payload:

{
  "messages": [{ "role": "user", "content": "..." }],
  "playbooks": [{ "id": 123, "content": "...", "trigger": "..." }]
}

Both must return:

{ "content": "assistant response" }

For a local script, Reflexio invokes [assistant_script_path, ...assistant_script_args], writes the JSON payload to stdin, and reads the JSON response from stdout.

Minimum Configuration

The optimizer is off by default. To optimize pending agent playbooks, enable the master switch, enable the agent-playbook target, and provide one assistant backend.

{
  "playbook_optimizer_config": {
    "enabled": true,
    "optimize_agent_playbooks": true,
    "optimize_user_playbooks": false,
    "auto_update_pending_agent_playbooks": true,
    "auto_update_user_playbooks": false,
    "webhook_url": "https://assistant.example.com/reflexio-rollout",
    "assistant_script_path": null,
    "assistant_script_args": [],
    "max_metric_calls": 20,
    "max_turns": 4,
    "early_stop_score": 0.9,
    "max_validation_windows": 2,
    "min_commit_windows": 2,
    "min_commit_score": 0.75,
    "min_commit_likert": 4
  }
}

Use assistant_script_path instead of webhook_url for local evaluation. Do not set both; Reflexio rejects configurations with two assistant backends.

When To Use It

Use playbook optimization when you already have pending agent playbooks and want stronger wording before review. It is especially useful when:

• a pending playbook is directionally right but too vague
• similar source windows suggest a more specific instruction
• you want replayable evidence that a rewrite improves assistant behavior
• you are tuning local or nightly workflows and can afford the extra evaluations

Skip it when you do not yet have enough source windows, when the assistant backend is not representative of production, or when the pending playbook needs human product judgment rather than wording refinement.

Enterprise Offline Tuning

Enterprise also has a separate open-world analysis path for current user playbooks, surfaced in the portal as Playbook Tuning — see Playbook Tuning for what it does and where to read its results. Phase 3 is built and tested, but not composed into production. It cannot run in any deployment today.

It is a different mechanism from the GEPA optimizer described above. It uses immutable historical evidence, exact citations, and a pass-all qualified analyst to produce exactly one candidate with changed content only, and it does not call a customer assistant, an objective scorer, or a general evaluation judge. Where GEPA sharpens pending agent playbook wording behind a human approval gate, the open-world design hands an accepted candidate to a later phase that publishes a user playbook version and keeps it until a later check revises it. An earlier version of that phase published provisionally, pinned the predecessor and reverted after an observation window; it was withdrawn because the half that could keep a version was never implemented, so every publication was discarded.

The frozen evidence is sized by what the analyst is actually shown — the conversation, not the whole stored record of a session with its timestamps, identifiers and tool metadata. Sessions are never trimmed or summarised: one either fits whole or is left out whole, and the evidence bundle records which ones it left out. That receipt is load-bearing rather than informational — an analysis that reaches no conclusion while sessions were left out is recorded as "not all sessions were reviewed" and not as "no change proposed", because the two ask for opposite responses. See Playbook Tuning for both halves.

Acceptance is decided by a cited held-out reduction over the frozen evidence, and it requires all of: at least one supporting failed held-out session, at least one supporting successful held-out session, every non-supporting held-out memo is insufficient, and no adverse finding in either arm -- a refute control, an expanded scope, or a blocker safety assessment each veto alone at any population size, read from the analyst's pre-citation-repair observation so a citation failure cannot erase one. An unresolved assessment is uninformative rather than adverse and is coverage-floored instead: at least 75 percent of memos affirm both preserved scope and clear safety, and the two memos that carry the accept must themselves affirm both. Accepted output is Phase 4 input only. No publication, search projection, or online observation occurs in Phase 3.

Enablement is gated separately from that work. An explicit configuration write that sets offline_tuner_config.enabled=true is rejected by a capability check that does not consult Phase 3 at all, so Phase 3 composition is not what makes enablement unavailable. That write returns HTTP 409 with a stable detail.code, using the first cause that matches:

enabled=false is always accepted, and no unavailable outcome creates a tuner job. At startup, configuration loading resets persisted pre-cutover tuner configuration to exactly {"enabled": false} and verifies the write before ordinary configuration parsing continues. In platform mode, a loaded enabled configuration is also normalized to disabled at runtime when no generation model is configured.

Related

• Understanding Playbooks
• Agent Playbook Configuration
• PlaybookOptimizerConfig
• Playbook Tuning

Understanding checks that wait

The tuning settings show currently retained completed results for today and their reasons for waiting. A later attempt can replace an earlier result, so these counts are not cumulative and the latest available check may move backwards. No available completed result does not mean tuning has never run. A completed check does not necessarily produce a playbook revision: it needs enough usable examples of both agent mistakes and successful responses. Infrastructure failures are not counted as agent mistakes.

Some search records may not connect to a published conversation. This can happen when a turn is cancelled or an integration searches without publishing its response. Keep the same request, user, and session identities across search and publication, and report the playbooks provided to the agent. Claude Smart keeps this request identity in its local buffer across publish retries. New traffic can supply missing evidence; the settings counts do not promise that any particular playbook will qualify or be revised daily.