All

Playbook Tuning

Automatic playbook improvement that publishes a new version and keeps it until the evidence says to revise it.

Hosted Enterprise

This feature is available in Reflexio Enterprise at reflexio.ai. It is not available in Local OSS.

Playbook tuning improves your user playbooks from what actually happened in your sessions. It reads the sessions a playbook was used in, and when the evidence points to a specific wording change worth making, it writes a new version of that playbook.

What tuning needs before it can improve a playbook

Tuning is off by default for every organization. Turning the switch on under Settings → Playbook tuning enables the daily checks; it does not set up your agent integration.

Your agent must also connect playbook usage to published sessions. Report the user playbooks actually injected into the model context in the Agent interaction's retrieved_learnings, using kind: "user_playbook" and its ID as learning_id. Pass the same user_id and session_id to search and publication. The Python SDK supports this session-based path: a search without request_id can link to a retained published request when that user, session, project and playbook has exactly one exposure and exactly one distinct request reporting the playbook. Unreported or ambiguous matches remain unlinked. Where your publishing API accepts a caller request_id, matching it on search and publication provides explicit correlation. See Report the Context You Injected.

Settings checks the selected project's stored references and resolvable search exposures when you enable tuning. Missing integration shows guidance and a Check again action. Saving stays available while a check is running or unavailable; checks never change your configuration. A failed check is unknown, not proof that links are missing. Once tuning is saved enabled, project pages also show a reminder when a completed check finds missing integration. Remind me later hides that reminder for 24 hours in this browser for your account and project. Checks refresh on project navigation or window focus once their five-minute cache expires, and successful configuration saves refresh the result.

Tuning also needs session outcomes. A check compares sessions that went well against sessions that did not, so it needs to know how sessions ended.

Your agent reporting them with mark_session_outcome is the direct way. Where you do not report one, Reflexio can infer an outcome from its own success evaluation of that session — but only for sessions a playbook was actually used in. It judges such a session when tuning reaches that playbook, as soon as the user has replied after the playbook was used, or after about ten minutes of quiet if they did not. The inferred outcome covers the conversation up to that reply, so later turns do not invalidate it; if another playbook is used later in the same session and gets a reply, the session is judged again up to that point.

Your report always wins. A report from your agent replaces an inferred outcome, and an inferred outcome never replaces yours. Once you have reported an outcome for a session, a conflicting report for it is rejected.

Where the evaluation cannot attribute the result to the agent's own conduct — a provider outage, a missing tool, a data source that returned too little — the inferred outcome is unknown, which keeps the session as validation evidence without treating it as a failure the playbook caused.

An organization that reports no outcomes and has no evaluated traffic will see tuning enabled, playbooks queued, and checks completing — and no version will ever be published, because no playbook can reach the evidence threshold. When tuning is on and no outcome has ever been recorded, the Playbook tuning section says so directly. See Reporting session outcomes for the call.

If Status reads anything other than Ready, tuning cannot run at all yet. config_reset_incomplete means setup has not finished on our side and clears without action from you. deployment_unsupported means DEPLOYMENT_MODE is not a recognised value and needs correcting.

What Tuning Does

Tuning is not a review queue and not a suggestion box. When it acts, it acts:

It reads finished sessions in which a playbook was used.
If those sessions support a specific change, it publishes a new version of the playbook automatically. There is no approval step.
The new version stays in use. It is the version your agents retrieve from the moment it is published.
Later checks read the sessions that happen under that new version, and revise it again if the evidence points to another change worth making.

There is no trial period and no automatic revert. A published version is kept until tuning's own next check finds evidence for a further revision. Nothing ever rules on whether a published version "worked" — there is no such verdict, only a later pass that may find grounds to change it again.

This replaced a 14-day trial

An earlier design put each new version on trial for 14 days and restored the previous one unless the sessions in that window supported it. It was withdrawn because the half that could keep a version was never implemented: the support count was structurally always zero, so every published version was discarded at day 14 no matter how well it did. Rather than build a second judge, tuning now keeps what it publishes and lets its own next check revise it.

If you saw the old behaviour described elsewhere, this page is the current one.

Which Sessions Get Read

A check reads the completed conversation covered by each session outcome. Short conversations are used directly. For longer conversations, Reflexio processes ordered, bounded batches and retains a compact account of goals, constraints, actions, corrections, tool results, and unresolved questions. Original excerpts remain attached to those details so the tuner can verify its citations.

Compaction preserves useful evidence within the model’s input budget, but can omit context. Missing or contradictory evidence weakens a proposed change; a summary alone cannot justify one. Very large sources still have a separate read-safety limit. A session must also fit the configured model's bounded compaction budget; sessions that cannot fit are excluded before model calls. Evidence is compacted when the corresponding tuning stage needs it. Neither compaction nor these limits delete the stored session.

That limit is measured on the conversation itself — what was said, and who said it. It used to be measured on the whole stored record of a session, which carries a great deal of bookkeeping alongside the conversation: timestamps, identifiers, tool metadata. Conversation is only about a fifth of that record, so sessions were being turned away for being large when the conversation inside them was comfortably small. Long, detailed sessions — often the most informative ones — were the sessions most likely to be dropped.

Two consequences you may notice:

More of your sessions are now read. On one production organization's sessions, the share of sessions usable as evidence went from 77% to 100%. Seventeen sessions there had been counted as unusable purely for their size, leaving that organization two percentage points above the threshold at which tuning declines to run at all.
A session left out no longer counts against you. A session that is merely large is intact — nothing about it is missing or damaged — so it is no longer counted alongside sessions that genuinely could not be reconstructed.

Whether a session is left out is still recorded, and if a check reaches no conclusion while some sessions were left out, it says so — see Two Kinds of No Change below.

What Happens When You Edit a Playbook

If you edit a playbook tuning has published a version of, your version is what your agents retrieve, immediately. Two things follow, and the second is the one people get wrong.

Nothing is put back over the top of it. There is no automatic revert in this design, so no earlier version can reappear and displace what you wrote. And if a tuning run happens to be in flight on that playbook when you save, that run is discarded rather than published — it froze its evidence against the version you just replaced, so its conclusion is about text that no longer exists.

But an edit is not a permanent opt-out. Tuning's next check reads the sessions that ran under your version, and if they show it is not working, it will rewrite it — the same as for any other playbook. An edit wins the moment it is made; it does not exempt the playbook from future tuning.

If you want a playbook to stop changing altogether, turn tuning off for the organization. There is no per-playbook opt-out.

When Tuning Runs

Tuning runs on its own in the background. There is no schedule to choose and no time of day to set.

The unit of work is one playbook. A check takes a playbook that has not been looked at yet today, reads its sessions, and either publishes a new version or records why it did not. Every check is recorded against that playbook for the current UTC day, whatever its outcome — so a playbook is checked at most once a day, and the day's queue empties as the checks are made.

Only playbooks in recent use are queued

A playbook joins the day's queue only if it has been used in the last 14 days. One that has seen no use in that window is not queued, and stays unqueued until it is used again — no waiting period ends on its own.

This is not a cap on effort but a consequence of what a check needs: tuning reads the sessions where a playbook was actually applied, and a playbook with none in the window has nothing new to learn from. Checking it would spend a slot on a question its own evidence cannot answer, ahead of a playbook that can.

The Settings page says when this is happening, rather than reporting an empty queue as being caught up.

At most once a day is a limit, not a promise

"At most once a day" caps how often a playbook can be checked. It is not a guarantee that every playbook is checked every day.

Checks run one at a time, and each one takes several minutes because it reads whole conversations and calls a model. So there is a ceiling on how many checks complete in a day, and it is shared across everyone using Reflexio. An organization with more playbooks than that ceiling will see its playbooks checked in rotation rather than daily — the queue is worked oldest-first, so no playbook in it is passed over, but a given playbook may wait.

If you want a specific playbook looked at now rather than waiting for its turn, use Trigger playbook tuning now on the Settings page.

The day rolls over at 00:00 UTC, and that is the only clock involved. A playbook checked at 23:50 UTC is eligible again ten minutes later.

Two exceptions to once a day

If the model provider is busy and turns a check away, Reflexio waits and tries again during the same check rather than giving up. If it is still being turned away after that, the check stops and that playbook keeps its place in the day's queue instead of losing the day to a condition that had nothing to do with it. It comes back after a short cooldown.

If a check finds too little evidence to work with — typically a playbook checked minutes after it was first used, before any of its sessions has an outcome — the playbook comes back the same day once new activity arrives in a session it was used in. It is checked again after a five-minute cooldown.

The two exceptions share one fixed limit on how many times a playbook can come back in a day. Every other outcome — including a check that looked and decided nothing was worth changing — spends the day.

Turning It On

Tuning is off by default. It is a single organization-wide switch, at Settings → Tuning (/settings#tuning-settings).

That section shows three things:

Row	What it tells you
Status	Whether tuning can run at all, and why not when it cannot.
On trial	Left over from the withdrawn trial design. Nothing is ever on trial now, so this reads zero.
Latest available check	The newest retained completed result. Later attempts can replace earlier results, so this is a snapshot rather than complete history. No completed result available means no completed result is retained.

There is nothing else to set. There is no threshold to tune, no model to pin, no schedule to choose, and no per-playbook opt-out — tuning is either on for the organization or off for it.

Turning it on can be refused

Because the capability is unavailable, saving Settings with tuning switched on is rejected and the switch returns to off. The Status row explains which of the three reasons applied.

Trigger playbook tuning now

Below the three rows, the section carries one action: Trigger playbook tuning now. It does the same thing an automatic check does, to one playbook, immediately.

It appears only while tuning is available and switched on for your organization. Where either is not true it is absent, not greyed out — the Status row and the switch already say why, and a disabled button would send you looking for a setting that is right there.

This button is not a preview and it is not free. Pressing it:

runs a real check on one playbook using your generation model, and costs the same as an automatic check would; and
uses up that playbook's check for the day. It will not be looked at again until after 00:00 UTC, whatever the outcome.

Because of that it takes two clicks. The first opens a line stating both costs; the second runs the check. The first click on its own spends nothing.

Alongside the button the section says how many playbooks are waiting to be checked. Once every one of yours has had its check for the day, the button is replaced by Every playbook has been checked today. — that is your organization being caught up, not a failure.

An empty queue has a second cause, and the section distinguishes them. If some of your playbooks have not been used recently, it says so instead: those playbooks are not queued and will rejoin once they are used again. That wording is about some playbooks — you may well have had other playbooks checked the same day.

If the check cannot run, the section says so in place of a result. "Nothing to change" means the sessions behind that playbook did not warrant an edit right now; the day is still spent.

What the section cannot tell you yet

While a check is running in the background, Settings will not say so. The Checking… state is shown only for a check you started from that browser tab. A background check in progress is invisible here; it will appear in the playbook's own Version history when it finishes.

Reading What Happened

Every version of a playbook, and every decision made about it, is readable from Playbooks. Open the row's actions menu and choose Version history.

The drawer shows the playbook's versions newest-first, each with a badge saying where it stands — a version tuning published reads Kept, and one you replaced reads Replaced — and a plain-language sentence explaining that outcome. The badges On trial, Trial ended and Reverted belong to the withdrawn trial design and can no longer occur.

Expanding a published version adds:

Publication — when it was published.
Sessions — how many sessions under this version supported it, disagreed with it, or were not relevant, plus a tally of sessions that could not be compared and why.
What changed — a diff of the playbook text against the version it replaced.
Technical detail — the identifiers and digests, for a support conversation.

Below the versions, the drawer lists checks that produced no new version: the times something looked at this playbook and stopped without changing it, and the reason. These entries are not attributed to tuning specifically, because other optimization work on the same playbook is recorded in the same place.

Two Kinds of No Change

Two of those reasons look similar and mean opposite things, so the drawer keeps them apart:

Reason	What it means	What to do
No change proposed	The sessions were read and did not point to a specific change worth making.	Nothing. There was nothing to find in this evidence.
No change proposed — not all sessions were reviewed	Some sessions were left out because they were too large to include, so what the check saw was not the whole picture.	Nothing is required, but the next check may reach a different answer once more sessions arrive.

They shared one sentence until recently, which meant the second always read as the first — "there is nothing here" when the honest answer was "we did not look at everything". A check that leaves sessions out now records which ones, and that record is what separates the two.

The Playbooks table also gains a Trial column while tuning is available, and the Settings On trial link filters that table at /playbooks?tuning=on_trial. Both are left over from the withdrawn trial design: nothing is on trial any more, so the column shows an em dash for every row and the filtered list is empty.

What Tuning Will Not Do

These are deliberate absences, not gaps waiting to be filled:

It will not ask you to approve or reject a version. Publication is automatic.
It will not revert a version for you. Nothing is put back automatically; to undo a published version, edit the playbook, and your version is what ships from that moment. (It is not an opt-out — see What Happens When You Edit a Playbook.)
It will not let you publish a version immediately. Trigger playbook tuning now brings one playbook's check forward; it does not change what the check is allowed to decide, and there is no way to re-run a playbook that has already had its check today.
It will not let you choose which playbook is checked next. The automatic run takes the playbook that has gone longest without one, across your whole organization.
It will not show the analyst's own written notes about your sessions. Those notes are not returned to the portal at all.
It will not change agent playbooks. Tuning works on user playbooks only; agent-wide rules keep their human approval gate.

Version numbers are display only

v1, v2 and so on are assigned by the portal for readability, in creation order. They are not stored anywhere and mean nothing outside this screen. Quote the playbook ID in a support thread, not the version label.

Playbooks — what user and agent playbooks are
Playbook Optimization — the separate, GEPA-backed refinement step for pending agent playbook wording, which is configured, approval-gated, and unrelated to tuning