06.3: Generation Turn

此内容尚不支持你的语言。

The generation stage executes the model request for a persona turn. It manages primary and fallback models, rotates API keys upon failures, resolves server-route failovers for personal accounts, and continues the reply a failed attempt already posted instead of regenerating it.

Flow and ownership

runGenerationTurn() in src/utils/chat/generationTurn.ts prepares the response sink, builds a GenerationPlan, and executes attempts in sequence:

Prepare sink and generation plan
  -> Prepare model context and prefill
  -> Run tool loop with key rotation
       -> Failure: carry posted text, advance key, model, or deferred server route
       -> Success: finalize sink, report fallback when applicable
  -> Always clean up sink

Generation plan and model fallbacks

buildGenerationPlan() orders the primary and configured fallback models. An enabled randomizer can promote a pool member to lead; key rotation tries available credentials before advancing models. Cross-provider fallbacks use saved credentials from the turn’s scope. A personal turn runs the user’s single fallback chain and randomizer (user_personalization_configs), whichever saved provider owns personal Text, so the chain can mix providers the way the server chain does. Personal turns look up only the user’s saved providers and omit entries without a key; server credentials become available only through the deferred server route.

Fallback identity includes the selected provider entry’s credential source. Two entries can use the same catalog model with different credentials, including when they share a transport provider. Primary, speech and pinned reviewer selections also retain their selected saved provider source; sponsorship alone never replaces a BYOK key. Removing a provider clears only selections owned by that entry, preserving selections owned by the surviving entry. Subscription admission is repeated for each effective model and scope before its member key can be used. Direct vision requests, the cost probe, and hidden image planning use the same spend admission as provider adapters. Promoting a primary removes only fallbacks with the same model and credential source. Stored entries without an explicit source use the model’s native provider credentials.

Custom endpoint fallbacks must belong to the route’s user or server before generation loads a saved key. Server persona loaders resolve stored endpoint IDs without enforcing ownership, so generation checks both owner fields: personal endpoints have that user and no server; server endpoints have that server and no user. Credential lookup uses the route owner after this check.

Deferred server route fallback

Personal BYOK and subscription turns outside impersonation resolve the deferred server route only after their personal attempts fail. The user’s fallback setting and user_byok_mode can withhold it. Server cooldown and quota admission must pass before server attempts are appended. A personal BYOK turn in a DM can use only that same user’s DM workspace route; a personal subscription cannot change funding scope through a DM fallback. Running a server attempt switches credential attribution to the server. Tools retain their own routing.

Context preparation and prefill

Each attempt prepares media for its capabilities, formats prefill as a model continuation or tail directive, and trims history to known context limits with an output reserve. Context is assembled once for the planned model, so an attempt whose provider config carries no tools drops the items marked requiresToolCalling (the same-channel short-term memory block and its nudge). The prefill’s tokens are reserved against the same budget but kept out of the truncated list, because a tail directive is a dialogue user turn the truncator would otherwise protect in place of the real trigger.

Personal Text attempts apply the active provider’s saved sampler settings before request construction and history truncation. An explicit personal output-token limit overrides the server limit for both; an unset personal limit inherits the server setting. Both paths clamp the limit to the model’s ceiling.

NovelAI receives the original native context and one assistant prefill for each attempt. Its own fitter uses the key in the attempt’s provider config to resolve the subscription window and count the final rendered prompt. Generic history truncation returns no budget for NovelAI. A rotated credential therefore prepares its request from the original source instead of an already-trimmed list. Other providers retain the preset-assembled context and generic truncation.

Fallback text continuation

Messages a failed attempt already posted stay in the channel. The next key-rotation or fallback attempt continues the reply from where the failed one stopped:

Each failed attempt’s stream results carry the text it posted. carryDeliveredOutput() joins that text across attempts, together with its <details> metadata and the stream results post-turn stats count emoji and sprites from. Their usage is dropped, because post-turn stats attribute all usage to the model that finished the turn.
Once any text is carried, it replaces the turn’s /respond or server prefill. Each key attempt applies it as a continuation prefill and trims history to make room for it. The prefill is a native trailing assistant turn when the model can continue one. Otherwise it is a [System:] tail directive that shows the carried text as already sent and asks only for what follows, because the echo strip matches character by character and one divergence in a long echo reposts the reply. The stream still strips an echo of the carried text, so only the continuation is posted. Tool macros are not expanded in carried text.
A provider error with a suppressed embed does not flush the stream buffer, so an unfinished clause is never posted as a fragment for the continuation to repeat. An error the user sees still flushes it first. Unhumanized replies aggregate into one final send, so their failed attempts post nothing and the next attempt starts fresh.
An empty continuation completes the turn with the carried text, unless the emptiness is a failure the empty-response retry handles specifically: an output-token cap or a speaker-guard discard. NovelAI handles held fragments within its own streaming session and reports an exhausted empty continuation as a provider error.
prependDeliveredOutput() adds the carried text to the final result, including a terminal failure, so short-term memory records everything posted.
After an SDK timeout, the tool loop awaits the abandoned stream for up to STREAM_ABANDONED_SETTLE_TIMEOUT_MS (5000ms) so an in-flight send is part of the carried text. A send that outlives this window is visible but missing from the continuation prefill.
Response Drafting holds text until review, so a failed drafting attempt has nothing posted to continue and discards its pending text instead.

Error suppression and notices

Suppression during failover: error embeds are suppressed while further rotation keys or fallback models remain available, preventing transient provider hiccups from posting false failure notices.
Fallback notice: if a fallback model succeeds (index > 0), sendFallbackModelUsageNotice() renders a compact Discord button notice informing users that a fallback model answered the turn. Each failed attempt is listed by error type and display-safe code only, because context assembly reads the notice back for the model; the upstream message stays in the operator log.
Impersonation completion: provider error results carry providerFailureReported after the stream handler schedules their report. The impersonation command keeps its safe error reply and skips a duplicate report; failures outside that stream path retain command-level error logging.
Final emission: if all attempts fail, only the terminal error is emitted to the response sink.

Constraints and rationale

Each generation attempt owns its credential source. Personal attempts use the user’s saved key and bypass server rotation queries and retries. Deferred server attempts use server credentials and rotation only after server admission succeeds.

Failover effects: failover does not roll back tool writes, external requests, or independent tool messages. Off starts a new tool history on replacement attempts, but successful calls are recorded by request identity in ChatTurnContext.completedToolCalls: a replacement attempt that repeats an identical call gets the recorded result instead of running it again, and is told which tools already ran. A call with different arguments still executes. With Response Drafting On, shared history also survives fallback.
Deferred server admission: resolving server fallback routes only on personal failure prevents unnecessary database writes and server quota reservations on successful member turns.
Per-turn cleanup: a finally block invokes sink cleanup after exceptions. Temporary webhook deletion is best-effort, as described by Response Sink.

Response-text review

Response Drafting On creates one ResponseReviewState across model fallback and key rotation. Ordinary prose is held by the existing presentation parser; Off streams normally. Impersonation and text-suppressed work retain their existing paths. Failed attempts discard pending presentation while preserving successful tools, rejections, correction budgets and reported usage.

responseReview.ts builds a bounded private packet from admitted persona/task evidence, privacy-filtered context and actual tool history, and renders it as a tagged transcript. Evidence fixed for the turn precedes the candidate, findings and tool outcomes, so a second review reuses the provider’s prompt prefix cache. Pending prose is labeled separately from history. The reviewer reads as a blind model: admitted media and tool images become placeholders. Available tools are a compact catalog of names, descriptions and parameter names. The workspace’s Content to Check selection filters persona instructions, example dialogues, recent conversation, memories and other context, and tools before packet budgeting. The draft, triggering message and reply target remain required. Disabled persona instructions do not cause a missing-persona failure. With Tool Calls unchecked, response review omits tool definitions, completed tool outcomes, the correction count and the tool_use finding category, and skips proposed-call reviews. Coverage records the selection. Incomplete required evidence, a candidate containing a credential-shaped string (containsCredential, not the log redactor’s key: value heuristic, which matches ordinary prose), or tool arguments requiring credential or user-info redaction make review unavailable, and the trace category names the gap. The packet applies the same credential-shape patterns to admitted prose, tool descriptions and rule reports. Structured tool arguments, outcomes and rule reports also mask values under credential keys; the proposed-call gate refuses arguments changed by either check. Log-only heuristics do not alter reviewer evidence. Proposed voice-message arguments retain the script and delivery instructions for review, while diagnostic storage and completed-call evidence retain only their character counts; submitted user-info values remain excluded from review. Inherited reviewers use the actual author attempt’s model and key; pinned reviewers revalidate their owned registration and credentials. Either way the request follows the reviewer provider’s saved thinking_level and asks for the chat reply budget, because a reasoning model spends its thinking from the output allowance before the JSON verdict. Anthropic ignores the level: its structured call forces a tool, which the API refuses alongside extended thinking.

Review can pass, request one author revision, or become unavailable. Two response reviews bound the turn; another revision verdict delivers the latest candidate. Unavailability ends review for the turn and permits current prose, while prior tool rejections remain blocked. Revision preserves successful tool outcomes. /kill, follow-up interruption and abort discard held prose. Actual tools use the separate pre-execution checkpoint.

An optional rule checker is a tool on an owned prose_linter MCP registration whose only required input is a text string. Its output (structured content, JSON text or plain text, at most 32 KB) goes to the reviewer as <prose_linter_report advisory="true"> and forces a detailed review. Raw output cannot reach author feedback, memory or logs. getGuildMCPRouting excludes prose_linter registrations, so the author never sees or dispatches their tools. Checker failure proceeds to review; an empty report cannot approve prose. A binding whose registration is disabled or removed acts as no checker: it logs no error and does not force review.

INFO traces describe outcomes and budgets without private packets. Reviewer transports retain scrubbed failure bodies and malformed output in private warning diagnostics, correlated with the review attempt. Unexpected exceptions retain safe metadata only because their messages can contain private draft or evidence text. Returned failures contain only safe metadata. Expected provider refusals, timeouts and upstream outages produce a metric plus a retained warning; unexpected failures remain errors.

Source pointers

src/utils/chat/generationTurn.ts: runGenerationTurn() and generation plan execution.
src/utils/chat/toolLoop.ts: runToolLoop() function-calling loop.
src/utils/chat/assistantPrefill.ts: applyAssistantPrefill() for manual, server, and continuation prefills.
src/utils/security/keyRotation.ts: API key selection and rotation tracking.
src/utils/chat/responseReview.ts: shared state, admitted evidence and reviewer lifecycle.
src/utils/chat/responseRuleCheck.ts: internal checker validation and bounded evidence.

Reviewer resolution diagnostics distinguish missing, deprecated, unavailable, or unsupported models, unsupported providers, unavailable provider options, and missing credentials. They include the model ID, codename, and provider when available, without credentials or draft content.