Skip to content

Refactor Record

Historical record of the plugin-architecture-prerequisite refactor (refactor/plugin-architecture branch, Phases 1–5.5e). Covers module restructuring decisions, behavioral verification results, DB layer reorganization, and cache invalidation ownership after the repository migration.

Snapshot date: 2026-05-28


Added a focused assembler seam for backend-sensitive tool schemas. Built-in tools now pass through src/tools/assembly.ts after centralized availability filtering and before provider adapter serialization. Tools without assembleForContext() are returned unchanged; tools with dynamic backend constraints can return a narrowed per-turn Tool variant or null.

Initial adopters:

  • web_search now advertises only categories supported by the active backend: SearXNG all categories, Brave base categories, DuckDuckGo/Felo text-only, and no tool when no search backend is available.
  • generate_image now prunes image-to-image, inpaint, and outpaint arguments unless the configured standard image backend supports them. Custom image endpoints declare image request support through workflow_supports; ComfyUI defaults to text-to-image, image-to-image, and negative prompts, while generic endpoints default to text-to-image only.
  • generate_voice_message moved script-markup and VoiceDesign schema shaping out of availability.ts into the tool assembler hook.

Deferred candidates: video generation provider/model option tables, permission-sensitive Discord message/thread tools, sticker-name schema hints, and fetch-url backend details.

Image tags moved out of /novelai image-tags into provider-neutral commands: /persona image-tags, /personal image-tags, and /config image-tags default-positive/default-negative. User and persona tags are rendered in context as public Physical Appearance lines. generate_image now receives default positive tag guidance, while default negative tags are consumed only by NovelAI or custom image endpoints with the Negative Prompt support checkbox enabled.


Records which refactor phases produced real responsibility-owned modules and which left thin facades over a new god file. The integrity check script scripts/archived/checkRefactorIntegrity.ts was used to detect facade-shaped barrels, active *.legacy.ts files, and oversized runtime/orchestrator files.

Phase itemCurrent surfaceHidden or remaining implementationStatusOwner / responsibility
#1 Fragment localessrc/locales/en-US/ 36 files, 6,184 lines; src/locales/ja/ 36 files, 6,195 linesNone identifiedReal splitLocale categories
#2 Simplify stringHelper.tsDeletedProcessor modules under src/utils/text/processors/; markdown-table helpers under src/utils/text/markdownTable.tsCompleteText processors
#3 Decouple index.tssrc/index.ts, 24 linessrc/init/* modulesReal splitStartup initialization
#4b Repository patternRepository classes under src/utils/db/repositories/; repositoryReadSql.ts 7-line barrel; repositoryWriteSql.ts 6-line barrelDomain SQL lives in src/utils/db/repositories/*Sql.ts; large LLM/persona/server transaction modules tracked in the Intentional Large File tableReal domain split with tracked compatibility barrelsRepository-owned SQL by domain
#4b Import/Export splitsrc/utils/db/repositories/ExportRepository.ts, 671 lines; src/utils/db/repositories/ImportRepository.ts, 774 linesDeleted ImportExportRepository.ts, repositoryExportSql.ts, repositoryImportSql.ts; no remaining sibling filesCompleteExport-direction SQL (ExportRepository); import-direction SQL + cache invalidation (ImportRepository)
#5 /tool status splitsrc/commands/tool/status.ts, 32 lines; src/utils/metrics/status/command.ts, 73 linesDeleted statusCommandMetrics.ts and status/commandImplementation.ts; owned modules under src/utils/metrics/status/ are all <600 linesCompleteStatus command coordination, page builders, and redaction-aware formatters
#5 /tool compact splitsrc/commands/tool/compact.ts, 33 linessrc/utils/compaction/compactOrchestrator.ts, 1,102 linesFacade-only splitCompaction workflow stages
#6 Base stream adapterProvider adapters and BaseStreamAdapterProvider-specific large files remain provider-ownedNo facade findingProvider stream adapters
#6.5 Provider registrysrc/utils/providerInfoRegistry.ts and provider-local providerInfo.ts filesNone identifiedReal splitProvider metadata discovery
#7 Tool registry splitsrc/tools/toolRegistry.ts, 514 linessrc/tools/availability.ts, 387 linesReal partial splitTool registry vs. availability
#8 Discord UI helperssrc/utils/discord/interactionHelper.ts, 1 line; owned modules under src/utils/discord/ui/src/utils/discord/ui/interactionCore.ts retains shared Discord UI internals after legacy-file deletionLegacy file eliminatedDiscord UI flows
#8 Webhook helperssrc/utils/discord/webhookManager.ts, 1 line; owned modules under src/utils/discord/webhook/src/utils/discord/webhook/webhookCore.ts retains shared webhook internals after legacy-file deletionLegacy file eliminatedWebhook lifecycle, identity, dispatch, fallback
#9 Matrix bridgematrixManager.ts, 9 lines; public responsibility files mostly thinsrc/utils/bridges/matrix/runtime.ts, 1,405 linesFacade-only splitMatrix client, events, rooms, state sync, user mapping, media
#10 Context buildercontextBuilder.ts, 5 lines; owned modules under src/utils/text/context/ are all <600 linesDeleted context/core/builderImplementation.tsCompleteContext assembly pipeline
#11 Stream orchestratorstreamOrchestrator.ts, 1 line; owned modules under src/utils/discord/stream/ are all <600 linesDeleted stream/core/orchestratorImplementation.tsCompleteStream state machine, stop registry, buffer flushing, segment processing, message delivery, UI updates, and thought logs
#12b / #12c / 5.5d ChattomoriChat.ts, ~145 lines; stage modules under src/utils/chat/Deleted turnRunner.ts; chat implementation lives in admission.ts, admissionQueue.ts, channelQueue.ts, turnPlanner.ts, contextPipeline.ts, contextAnnotations.ts, contextEmbeds.ts, contextMedia.ts, generationTurn.ts, toolLoop.ts, responseEmitter.ts, postTurnEffects.ts, and small queue/identity helpersCompleteChat admission, queueing, turn planning, context, provider turn, tool loop, response, post-turn effects
#13 Event handler eager-loadNot completedN/AOut of scopeEvent loading
Extra: Web-search unification (Phase 1)Single web_search(query, category) BaseTool under src/tools/webSearch/; engine layer (braveEngine.ts, duckduckgoEngine.ts, feloEngine.ts, dispatcher.ts) implementing WebSearchEngine chainInternalBraveWebSearchTool / InternalBraveImageSearchTool / InternalBraveVideoSearchTool / InternalBraveNewsSearchTool under src/tools/restAPIs/brave/internal/braveServiceClasses.ts — no longer LLM-visible. DDG/Felo accessed via DuckDuckGoHandler.executeWebSearchInternal() and executeFeloSearchInternal().Complete (Phase 1)Engine-chain dispatch for web search; replaces the previous 4-tool Brave surface and the per-adapter Brave-key dedup logic. Phase 2 will add a SearxngEngine to the chain.
Extra: URL-fetch unification (Phase 1 + Browser Engines Phase 2)Single fetch_url(url, max_length?, start_index?, raw?) BaseTool under src/tools/fetchUrl/; dispatcher chain is configurable with default crawl4ai -> browserless -> mcp_fetchBundled MCP fetch remains connected but is consumed internally through FetchHandler.executeFetchInternal() / McpFetchEngine; Crawl4AI is consumed internally through restAPIs/crawl4ai/ and Crawl4aiEngine; Browserless is consumed internally through restAPIs/browserless/ and BrowserlessEngine; raw global MCP fetch and browser sidecar REST calls are hidden from the LLM. Guild url_fetcher replacements suppress bundled fetch_url when present.Complete (Phase 1 + Crawl4AI and Browserless Phase 2)One LLM-visible URL-fetch surface while preserving existing MCP fetch fallback behavior. Crawl4AI adds optional browser-rendered markdown via /md; Browserless adds optional rendered HTML via /content converted to markdown in-process.

The chat entry point exposes the named stage sequence:

const incoming = normalizeChatInvocation(...);
const admission = await evaluateChatAdmission(incoming);
if (admission.disposition !== "run") {
await handleChatDisposition(admission);
return;
}
await runWithChannelLock(admission, async (lockedTurn) => {
const turnPlan = await planChatTurns(lockedTurn);
for (const turn of turnPlan.turns) {
const context = await buildChatTurnContext(turn);
const responseSink = createChatResponseSink(context);
const result = await runGenerationTurn(context, responseSink);
await runPostTurnEffects(context, result);
}
});

turnRunner.ts was deleted; response delivery is represented as a response sink used during generation because Discord streaming happens while the provider turn is running.

These files were retained above the 600-line heuristic because each owns one clear responsibility, has a narrow public surface named after that responsibility, and splitting would have introduced meaningless file fragmentation.

PathLinesResponsibilityRationale
src/utils/db/repositories/LlmModelRepository.ts~600Global model catalogllms, embedding_models, image_diffusion_models, video_generation_models reads are cohesive: all share provider normalization, deprecation filtering, and OpenRouter scope delegation. Split from LlmRepository + llmReadSql.ts.
src/utils/db/repositories/LlmProviderRepository.ts1,709Saved provider configs, custom endpoints, OpenRouter registrations7 tables share OpenRouter scope SQL helpers and cache-invalidation boundary. Exceeds the 1,000-line heuristic by 709 lines; no further split warranted before provider-table partitioning.
src/utils/db/repositories/LlmOverrideRepository.ts556Channel/persona override assignments and fallback refschannel_llm_overrides, persona_configs (llm_id), and tomori_configs (fallback columns) writes share cache-invalidation semantics; bulk restore calls private SQL helpers to avoid per-override cache thrashing.
src/utils/db/repositories/PresetRepository.ts1,150TomoriBot preset export/import + SillyTavern preset CRUD + ST card conversionsillyTavernImport.ts (545 lines) is pure text-processing tightly coupled to convertSillyTavernJsonToPresetData and the ST preset insertion workflow. Extracting it would add an import-dependency layer with no cohesion gain — callers always pair parsing with insertion.
src/utils/db/repositories/PersonaRepository.ts859Persona state loading + writeloadTomoriState and loadAllPersonasForServer stay together because both construct the same composite persona runtime state. Combined 859 lines after Stage C inline of personaReadSql + personaWriteSql.
src/utils/db/repositories/ServerRepository.ts941Server identity: setup, emojis/stickers, webhooks, blacklistsqlSetupServer is one atomic transaction (~400 SQL lines) that creates server, persona, config, and initial emoji rows — splitting it would separate transactional setup context from its server repository owner.
src/utils/db/repositories/ServerScheduleRepository.ts850Reminder + random-trigger schedule domainReminders and random triggers share scheduled-work nudge behavior and form a cohesive scheduling domain split from server identity.
src/utils/db/repositories/ExportRepository.ts671All data export operationsExport methods are all read-only; sanitizeForJson + sanitizeMemoryItems helpers are tightly coupled to every export path. The large exportServerData method carries the full config COALESCE query (60+ fields).
src/utils/db/repositories/ImportRepository.ts774All data import operations + cache invalidationsqlImportServerConfig carries the full tomori_configs UPDATE (60+ fields, repeated twice for server-id vs tomori-id fallback). Private SQL helpers (ensureUserId, resolveServerId, resolveMainTomoriScope) are shared across all import paths.

Two public API boundary barrels were retained. Each marks a stable subsystem boundary where callers consume the capability as a single unit and the internal split is an implementation detail.

PathLinesRationale
src/utils/text/contextBuilder.ts5Callers consume context building as one subsystem capability while mention normalization, preset routing, native assembly, memories, RAG, server assets, participants, and dialogue history live in responsibility-owned modules. 11 import sites.
src/utils/discord/streamOrchestrator.ts1Callers consume Discord streaming as one subsystem capability while stop requests, buffer flushing, segment processing, message delivery, UI updates, errors, text config, mention resolution, and thought logs live in responsibility-owned modules. 15 import sites.

For each function deleted during Phases 1–5: did its behavior survive somewhere — renamed, relocated, or inlined — or did it silently disappear?

A behavioral regression is narrowly defined: a function performed a real, observable behavior; the function was deleted; and no equivalent code (by name OR inlined body) exists in the current tree. Pure dead-code removal, rename-only moves, and intentional consolidations are not regressions.

For each phase commit:

  1. git diff <commit>^..<commit> --diff-filter=D --name-only enumerated fully-deleted files.
  2. git diff <commit>^..<commit> searched for ^-export , ^-function , ^-async function to find function definitions that vanished from surviving files.
  3. Each candidate deletion was checked against current src/ with Grep — first by function name (catches renames), then by 1–2 distinctive identifiers from the old body (catches inlining).
  4. A finding is only a regression if BOTH the name AND the distinctive-body search come up empty.

Type-only deletions, deleted tests, and locale-file shuffles in Phase 1 were skipped.

PhaseCommitTitleAudit verdict
1d9d284bbPhase 1: Locales, String Helpers, Index Entry PointClean
2.19836d968Phase 2: Data Access (1/2)Clean
2.2b3a3bd5fPhase 2: Data Access (2/2)Clean
3.1aff0b537Phase 3: Core Abstractions & Integrations (1/2)Clean
3.27a631afbPhase 3: Core Abstractions & Integrations (2/2)Clean
4643aaef1Phase 4: Context & OutputClean
544c975e0Phase 5: OrchestratorClean (caveat below)

Phase 1chunkMessage, cleanLLMOutput, replaceMentionHandles, normalizeCustomEmojisForLlm, findMarkdownCodeRanges, truncateBeforeGenericSpeakerLine, isGenericSpeakerStopLabel, escapeRegExp → moved to src/utils/text/processors/. index.ts bootstrap split into src/init/* modules.

Phase 2.1 — Pure adapter-layer insertion: queries previously called inline against Bun.sql were wrapped in *Repository classes with the same SQL bodies. Compile errors at every caller site forced exhaustive rewiring.

Phase 2.2 — Status command internals → src/utils/metrics/statusCommandMetrics.ts and submodules. Compact command internals → src/utils/compaction/compactOrchestrator.ts. Channel LLM cache functions → src/utils/cache/channelLlmCacheStore.ts.

Phase 3.1 — Stream adapter classes refactored to extend BaseStreamAdapter. Duplicated methods hoisted; provider-specific overrides remain in subclasses.

Phase 3.2interactionHelper.ts exports split across src/utils/discord/ui/{buttons,confirmation,embeds,errors,modals,pagination,statusComponents,interactionCore}.ts. Matrix bridge moved from src/utils/matrix/index.ts to src/utils/bridges/matrix/.

Phase 4 — Internal context-building functions extracted from src/utils/text/contextBuilder.ts into src/utils/text/context/{history,memories,rag,templates,types}.ts. Stream orchestration helpers split into src/utils/discord/stream/ submodules.

Phase 5 (caveat) — Phase 5 moved tomoriChat.ts (~9,500 lines) into src/utils/chat/turnRunner.ts as a near-verbatim relocation. Function bodies survived intact at commit 44c975e0. The behavioral regressions later catalogued in the Phase 5.5d appendix (plans/archive/refactor/phases/phase-5.5d-chat-drain.md) were introduced by Phase 5.5d’s drain of turnRunner.ts, not by Phase 5’s move.

Phases 1–5 were predominantly relocation refactors: files were deleted and recreated under new paths with the same function set. Import-site rewrites force compile errors at every caller, which surfaces missing functions immediately.

Phase 5.5d broke that pattern — it was a reshape refactor that dissolved runChatTurn() into named stages with different signatures and control flow. There was no 1:1 import rewrite to force errors; pieces of the old function body could be quietly dropped while the file still compiled. The Phase 5.5d appendix proposes a per-function before/after diff audit as the template for any future drain work.


Records the final shape of src/utils/db/ after Phase 5.5e consolidation of the 26 orphan files at the db/ root.

src/utils/db/
├── client.ts # infrastructure
├── initializeDatabase.ts # infrastructure
├── sqlSecurity.ts # infrastructure
├── sqlSplitter.ts # infrastructure
├── ragAvailability.ts # renamed from ragDetection.ts
└── repositories/
├── index.ts # ≤50 lines: instances + types only
├── IRepository.ts
├── ServerRepository.ts # core: setup, emojis/stickers, webhooks, blacklist
├── ServerScheduleRepository.ts # reminders + random triggers (split from ServerRepository)
├── UserRepository.ts # + personalSpotlight (folded; 965 lines — under limit)
├── PersonaRepository.ts # + persona-scoped memoryLimits checks
├── ConfigRepository.ts
├── LlmModelRepository.ts # global model catalog (split from LlmRepository)
├── LlmProviderRepository.ts # saved configs + OpenRouter registrations (split from LlmRepository)
├── LlmOverrideRepository.ts # channel/persona override assignments (split from LlmRepository)
├── ToolRepository.ts # + guildMcpDb
├── RagRepository.ts
├── ExportRepository.ts # split from ImportExportRepository
├── ImportRepository.ts # split from ImportExportRepository
├── PersonalMemoryRepository.ts # + checkPersonalMemoryLimit
├── ServerMemoryRepository.ts # + checkServerMemoryLimit
├── ShortTermMemoryRepository.ts
├── ConditioningMemoryRepository.ts # + conditioningDb
├── WhitelistRepository.ts # also absorbed whitelist delegation from ServerRepository
├── PresetRepository.ts
└── CooldownRepository.ts

Outside src/utils/db/:

src/utils/persona/personaAccess.ts # moved from db/personaAccess.ts (pure functional, no DB access)
src/utils/misc/memoryLimits.ts # env-loading half of db/memoryLimits.ts

SQL inlining: All *ReadSql.ts / *WriteSql.ts siblings were dissolved into their repository as private methods and deleted. The public/private boundary is enforced by TypeScript’s private keyword, not by folder convention.

Budget was ~1,000 lines per Repository file once SQL is inlined.

RepositoryClass LOCSQL sibling LOCCombinedNotes
RagRepository1370137
ToolRepository15623179+~110 from guildMcpDb fold
ConditioningMemoryRepository1190119+~130 from conditioningDb fold
ShortTermMemoryRepository147213360
PersonalMemoryRepository1750175
ServerMemoryRepository1380138
PersonaRepository91717808
UserRepository490407897965 post-personalSpotlight fold — under limit
ImportExportRepository2231,5071,730Over budget; PresetRepository split reduces; final split by direction into ExportRepository + ImportRepository
ConfigRepository6005321,132Marginally over; SQL inlined and re-measured — no split warranted
ServerRepository3521,2781,630Over budget; split into core + ServerScheduleRepository
LlmRepository7063,0073,713Severely over; 3-way split into LlmModelRepository + LlmProviderRepository + LlmOverrideRepository

Group A — Infrastructure (stays at db/ root)

FileLOCDisposition
client.ts208Unchanged
initializeDatabase.ts108Unchanged
sqlSecurity.ts116Unchanged
sqlSplitter.ts129Unchanged

Group B — SQL barrels (deleted)

FileLOCDisposition
repositoryExportSql.ts753SQL inlined as private methods into ImportExportRepository; file deleted
repositoryImportSql.ts754Same
repositoryReadSql.ts7Deleted (barrel into SQL siblings, which were themselves dissolved)
repositoryWriteSql.ts6Deleted (same reason)

Group C — Folded into existing repositories

FileLOCTarget RepositoryWhy
emojiStickerSync.ts331ServerRepositoryServer-scoped sync logic; SQL (~250 lines) inlined as private methods
managedWebhookDb.ts197ServerRepositorySingle-table, server-scoped; encryption pattern matches guildMcpDb
guildMcpDb.ts237ToolRepositoryMCP servers are tool sources
conditioningDb.ts358ConditioningMemoryRepositoryConditioning history is the natural extension of conditioning memory
personalSpotlight.ts361UserRepositoryPost-fold UserRepository is 965 lines — under limit

Group D — New repositories

New RepositorySource files absorbedWhy a new repository
WhitelistRepositorychannelWhitelist.ts (283), personaWhitelist.ts (122), roleWhitelist.ts (71)Folding into ServerRepository would push it past budget; whitelists are a coherent standalone domain
PresetRepositorypresetExport.ts (214), presetImport.ts (264), stPresetDb.ts (285), sillyTavernImport.ts (545)1,308 combined LOC; ST card ingestion is a distinct concern from TomoriBot export/import
CooldownRepositorycooldownManager.ts (365), cooldownsCleanup.ts (82), messageCooldown.ts (305)Duplication between cooldownManager and messageCooldown (both had isExemptFromCooldown variants) collapsed into one canonical pair

Group E — Moved out of db/

FileLOCDestinationWhy
personaAccess.ts22src/utils/persona/personaAccess.tsPure functional composition of isPersonaAllowedByWhitelistStatus + isPersonaAllowedByPersonalSpotlight; no DB access
ragDetection.ts45src/utils/db/ragAvailability.ts (renamed)Startup-time infrastructure; RagRepository does CRUD on documents/chunks and should not depend on availability detection

Group F — Split between locations

FileLOCSplit
memoryLimits.ts504getMemoryLimits() + env helpers + content validators → src/utils/misc/memoryLimits.ts; checkPersonalMemoryLimit()PersonalMemoryRepository; checkServerMemoryLimit()ServerMemoryRepository; checkTriggerWordLimit(), checkSampleDialogueLimit(), checkAttributeLimit()PersonaRepository

LlmRepository → 3-way split (combined ~3,713 lines, severely over budget)

New repositoryTables owned
LlmModelRepositoryllms, embedding_models, diffusion_models, video_generation_models
LlmProviderRepositorysaved_provider_configs, user_saved_provider_configs, custom_endpoints, openrouter_*_registrations
LlmOverrideRepositorychannel_llm_overrides, persona_llm_overrides, fallback refs

toExportShape() / fromExportShape() moved to LlmProviderRepository (saved provider configs and OpenRouter registrations are the exportable state; model catalog is global seed data).

ServerRepository → 2-way split (combined ~1,630 lines, over budget)

RepositoryTables owned
ServerRepository (core)servers, server_emojis, server_stickers, managed_webhooks
ServerScheduleRepositoryreminders, random_triggers

setupServer is a single unavoidably large transaction (~400 SQL lines); the marginal overrun of the core file was accepted and documented inline.

ConfigRepository — combined ~1,132 lines. SQL inlined and re-measured; no further split warranted given the uniform config-read-write surface.

ImportExportRepository — after PresetRepository absorbed ~1,308 LOC, the remaining export/import SQL still exceeded the 1,000-line heuristic. Split by direction: ExportRepository (read-only export paths) and ImportRepository (import paths + cache invalidation).


Records where cache invalidation lives after the repository migration. All invalidations listed are co-located with the write they guard: they execute after a confirmed successful DB write, in the same code path.

CacheOwning repository methods
User row/preferencesUserRepository.register, setPrivacyLevel, setPrivacyOptOut, toggleCrossServerShmOptIn, update, fromExportShape; personal-memory writes in PersonalMemoryRepository / ImportRepository
User blacklistUserRepository.removeBlacklistEntry and equivalent blacklist-add methods
Tomori state: config/provider/model settingsConfigRepository.update*Config() methods; LLM/provider-specific writes owned by LlmModelRepository, LlmProviderRepository, LlmOverrideRepository
Tomori state: persona/profile/settingsPersonaRepository.update; import/persona-creation/delete/default/swap flows
Tomori state: server memories/documents/historyServerMemoryRepository.add and edit/remove variants
Guild MCP configToolRepository.insertMcpServer, deleteMcpServer, updateMcpServerEnabled
Channel LLM overridesLlmOverrideRepository.setChannelLlmOverride, deleteChannelLlmOverride, clearAllChannelLlmOverrides
Whitelist decisionsWhitelistRepository.upsertChannelWhitelist, removeChannelWhitelist, replacePersonaWhitelistChannels, removeChannelPersonaWhitelist, upsertRoleWhitelist, removeRoleWhitelist
Full import/exportImportRepository.importPersonalSettings, importPersonalMemories, importServerConfig, importServerMemories, importPersonalData, importServerData

Caller-Owned (Intentionally Outside Repositories)

Section titled “Caller-Owned (Intentionally Outside Repositories)”
CacheCall sitesReason
Personal spotlightsrc/commands/personal/spotlight/set.ts, manage.tsDedicated personal-spotlight DB module; ownership stays here unless it later moves under a repository
ST preset cachesrc/utils/db/stPresetDb.ts (now PresetRepository)Write-after-success placement preserved during fold
Emoji/sticker cachesrc/events/guildEmojisUpdate/refreshEmojis.ts, guildStickersUpdate/refreshStickers.tsEvent-driven cache; invalidation follows Discord events, not DB writes
Matrix link cachesrc/commands/server/matrix/link.ts, unlink.tsMatrix bridge module — not part of the repository layer
Webhook cachesrc/utils/discord/webhook/ internal helpersCache keys are Discord webhook lifecycle state, not repository reads