Setup: Local LLM

TomoriBot can use any OpenAI-compatible local LLM server for text generation and embeddings.

No env vars needed

Local models are registered through Discord slash commands and stored encrypted in the database. There’s no .env setting for them.

1. Run your model server

This guide walks through the process using Ollama as an example because it is the easiest to get started with, you can use anything you want especially if you’re a more technical user already!

Install Ollama. The examples below use Google’s Gemma 4, but anything in Ollama’s library works.

Which size should I pull?

Local models run in your GPU’s VRAM (the memory built into your graphics card, separate from your system RAM). Rule of thumb: a model needs at least its download size free in VRAM, plus ~1-2 GB of headroom for the conversation context. Pick the largest Gemma 4 that fits your card:

Your GPU VRAM	Best fit	Download (approx.)
~8 GB	gemma4:e2b	7.2 GB
~12 GB	gemma4:12b	7.6 GB
~16 GB	gemma4:12b (fits fully), or gemma4:26b	7.6 / 18 GB
24 GB+	gemma4:26b or gemma4:31b	18 / 20 GB

Downloads are Ollama’s default-quantization sizes; see the model page for exact figures. Not sure how much VRAM you have? On Windows: Task Manager > Performance > GPU, read “Dedicated GPU memory.”

Pull your chosen size and start the server:

ollama pull gemma4:12b     # swap for the tag that fits your VRAM
ollama serve               # listens on http://127.0.0.1:11434

Confirm it is reachable from the machine TomoriBot runs on:

curl http://127.0.0.1:11434/v1/models

Note the exact installed tag, as this is the Model Name you’ll register:

ollama list
# NAME              ID            SIZE
# gemma4:12b        a1b2c3d4...   7.6 GB

2. Register it in Discord

Run /providers (server-wide) or /personal providers (just you), choose Add New Custom Endpoint, and enter:

Field	Value for Ollama
endpoint_label	A name you choose, e.g. home-ollama
API Compatibility	OpenAI-Compatible (recommended) or Ollama
endpoint_url	http://127.0.0.1:11434/v1 for OpenAI-Compatible · http://127.0.0.1:11434 for Ollama
auth_token	(leave blank)

Pick the URL that matches the API compatibility

Both OpenAI-Compatible and Ollama accept the bare root and normalize it to the /v1 base. /chat/completions is appended automatically, so do not add it. URLs that already carry a path, such as https://openrouter.ai/api/v1 or a gateway prefix, are stored verbatim.

After saving the connection, select it and choose + Add new Text Model from its model dropdown. Fill in:

• Model Name (exact API ID): gemma4:12b, the exact tag from ollama list.
• Context Window Override: optional, Ollama / KoboldCPP only. Set this (e.g. 8192, 16384) to raise Ollama’s default num_ctx, which is otherwise small enough to truncate long TomoriBot context. Leave blank to use the server default.
• Toggles: enable Tool calling if the model supports function calling; enable Image input only for a vision model; Structured output if the model handles JSON schemas well. For our example, Gemma 4 supports all of them, so tick them all.

TomoriBot validates the connection when you save it. If it reports the endpoint is unreachable, the usual cause is a localhost/Docker mismatch or a missing/extra /v1 (see gotchas).

Adding the model makes it the active text model automatically. Start chatting to try it. If it isn’t active for some reason, run /config > Models > Switch Models and select your newly registered model.

3. (Optional) Local embeddings for RAG

Select the saved endpoint and use its model dropdown to add an Embedding model (e.g. ollama pull nomic-embed-text, Model Name nomic-embed-text:latest). RAG features also need pgvector installed in Postgres. You can see the manual setup guide here.

4. (Optional) Share one GPU with ComfyUI

If the text model and ComfyUI run on the same GPU, TomoriBot can unload the text model while a ComfyUI image or video job runs. Text replies that arrive during the job wait until the GPU is free again.

Open the saved endpoint, press Edit Endpoint, and check Unload model during ComfyUI jobs under Endpoint Behavior. TomoriBot checks which server is at the URL when you save:

• Ollama: works as installed. TomoriBot asks Ollama to unload the model before the job, and Ollama reloads it on the next text request.
• KoboldCpp: start KoboldCpp with --admin and --admindir, and set the endpoint’s auth token to the admin password. TomoriBot uses KoboldCpp’s admin reload API to unload the model and load it again afterwards.

Any other server is refused when you save, because generic OpenAI-compatible servers have no shared way to unload a model.

Other servers

TomoriBot supports any OpenAI-compatible endpoint, but the servers below are recommended. All of these use the same flow: only the URL and a couple of notes change.

KoboldCPP

• Start with OpenAI-compat enabled (built in). Default: http://127.0.0.1:5001/v1.
• API Compatibility: OpenAI-Compatible. endpoint_url: http://127.0.0.1:5001/v1.
• Honors the Context Window Override like Ollama.
• To unload the model during ComfyUI jobs, also launch with --admin and --admindir (see Share one GPU with ComfyUI).
• Loads GGUF models; the Model Name is whatever the loaded model reports (often the file stem), check KoboldCPP’s /v1/models response.

llama.cpp (llama-server)

• Build or install llama.cpp, then serve a GGUF with its bundled OpenAI-compatible server:

  llama-server -m model.gguf -c 16384 --host 0.0.0.0 --port 8080

• API Compatibility: OpenAI-Compatible. endpoint_url: http://127.0.0.1:8080/v1.
• Set the context window at launch with -c which is the modal’s Context Window Override is Ollama/KoboldCPP-only and has no effect here.
• Model Name is whatever /v1/models reports; give it a clean one with --alias my-model.
• If you started it with --api-key, put that key in auth_token.

LM Studio

• In LM Studio, start the Local Server (Developer tab). Default: http://127.0.0.1:1234/v1.
• API Compatibility: OpenAI-Compatible. endpoint_url: http://127.0.0.1:1234/v1.
• Model Name is the identifier LM Studio shows for the loaded model.

vLLM

• Serve with the OpenAI-compatible server: vllm serve <model> → http://127.0.0.1:8000/v1.
• API Compatibility: OpenAI-Compatible. endpoint_url: http://127.0.0.1:8000/v1.
• If you launched vLLM with --api-key, put that key in auth_token.
• Model Name is the served model path/name (matches /v1/models).

LiteLLM (proxy over many backends)

• Run the LiteLLM proxy; default: http://127.0.0.1:4000/v1.
• API Compatibility: OpenAI-Compatible. endpoint_url: http://127.0.0.1:4000/v1.
• Model Name is the model alias you defined in LiteLLM’s config.
• If the proxy enforces a master key, set it in auth_token.

ChatMock (ChatGPT account / Codex CLI)

Has its own dedicated guide because of a system-prompt workaround: Setup: ChatMock.

Picking models from Hugging Face

Beyond Ollama’s curated library, Hugging Face hosts thousands of community models. KoboldCPP, llama.cpp, and LM Studio can all load the GGUF format which is a single-file package you download and point the server at.

1. Find a GGUF. Search Hugging Face for your model plus “GGUF” community quantizers like bartowski publish GGUF builds of most popular models soon after release. Prefer an instruct/chat variant (names ending in -Instruct or -Chat); base models don’t hold a conversation.
2. Pick a quant that fits your VRAM. A repo lists the same model at many quant levels, and a file’s size ≈ the VRAM it needs (plus ~1-2 GB for context, same rule as the sizing table above). Download the single .gguf for your choice.
3. Load it. Start KoboldCPP or llama-server with that file (see Other servers), then register the endpoint in Discord as usual.

Which quant? Q4 or Q5 is the sweet spot

Quantization stores each weight in fewer bits to shrink the model, at a small quality cost. The code in names like Q4_K_M / Q5_K_M is the bits-per-weight: 4-bit (Q4) or 5-bit (Q5) is the usual sweet spot as most of the quality for roughly half the size of 8-bit. Below 4-bit degrades quickly. And for a fixed VRAM budget, a larger model at Q4 usually beats a smaller model at Q8.

Notes & gotchas

• One endpoint entry per label. To register several models that share one server, select the saved endpoint and use its model dropdown again. Use distinct labels for genuinely different servers or API protocols.
• Model Name is the API identifier. It is the exact string sent to the server. Getting it wrong is the most common “it connected but responses fail” cause.
• Running TomoriBot in Docker? localhost inside the container is not your host. Use http://host.docker.internal:<port> or the host’s LAN IP, and bind the model server to 0.0.0.0. The bundled Compose file maps that name on Linux too. With your own Compose file or docker run on Linux, add extra_hosts: ["host.docker.internal:host-gateway"] (--add-host=host.docker.internal:host-gateway for docker run).
• Private addresses need ALLOW_PRIVATE_NETWORK to allow them. With true, endpoints on localhost, LAN hosts, and host.docker.internal work over plain HTTP. A host:port list such as localhost:5001 allows only those servers, which suits sharing your models with a whole Discord server. FETCH_URL_ALLOW_PRIVATE_NETWORK is a separate setting for the fetch_url tool and does not change endpoint rules.