# On-device models Agentak has two model APIs that run inference on the visitor's device. No prompt is sent to a model provider, no API key is needed, and usage cost is zero. The browser still needs to download runtime files or model weights before the first answer. ## On Device with wllama On Device (wllama) runs llama.cpp compiled to WebAssembly in a Worker in the current tab. The default loader imports wllama from jsDelivr, loads its wasm, and downloads public GGUF weights from Hugging Face. The browser caches the model file, and only one model is held in memory at a time. The built-in catalog is intentionally small: | Model | Download | Loaded context | KV per token | Reasoning | Tools | | -------------- | -------: | -------------: | -----------: | --------- | ----- | | LFM2.5 350M | 219 MB | 32,768 | 12 KiB | No | Yes | | MiniCPM5 1B | 688 MB | 16,384 | 24 KiB | Yes | Yes | | LFM2.5 1.2B | 696 MB | 32,768 | 12 KiB | No | Yes | | LFM2.5 2.6B | 1.6 GB | 32,768 | 16 KiB | No | Yes | | Qwen3.5 2B | 1.6 GB | 32,768 | 12 KiB | Yes | Yes | | Qwen3.5 4B | 1.9 GB | 16,384 | 32 KiB | Yes | Yes | | Granite 4.1 3B | 2.0 GB | 8,192 | 80 KiB | No | Yes | The LFM2.5 rows use Liquid's QAD Q4_0 checkpoints. These are distilled while quantized rather than quantized after training, so they hold about 97% of the full-weight score at the size of a Q4. The other rows use K-quants, and the quant falls as the model grows: the Qwen3.5 4B fits only as a dynamic Q2_K_XL. For tool calling, the LFM2.5 rows and Granite 4.1 3B are the strongest for their size. Ability falls off sharply below about 1B: Qwen reports BFCL-V4 of 50.3 for Qwen3.5 4B, 43.6 for the 2B and 25.3 for the 0.8B. All seven accept text. A local model can stream text, reasoning, and tool calls when its chat template supports them. For wllama, Agentak maps `off` to thinking disabled and every non-off level to thinking enabled; the local template receives an on/off choice rather than the full scale. The weights and KV cache share a 4 GiB WebAssembly heap, so the loaded window is what the heap has room for after the weights. The cost of a token is not the same for every model. A hybrid keeps attention in only a few of its layers and a cheap state in the rest, so it pays 12 to 32 KiB a token. Granite 4.1 3B has attention in all forty of its layers and pays 80 KiB, which is why it carries the shortest window here despite being trained for 128K. Each window above is the largest that keeps the weights and the cache inside about 2.6 GiB together. Performance depends on the device; a laptop may answer at a useful speed while a phone may not. ## Availability `wllamaSupported()` requires: - `WebAssembly` - `Worker` - a document that may load what the runtime is made of The default loader imports a remote script, which an MV3 extension page cannot do. A host that supplies its own source through `useWllamaSource()` answers that question for its own document, and the local row is then listed there too. The Agentak side panel does exactly this and lists wllama; see [Chrome extension](/extension). A phone lists the row on the same terms. The cost is higher there — the download is hundreds of MB over a connection that is often metered, a mobile browser reclaims the wasm heap when the tab goes to the background, and the answer is slow on the cores and the battery a phone gives a page — so the picker states the download size in every row and leaves the choice to the reader. Start with the smallest model on a phone. ## Supply your own wllama build Install wllama and call `useWllamaSource()` before the first local turn: ```sh pnpm add @wllama/wllama ``` ```ts import { useWllamaSource } from "agentak/pi"; useWllamaSource({ module: () => import("@wllama/wllama/esm/index.js"), wasm: "/wllama/wllama.wasm", }); ``` `module` replaces the JavaScript module loader and `wasm` replaces the jsDelivr `WLLAMA_WASM_URL` that Agentak otherwise passes to wllama. The loaded package must match the API expected by Agentak. `useWllamaSource(undefined)` restores the built-in CDN source. Model weights still use the URLs in `WLLAMA_MODELS`, so a fully offline deployment also needs a proxy or a catalog of its own. ::note wllama starts its Worker from a `blob:` URL that it builds at run time. A page whose CSP has no `blob:` in `worker-src` — every MV3 extension page, and some strict sites — blocks that, and shipping the module is not enough on its own. The side panel writes the worker out as a file at build time and points wllama at it; `extension/wllama/` in the repository is the working example. :: The public local-model helpers include: - `WLLAMA_MODELS`, `WLLAMA_PROVIDER_ID`, and `WLLAMA_MODEL_ID` - `findLocalModel(id)` - `loadWllamaModule()`, `wllamaWasmUrl()`, and `useWllamaSource()` - `WLLAMA_MODULE_URL` and `WLLAMA_WASM_URL` - `wllamaSupported()` ## Chrome Built-in AI Chrome Built-in AI uses the browser's `LanguageModel` Prompt API and its Gemini Nano model. Agentak lists it only when `promptApiSupported()` detects that API. ::warning Chrome Built-in AI has limited agent capabilities. It accepts text only and does not support images, a reasoning stream, or thinking-level control. It calls tools through the prompt, not through an API for them, so a small model gets a call wrong more often than a network provider does. Use wllama or a network provider where tool calls must be reliable. :: Today it can require these Chrome flags and, for a regular site, an origin trial: - `#optimization-guide-on-device-model` - `#prompt-api-for-gemini-nano` An extension page can access the API without a site's origin-trial token. Chrome downloads about 4 GB of model data once. Agentak reports the download as a progress bar in the turn that waits for it — a thinking block carrying `::progress{…}` markers, which the chat draws as one bar. See [the widget guide](/widget#progress-in-a-turn). The wllama models report their own download the same way. Gemini Nano uses a measured 9,216-token window shared by input and output. Agentak reports estimated local usage and no cost. The tools take a share of that window, so a session with many of them leaves the model less room to answer in. ### Tool calls The Prompt API carries no tools. Gemini is trained on a Python language for them, so Agentak speaks it: the tools go into the system turn as function declarations, and a call comes back as text. ```tool_code print(default_api.search_docs(query="page tools", limit=3)) ``` Agentak reads that call out of the answer as it streams, reports it as a normal `toolCall`, and pi's loop runs it under the same approval gate as any other provider. The result goes back to the model as a turn that starts with `Result of `, which is what the system turn told it to expect. The turn ends at the first call: what a small model writes after one is the result it wishes it had. Agentak leaves out a block that holds no call the turn carries — an invented result, or a tool that is not there — and notes it in `diagnostics`. Code blocks the model writes for the reader pass through unchanged. Expect a small model to get a call wrong sometimes. Agentak reads the shapes Nano writes: the `print()` and `default_api.` wrappers are optional, arguments may be named or in order, strings may carry either quote, and a value is taken as the type the tool declares. Chrome helpers include `promptApi()`, `promptApiSupported()`, `ON_DEVICE_MODELS`, and the on-device provider and model IDs. ## Privacy and resource expectations On-device inference keeps prompts away from a model server, but it is not a zero-network feature on first use. Chrome obtains its model through Chrome. The wllama runtime and GGUF weights come from their configured URLs. Tell users about download size before selecting a local model, and provide a bundled loader when your CSP or deployment requires it.