On-device models

Run Gemini Nano or small GGUF models without sending prompts to a model server.

Agentak has two model APIs that run inference on the visitor's device. No prompt is sent to a model provider, no API key is needed, and usage cost is zero. The browser still needs to download runtime files or model weights before the first answer.

#On Device with wllama

On Device (wllama) runs llama.cpp compiled to WebAssembly in a Worker in the current tab. The default loader imports wllama from jsDelivr, loads its wasm, and downloads public GGUF weights from Hugging Face. The browser caches the model file, and only one model is held in memory at a time.

The built-in catalog is intentionally small:

Model	Download	Loaded context	KV per token	Reasoning	Tools
LFM2.5 350M	219 MB	32,768	12 KiB	No	Yes
MiniCPM5 1B	688 MB	16,384	24 KiB	Yes	Yes
LFM2.5 1.2B	696 MB	32,768	12 KiB	No	Yes
LFM2.5 2.6B	1.6 GB	32,768	16 KiB	No	Yes
Qwen3.5 2B	1.6 GB	32,768	12 KiB	Yes	Yes
Qwen3.5 4B	1.9 GB	16,384	32 KiB	Yes	Yes
Granite 4.1 3B	2.0 GB	8,192	80 KiB	No	Yes

The LFM2.5 rows use Liquid's QAD Q4_0 checkpoints. These are distilled while quantized rather than quantized after training, so they hold about 97% of the full-weight score at the size of a Q4. The other rows use K-quants, and the quant falls as the model grows: the Qwen3.5 4B fits only as a dynamic Q2_K_XL.

For tool calling, the LFM2.5 rows and Granite 4.1 3B are the strongest for their size. Ability falls off sharply below about 1B: Qwen reports BFCL-V4 of 50.3 for Qwen3.5 4B, 43.6 for the 2B and 25.3 for the 0.8B.

All seven accept text. A local model can stream text, reasoning, and tool calls when its chat template supports them. For wllama, Agentak maps off to thinking disabled and every non-off level to thinking enabled; the local template receives an on/off choice rather than the full scale.

The weights and KV cache share a 4 GiB WebAssembly heap, so the loaded window is what the heap has room for after the weights. The cost of a token is not the same for every model. A hybrid keeps attention in only a few of its layers and a cheap state in the rest, so it pays 12 to 32 KiB a token. Granite 4.1 3B has attention in all forty of its layers and pays 80 KiB, which is why it carries the shortest window here despite being trained for 128K. Each window above is the largest that keeps the weights and the cache inside about 2.6 GiB together. Performance depends on the device; a laptop may answer at a useful speed while a phone may not.

#Availability

wllamaSupported() requires:

WebAssembly
Worker
a document that may load what the runtime is made of

The default loader imports a remote script, which an MV3 extension page cannot do. A host that supplies its own source through useWllamaSource() answers that question for its own document, and the local row is then listed there too. The Agentak side panel does exactly this and lists wllama; see Chrome extension.

A phone lists the row on the same terms. The cost is higher there — the download is hundreds of MB over a connection that is often metered, a mobile browser reclaims the wasm heap when the tab goes to the background, and the answer is slow on the cores and the battery a phone gives a page — so the picker states the download size in every row and leaves the choice to the reader. Start with the smallest model on a phone.

#Supply your own wllama build

Install wllama and call useWllamaSource() before the first local turn:

pnpm add @wllama/wllama

import { useWllamaSource } from "agentak/pi";

useWllamaSource({
  module: () => import("@wllama/wllama/esm/index.js"),
  wasm: "/wllama/wllama.wasm",
});

module replaces the JavaScript module loader and wasm replaces the jsDelivr WLLAMA_WASM_URL that Agentak otherwise passes to wllama. The loaded package must match the API expected by Agentak. useWllamaSource(undefined) restores the built-in CDN source.

Model weights still use the URLs in WLLAMA_MODELS, so a fully offline deployment also needs a proxy or a catalog of its own.

Note

wllama starts its Worker from a blob: URL that it builds at run time. A page whose CSP has no blob: in worker-src — every MV3 extension page, and some strict sites — blocks that, and shipping the module is not enough on its own. The side panel writes the worker out as a file at build time and points wllama at it; extension/wllama/ in the repository is the working example.

The public local-model helpers include:

WLLAMA_MODELS, WLLAMA_PROVIDER_ID, and WLLAMA_MODEL_ID
findLocalModel(id)
loadWllamaModule(), wllamaWasmUrl(), and useWllamaSource()
WLLAMA_MODULE_URL and WLLAMA_WASM_URL
wllamaSupported()

#Chrome Built-in AI

Chrome Built-in AI uses the browser's LanguageModel Prompt API and its Gemini Nano model. Agentak lists it only when promptApiSupported() detects that API.

Warning

Chrome Built-in AI has limited agent capabilities. It accepts text only and does not support images, a reasoning stream, or thinking-level control. It calls tools through the prompt, not through an API for them, so a small model gets a call wrong more often than a network provider does. Use wllama or a network provider where tool calls must be reliable.

Today it can require these Chrome flags and, for a regular site, an origin trial:

#optimization-guide-on-device-model
#prompt-api-for-gemini-nano

An extension page can access the API without a site's origin-trial token. Chrome downloads about 4 GB of model data once. Agentak reports the download as a progress bar in the turn that waits for it — a thinking block carrying ::progress{…} markers, which the chat draws as one bar. See the widget guide. The wllama models report their own download the same way.

Gemini Nano uses a measured 9,216-token window shared by input and output. Agentak reports estimated local usage and no cost. The tools take a share of that window, so a session with many of them leaves the model less room to answer in.

#Tool calls

The Prompt API carries no tools. Gemini is trained on a Python language for them, so Agentak speaks it: the tools go into the system turn as function declarations, and a call comes back as text.

print(default_api.search_docs(query="page tools", limit=3))

Agentak reads that call out of the answer as it streams, reports it as a normal toolCall, and pi's loop runs it under the same approval gate as any other provider. The result goes back to the model as a turn that starts with Result of <tool>, which is what the system turn told it to expect.

The turn ends at the first call: what a small model writes after one is the result it wishes it had. Agentak leaves out a block that holds no call the turn carries — an invented result, or a tool that is not there — and notes it in diagnostics. Code blocks the model writes for the reader pass through unchanged.

Expect a small model to get a call wrong sometimes. Agentak reads the shapes Nano writes: the print() and default_api. wrappers are optional, arguments may be named or in order, strings may carry either quote, and a value is taken as the type the tool declares.

Chrome helpers include promptApi(), promptApiSupported(), ON_DEVICE_MODELS, and the on-device provider and model IDs.

#Privacy and resource expectations

On-device inference keeps prompts away from a model server, but it is not a zero-network feature on first use. Chrome obtains its model through Chrome. The wllama runtime and GGUF weights come from their configured URLs. Tell users about download size before selecting a local model, and provide a bundled loader when your CSP or deployment requires it.

Edit this page

Previous Providers and models Next Tools and approvals

On this page