←→ navigate · O overview · G go to · S notes · F fullscreen
A private AI chatbot that lives in your browser
Local Whisper
A site-aware chat widget that runs entirely in your visitor's browser — no backend, no API key, no per-token bill — and it answers from your page's own content.
100% local-first Site-aware RAG Pluggable scraper Works offline
Pre-built bots know nothing about your content. Ask "what's your refund policy?" on a docs site and you get "I don't have information on that." Useless.
$ Leaky & costly
API-powered bots send every visitor question to a third-party server. You pay per token, babysit an API key, and your visitors' questions leave the page.
The result
Small sites are stuck — too small to build a backend chatbot, too privacy-conscious to ship questions to a cloud API. So they ship nothing.
03 · WHAT IT IS
One <script> tag. A chatbot that knows your page.
Local Whisper is a floating chat widget. When a visitor asks a question, it reads your page, finds the relevant section, and answers from it — with a link that jumps straight to the answer. The whole thing runs in their browser.
1
script tag
0
backend
0
API key
100%
client-side
No config required. Paste the tag, save, refresh — a chat button appears that can already answer from whatever's on the page.
04 · WHY FREE + LOCAL, NOT A CHAT API
A cloud chat API is a backend you rent.
"Serverless" LLM APIs feel backend-free — but to keep your API key secret you must run a server, and that server is the backend. Then every question costs money and leaks data.
Chat API (cloud)
Local server
In-browser · Local Whisper
Cost
per-token bill
server + compute
free
Privacy
data leaves device
your server sees it
never leaves device
Infra / upkeep
key-hiding proxy
run & patch a box
none — static host
Knows your content
needs a RAG backend
if you build it
built-in, client-side
Works offline
no
no
yes, after cache
05 · WHY IN THE BROWSER?
Why not just run a model on a $5 server?
"Local" can mean a server you run, or the browser itself. For a site owner with a static page, the browser wins on every axis that matters.
Static hosts can't run backends.GitHub Pages, Netlify, a CDN portfolio — no server process exists. A chat server means leaving static hosting behind.
Infra is real work.Provision, scale, keep a box alive, secure and patch it. That's a side-project of its own.
The visitor's device is free compute.Modern phones and laptops run a small LLM fine — why pay for a server when the user already has a CPU?
Privacy scales for free."Your data never leaves your device" is automatically true when there's no server for it to leave to.
The shift
Move the work from your server (which costs you money and sees user data) to the visitor's device (which is free and already trusted with the data). The site owner ships a script tag, not a server.
06 · ARCHITECTURE
Two contexts, one postMessage bridge.
The widget is a sandboxed iframe — it can't read a cross-origin parent page directly. So a tiny script (embed.js) runs in the host page instead, where it's always same-origin to itself, scrapes the content, and hands it over.
🌐 Your page (host)
embed.js runs here — same-origin to itself
Scrapes visible content into sections
Scrolls to the section on link click
sections→postMessage
scroll-to←link click
🤖 Chat widget (iframe)
React app + Wllama (WebAssembly)
chunk → embed → retrieve → ground
Answer + "Related sections" links
This split is why it works on any site: the scraper lives where the page is readable, the model lives where it can run sandboxed. Neither privilege is violated.
07 · HOW THE CHAT WORKS
A real LLM, loaded as WebAssembly.
GGUF model → Wllama → WebAssembly.The model file is compiled to WASM and runs on the visitor's CPU. No server round-trip per message.
Tiny by design.Default is Gemma 3 270M (~278MB); presets run 270M → 1B — small enough to download once and keep.
Cached = offline.After the first load the model is in the browser cache; the widget works with no network.
You can swap it.Upload a different .gguf (up to 2GB) — bigger model, smarter answers, more RAM.
# the whole inference path, no server:model.gguf→ Wllama (WASM)→ visitor's CPU
→ answer in the widget
08 · THE ENGINE — WHY WLLAMA
Why Wllama — and how it actually runs.
It's llama.cpp, compiled to WebAssembly.Wllama wraps the same ggml / llama.cpp core that runs Llama-class models natively — real inference, not a toy demo.
No GPU, no server.Runs on the visitor's CPU, inside their tab. The browser is the runtime.
Multi-threaded, off the main thread.A Web Worker + SharedArrayBuffer use several cores — inference doesn't freeze the page.
Streams both ways.Model chunks stream in; tokens stream out. The first answer starts before the whole model is cached.
GGUF-native.Loads standard Hugging Face GGUF files — swap models by swapping a file, no recompile.
// how a model reaches the userGGUF model file
↓llama.cpp / ggml (C++ core)↓ Emscripten →
WebAssembly module
↓ runs in a Web Worker
visitor CPU (multi-threaded)↓streamed answer in the widget
Wllama is the bridge that makes "a real LLM in a browser tab" practical — the same inference code people run on servers, retargeted to WASM.
09 · DEEP DIVE — THE LOCAL-LLM LANDSCAPE
Wllama vs Ollama vs the rest.
Several runtimes can run an LLM on your own machine. They split on one question — does it need installing? — and that decides whether it can power a chatbot embedded on someone else's static website.
Runtime
How it runs
Install on visitor's device?
Powers an embedded site widget?
Wllama (this project)
llama.cpp → WebAssembly, in a Web Worker
no — loads with the page
✓ the pick
WebLLM
MLC-compiled models → WebGPU / WASM, in-browser
no
alt in-browser engine
transformers.js
ONNX Runtime in-browser (WASM / WebGPU)
no
embeddings more than chat
Ollama
native llama.cpp app + local REST server (localhost:11434)
yes — app + running process
✗ visitor must install it
LM Studio
desktop GUI + OpenAI-compatible local server
yes — app + running process
✗ visitor must install it
The crux: Ollama and LM Studio are excellent — for running a model on your machine (bigger models, GPU, very fast). They can't power a widget on a stranger's static site, because every visitor would need them installed and running. Wllama is the one that runs on the visitor's device with zero install — which is the whole point of Local Whisper.
10 · THE GAP
A general model doesn't know your content.
Ask "what's your refund policy?" and a generic model will happily invent one. It's smart about language, not about your site. So you have to teach it — and there are two ways.
Option A · Train
Bake your content into the model's weights (fine-tuning). The model memorizes your site.
Option B · Retrieve
Hand the model the right snippet at question time (RAG). The model reads your site on demand.
Next two slides: why Option A doesn't work in a browser, and why Option B is the whole point of this project.
11 · WHY NOT TRAIN A MODEL?
Teaching a model your content by training doesn't fit the browser.
Training is heavy.Fine-tuning needs GPU clusters and hours-to-days — not a visitor's laptop on a Tuesday.
It's per-site.You'd retrain for every site, and again for every content update. A blog-post edit would mean a retrain.
Browsers are RAM/CPU bounded.Even running a tiny model is near the ceiling; training one is off the table.
It breaks the promise.Real training wants your data centralized — exactly the server + data-leaving-device setup Local Whisper exists to avoid.
The models are tiny by design.A 270M model can't absorb deep site knowledge through weights without forgetting how to talk.
The reframe
We don't need the model to remember your site — we need it to read your site on demand. That's a different (and far cheaper) problem.
RAG: hand the model the right paragraph, then let it talk.
Retrieval-Augmented Generation. Instead of baking knowledge into weights, we keep your page as text, find the relevant chunk for each question, and paste it into the model's context. The model stays general; your content stays fresh.
0
training
↻
re-scrape to update
cite
grounded answers
🔗
linkable
Why it wins here: works with a tiny model, updates the instant your page changes, grounds every answer in real text (less invention), and stays 100% in the browser. It's the only option that fits all the constraints at once.
13 · HOW RAG WORKS
Five steps, all in the visitor's browser.
📄
Scrape
read the page into sections
→
✂️
Chunk
split into passages
→
🔢
Embed
bge-small → vectors
→
🔍
Retrieve
top-k matches
→
💬
Ground
answer + links
The embedder (bge-small, ~35MB) is a separate model from the chat model — it loads lazily on the first grounded question, then its vectors are cached in the visitor's IndexedDB. Repeat questions are instant.
14 · THE TECHNIQUES, NAMED
Six techniques do the heavy lifting.
Each is a deliberate choice. Named here so you recognize them anywhere else you meet them — and can tell they're standard building blocks, not magic.
① WebAssembly inference
The LLM core (llama.cpp) compiled to WASM, runs on the CPU. No install, no GPU, no plugin.
② Sandboxed iframe + postMessage
The widget is isolated; the host script bridges content cross-origin. The security boundary that makes "any site" work.
③ Vector embeddings + retrieval
Text → vectors (bge-small); find the nearest chunks by similarity. How "relevant" is measured.
④ IndexedDB vector cache
Embeddings persist across visits; repeat questions skip the work entirely.
⑤ COEP / COOP isolation
Cross-origin headers that unlock SharedArrayBuffer (multi-threaded WASM). The one fiddly requirement.
⑥ Lazy loading
The ~35MB embedder loads on the first grounded question, not on page load. Fast first paint.
15 · PLUGGABLE CHATBOT
Bring your own scraper — or use the built-in one.
Default scraper reads visible text grouped under each heading — covers most sites, zero config.
Override with getSections for a CMS, a JSON-LD block, an API, or a specific content region.
Custom voice or brand? Set PRIVATE_CHAT_CONFIG.persona — one config line, no training.
Runs in your page's context — so it can fetch your own APIs and await async data.
Degrades gracefully — throw or return empty, the widget falls back to plain chat. No crash.
// your page, before the embed scriptwindow.PRIVATE_CHAT_CONFIG = {
persona: "Friendly, concise, calls me Acme",
getSections: async (doc) => [
{ anchor: "pricing",
title: "Pricing",
text: "…the section text…" }
]
};
16 · BEYOND ONE PAGE
One widget, the whole site.
Static cross-page index.Drop a site-index.json next to your page (built by the bundled crawler — Node, or Python via Scrapling) and the widget answers across pages, not just the current one.
SPA-aware.On client-side navigation (React / Vue routers) it re-scrapes automatically — no stale content after a route change.
Links go to the right place.A hit on another page links there with an anchor, then scrolls the visitor to the exact section.
Default: just the current page. Add a site-index.json only when you want cross-page answers — everything else works with zero configuration.
17 · THREE RUNNING MODES
Same widget, any deployment.
① Cross-origin embed
Widget on your domain. embed.js scrapes the host and bridges via postMessage. The common case.
② Same-origin embed
Widget served from the same origin (e.g. the portfolio). Iframe-side scrape is the fallback — same effect, simpler path.
③ Standalone
The app at its own URL, no embed. Plain general chat, no RAG — your private, offline ChatGPT-in-a-tab.
All three are exercised by the test suite. The RAG path only activates in the two embed modes — standalone is a general assistant.
18 · WHERE IT SHINES
Built for the small static site.
If your site is a portfolio, a docs page, or a landing page on GitHub Pages or Netlify — you can't run a backend chatbot and you don't want an API bill. Local Whisper is the chatbot that fits that world.
✓ One script tag, zero config
Drop it in, save, refresh — a chatbot that already knows your page.
✓ No backend to run
Static hosting stays static. Nothing to provision, scale, or keep awake.
✓ No bill
The visitor's CPU does the work — not your wallet. Free at any traffic level.
✓ Private by default
Questions never leave the device. Ideal for portfolios, docs, small-business sites — anywhere trust matters.
19 · WIDER POSSIBILITIES
Where else this unlocks something.
📘 Internal wikis & docs
Searchable by natural-language question, fully offline-capable, on-prem. No data egress.
🎓 Education
A tutor that reads the lesson and answers from it — runs on a student's offline laptop.
📡 Kiosks & offline
Museums, field tools, confined networks. Once cached, no network needed at all.
🏥 Regulated (health / finance)
"Data never leaves the device" becomes the feature, not a constraint compliance fights.
🛠️ Local dev tools
A chat that knows your README and docs without phoning home — pair it with any local project.
🌐 Personal sites
Resumes, blogs, link-in-bio pages — give visitors a way to ask instead of scroll.
The pattern: anywhere "private," "free," "offline," or "static-hosted" is the constraint — in-browser RAG turns it from a blocker into a feature.
20 · HONEST BOUNDARIES
What it can't do (yet).
Said out loud so the rest of this deck reads as honest, not salesy.
Tiny models hallucinate.270M params is sharp for its size, but it'll make things up. RAG grounds it; it doesn't guarantee perfection.
Retrieval is brute-force.It scans all chunks — very large sites will feel it in latency and memory.
Needs COEP / COOP headers.Cross-origin isolation is a WebAssembly requirement; some hosts make this fiddly.
First-question latency.The ~35MB embedder loads on the first grounded question (then it's cached).
No deep multi-step reasoning.It reads and answers. It doesn't plan long agentic chains or run tools.
Browser memory ceiling.A bigger model is smarter — until the tab runs out of RAM. There's a real trade-off curve.
None of these are fatal — they're the honest cost of "100% in the browser, zero backend." The project picks that trade on purpose.
21 · RECAP
Local Whisper, in one line.
A chat widget that runs a real LLM in your visitor's browser, reads your page, and answers from it — with links. No backend, no API key, no bill, no data leaving the device.
Why not a chat API: it's a rented backend — key to hide, per-token cost, data leaks.
Why in the browser: static hosts can't run servers; the visitor's CPU is free, private compute.
Why Wllama (not Ollama): llama.cpp compiled to WASM — the only local runtime that runs on a visitor's device with zero install.
Why not training: heavy, per-site, breaks the no-backend promise — persona + RAG already cover it.
Why RAG: grounded, fresh on re-scrape, linkable — and fits a tiny model, in-browser.
Pluggable: default scraper works; getSections + persona give full control.