
WebAssembly binding for llama.cpp
đ Try the demo app
đ See the blog post introducing WebGPU support in llama.cpp and wllama
đ Documentation
For changelog, please visit releases page
đĨđĨ V3 is out, with WebGPU, multimodal and tool calling support. Read the V3 release guide
For compatibility issues, please refer to @wllama/wllama-compat

llama-gguf-split work too)Limitations:
Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy headers. See this discussion for more details.rpc_workers, automatic by default), which needs multi-thread, and JSPI or the compat build (on Firefox, see setCompat).Demo:
Install it:
npm i @wllama/wllama
Wasm binaries do not come pre-built with this repo. You need docker installed on your machine to build them:
# recommend to clone as git submodule
git submodule add https://github.com/ngxson/wllama.git wllama
git submodule update --init --recursive
# run the build
cd wllama
npm ci
npm run build:wasm && npm run build
Then, import the module:
import { Wllama } from '@wllama/wllama';
let wllamaInstance = new Wllama(WLLAMA_CONFIG_PATHS, ...);
// (the rest is the same with earlier example)
For complete code example, see examples/main/src/utils/wllama.context.tsx
NOTE: this example only covers completions usage. For embeddings, please see examples/embeddings/index.html. For audio, see Speech and realtime audio
These APIs need a model with audio support and its mmproj file, for example LFM3-Omni. Use getLoadedContextInfo() to see what the model can do (has_audio_input, has_audio_output, has_audio_stream, has_live_asr, audio_voices).
// text to speech, returns a WAV file. Use stream: true to get chunks as they are generated
const { audio, sample_rate } = await wllama.createSpeech({
input: 'Hello world',
voice: 'female',
});
// transcribe an audio file
const { text } = await wllama.createTranscription({
file: await audioFile.arrayBuffer(),
language: 'en',
});
A realtime session uses the same events as the OpenAI realtime API and the /v1/realtime route of llama-server. It holds one slot until close() is called, and inference only runs while events() is being read.
const session = await wllama.createRealtimeSession();
await session.send({ type: 'session.update', session: { type: 'transcription' } });
// mono float samples at getLoadedContextInfo().audio_input_rate
await session.appendAudio(samples);
await session.send({ type: 'input_audio_buffer.commit' });
for await (const event of session.events()) {
if (event.type === 'conversation.item.input_audio_transcription.delta') {
console.log(event.delta);
} else if (event.type === 'conversation.item.input_audio_transcription.completed') {
break;
}
}
await session.close();
There are 3 kinds of session:
response.create after the audio, the answer comes as response.output_text.delta and response.output_audio.delta events. Use decodeRealtimeAudio() to read the audiosession.update with type: 'transcription': each committed audio buffer is transcribed on its owncreateRealtimeSession({ intent: 'transcription' }): live ASR, the text comes out while the audio goes in. Needs has_live_asr, LFM3-Omni does not have itWebGPU support is introduced via PR #215.
Upon updating to V3.1, WebGPU will be enabled automatically. By default, all layers will be offloaded to GPU. If the model is too big to fit into VRAM, you can manually adjust the number of layers via the n_gpu_layers parameter of LoadModelParams. Example:
// (optionally) will allow running WebGPU on Firefox via compat mode; performance will be significantly degraded
wllama.setCompat('default', 'firefox_safari');
await wllama.loadModel(files, {
n_gpu_layers: 4, // meaning 4 layers are offloaded to GPU; set to 0 to disable GPU inference
});
For complete code, see examples/basic/index.html
import { Wllama } from './esm/index.js';
(async () => {
const CONFIG_PATHS = {
default: './esm/wasm/wllama.wasm',
};
// Automatically switch between single-thread and multi-thread version based on browser support
// If you want to enforce single-thread, add { "n_threads": 1 } to LoadModelConfig
const wllama = new Wllama(CONFIG_PATHS);
// Define a function for tracking the model download progress
const progressCallback = ({ loaded, total }) => {
// Calculate the progress as a percentage
const progressPercentage = Math.round((loaded / total) * 100);
// Log the progress in a user-friendly format
console.log(`Downloading... ${progressPercentage}%`);
};
// Load GGUF from Hugging Face hub
// (alternatively, you can use loadModelFromUrl if the model is not from HF hub)
await wllama.loadModelFromHF(
{ repo: 'ggml-org/models', file: 'tinyllamas/stories260K.gguf' },
{ progressCallback }
);
const response = await wllama.createChatCompletion({
messages: [{ role: 'user', content: elemInput.value }],
max_tokens: 50,
temperature: 0.5,
top_k: 40,
top_p: 0.9,
});
console.log(response.choices[0].message.content);
})();
Alternatively, you can use the *.wasm files from CDN:
import WasmFromCDN from '@wllama/wllama/esm/wasm-from-cdn.js';
const wllama = new Wllama(WasmFromCDN);
// NOTE: this is not recommended, only use when you can't embed wasm files in your project
Splitting the model is not needed anymore: wllama loads a single GGUF file, even bigger than 4GB, so we recommend using the file as-is. Models that are already split with llama-gguf-split (files ending with -00001-of-00003.gguf, -00002-of-00003.gguf, and so on) are still supported.
To load a split model, pass to loadModelFromUrl or loadModelFromHF the URL of the first file and it will automatically load all the chunks:
const wllama = new Wllama(CONFIG_PATHS, {
parallelDownloads: 5, // optional: maximum files to download in parallel (default: 3)
});
await wllama.loadModelFromHF({
repo: 'ngxson/tinyllama_split_test',
file: 'stories15M-q8_0-00001-of-00003.gguf',
});
When initializing Wllama, you can pass a custom logger to Wllama.
Example 1: Suppress debug message
import { Wllama, LoggerWithoutDebug } from '@wllama/wllama';
const wllama = new Wllama(pathConfig, {
// LoggerWithoutDebug is predefined inside wllama
logger: LoggerWithoutDebug,
});
Example 2: Add emoji prefix to log messages
const wllama = new Wllama(pathConfig, {
logger: {
debug: (...args) => console.debug('đ§', ...args),
log: (...args) => console.log('âšī¸', ...args),
warn: (...args) => console.warn('â ī¸', ...args),
error: (...args) => console.error('â ī¸', ...args),
},
});
This repository already come with pre-built binary from llama.cpp source code. However, in some cases you may want to compile it yourself:
You can use the commands below to compile it yourself:
# /!\ IMPORTANT: Require having docker compose installed
# Clone the repository with submodule
git clone --recurse-submodules https://github.com/ngxson/wllama.git
cd wllama
# Optionally, you can run this command to update llama.cpp to latest upstream version (bleeding-edge, use with your own risk!)
# git submodule update --remote --merge
# Install the required modules
npm i
# Firstly, build llama.cpp into wasm
npm run build:wasm
# Then, build ES module
npm run build
Wllama was created and is maintained by Xuan-Son Nguyen. The WebGPU backend for llama.cpp is maintained by Reese Levine. We thank all other contributors to both wllama and llama.cpp, whose work made this project possible.