@wllama/wllama

wllama - Wasm binding for llama.cpp

WebAssembly binding for llama.cpp

👉 Try the demo app

👉 See the blog post introducing WebGPU support in llama.cpp and wllama

📄 Documentation

For changelog, please visit releases page

Important

đŸ”ĨđŸ”Ĩ V3 is out, with WebGPU, multimodal and tool calling support. Read the V3 release guide

For compatibility issues, please refer to @wllama/wllama-compat

  • 🔌 OpenAI-compatible API (fully-typed built-in)
  • 🚀 WebGPU support
  • đŸ”Ĩ Multimodal support (image and audio file input)
  • Speech support: text to speech, transcription and realtime audio
  • đŸ”Ĩ Tool calling support
  • Can run inference directly on browser (using WebAssembly SIMD), no backend or GPU is needed!
  • No runtime dependency (see package.json)
  • Load a GGUF file as-is, even bigger than 4GB, no need to split it (models already split with llama-gguf-split work too)
  • Auto switch between single-thread and multi-thread build based on browser support
  • Inference is done inside a worker, does not block UI render
  • Pre-built npm package @wllama/wllama

Limitations:

  • To enable multi-thread, you must add Cross-Origin-Embedder-Policy and Cross-Origin-Opener-Policy headers. See this discussion for more details.
  • One wasm instance has 4GB of memory. If the part of the model that is not on WebGPU is bigger, it is split across several wasm instances in workers (rpc_workers, automatic by default), which needs multi-thread, and JSPI or the compat build (on Firefox, see setCompat).

Demo:

Install it:

npm i @wllama/wllama
Install from this git repo

Wasm binaries do not come pre-built with this repo. You need docker installed on your machine to build them:

# recommend to clone as git submodule
git submodule add https://github.com/ngxson/wllama.git wllama
git submodule update --init --recursive

# run the build
cd wllama
npm ci
npm run build:wasm && npm run build

Then, import the module:

import { Wllama } from '@wllama/wllama';
let wllamaInstance = new Wllama(WLLAMA_CONFIG_PATHS, ...);
// (the rest is the same with earlier example)

For complete code example, see examples/main/src/utils/wllama.context.tsx

NOTE: this example only covers completions usage. For embeddings, please see examples/embeddings/index.html. For audio, see Speech and realtime audio

These APIs need a model with audio support and its mmproj file, for example LFM3-Omni. Use getLoadedContextInfo() to see what the model can do (has_audio_input, has_audio_output, has_audio_stream, has_live_asr, audio_voices).

// text to speech, returns a WAV file. Use stream: true to get chunks as they are generated
const { audio, sample_rate } = await wllama.createSpeech({
input: 'Hello world',
voice: 'female',
});

// transcribe an audio file
const { text } = await wllama.createTranscription({
file: await audioFile.arrayBuffer(),
language: 'en',
});

A realtime session uses the same events as the OpenAI realtime API and the /v1/realtime route of llama-server. It holds one slot until close() is called, and inference only runs while events() is being read.

const session = await wllama.createRealtimeSession();
await session.send({ type: 'session.update', session: { type: 'transcription' } });

// mono float samples at getLoadedContextInfo().audio_input_rate
await session.appendAudio(samples);
await session.send({ type: 'input_audio_buffer.commit' });

for await (const event of session.events()) {
if (event.type === 'conversation.item.input_audio_transcription.delta') {
console.log(event.delta);
} else if (event.type === 'conversation.item.input_audio_transcription.completed') {
break;
}
}
await session.close();

There are 3 kinds of session:

  • Default: a conversation. Send response.create after the audio, the answer comes as response.output_text.delta and response.output_audio.delta events. Use decodeRealtimeAudio() to read the audio
  • session.update with type: 'transcription': each committed audio buffer is transcribed on its own
  • createRealtimeSession({ intent: 'transcription' }): live ASR, the text comes out while the audio goes in. Needs has_live_asr, LFM3-Omni does not have it

WebGPU support is introduced via PR #215.

Upon updating to V3.1, WebGPU will be enabled automatically. By default, all layers will be offloaded to GPU. If the model is too big to fit into VRAM, you can manually adjust the number of layers via the n_gpu_layers parameter of LoadModelParams. Example:

// (optionally) will allow running WebGPU on Firefox via compat mode; performance will be significantly degraded
wllama.setCompat('default', 'firefox_safari');

await wllama.loadModel(files, {
n_gpu_layers: 4, // meaning 4 layers are offloaded to GPU; set to 0 to disable GPU inference
});
  • It is recommended to use the GGUF file as-is, there is no need to split it anymore. A file of 4GB or more is split in memory by wllama, and a model that does not fit in one wasm instance is spread across workers (see Limitations above). See the "Split model" section below if your model is already split.
  • It is recommended to use quantized Q4, Q5 or Q6 for balance among performance, file size and quality. Using IQ (with imatrix) is not recommended, may result in slow inference and low quality.

For complete code, see examples/basic/index.html

import { Wllama } from './esm/index.js';

(async () => {
const CONFIG_PATHS = {
default: './esm/wasm/wllama.wasm',
};
// Automatically switch between single-thread and multi-thread version based on browser support
// If you want to enforce single-thread, add { "n_threads": 1 } to LoadModelConfig
const wllama = new Wllama(CONFIG_PATHS);
// Define a function for tracking the model download progress
const progressCallback = ({ loaded, total }) => {
// Calculate the progress as a percentage
const progressPercentage = Math.round((loaded / total) * 100);
// Log the progress in a user-friendly format
console.log(`Downloading... ${progressPercentage}%`);
};
// Load GGUF from Hugging Face hub
// (alternatively, you can use loadModelFromUrl if the model is not from HF hub)
await wllama.loadModelFromHF(
{ repo: 'ggml-org/models', file: 'tinyllamas/stories260K.gguf' },
{ progressCallback }
);
const response = await wllama.createChatCompletion({
messages: [{ role: 'user', content: elemInput.value }],
max_tokens: 50,
temperature: 0.5,
top_k: 40,
top_p: 0.9,
});
console.log(response.choices[0].message.content);
})();

Alternatively, you can use the *.wasm files from CDN:

import WasmFromCDN from '@wllama/wllama/esm/wasm-from-cdn.js';
const wllama = new Wllama(WasmFromCDN);
// NOTE: this is not recommended, only use when you can't embed wasm files in your project

Splitting the model is not needed anymore: wllama loads a single GGUF file, even bigger than 4GB, so we recommend using the file as-is. Models that are already split with llama-gguf-split (files ending with -00001-of-00003.gguf, -00002-of-00003.gguf, and so on) are still supported.

To load a split model, pass to loadModelFromUrl or loadModelFromHF the URL of the first file and it will automatically load all the chunks:

const wllama = new Wllama(CONFIG_PATHS, {
parallelDownloads: 5, // optional: maximum files to download in parallel (default: 3)
});
await wllama.loadModelFromHF({
repo: 'ngxson/tinyllama_split_test',
file: 'stories15M-q8_0-00001-of-00003.gguf',
});

When initializing Wllama, you can pass a custom logger to Wllama.

Example 1: Suppress debug message

import { Wllama, LoggerWithoutDebug } from '@wllama/wllama';

const wllama = new Wllama(pathConfig, {
// LoggerWithoutDebug is predefined inside wllama
logger: LoggerWithoutDebug,
});

Example 2: Add emoji prefix to log messages

const wllama = new Wllama(pathConfig, {
logger: {
debug: (...args) => console.debug('🔧', ...args),
log: (...args) => console.log('â„šī¸', ...args),
warn: (...args) => console.warn('âš ī¸', ...args),
error: (...args) => console.error('â˜ ī¸', ...args),
},
});

This repository already come with pre-built binary from llama.cpp source code. However, in some cases you may want to compile it yourself:

  • You don't trust the pre-built one.
  • You want to try out latest - bleeding-edge changes from upstream llama.cpp source code.

You can use the commands below to compile it yourself:

# /!\ IMPORTANT: Require having docker compose installed

# Clone the repository with submodule
git clone --recurse-submodules https://github.com/ngxson/wllama.git
cd wllama

# Optionally, you can run this command to update llama.cpp to latest upstream version (bleeding-edge, use with your own risk!)
# git submodule update --remote --merge

# Install the required modules
npm i

# Firstly, build llama.cpp into wasm
npm run build:wasm
# Then, build ES module
npm run build
  • Add support for LoRA adapter
  • Support multi-sequences: knowing the resource limitation when using WASM, I don't think having multi-sequences is a good idea

Wllama was created and is maintained by Xuan-Son Nguyen. The WebGPU backend for llama.cpp is maintained by Reese Levine. We thank all other contributors to both wllama and llama.cpp, whose work made this project possible.