Make chat completion for a given chat messages.
OAI-compatible chat completion options
OAI-compatible chat completion response (only the final result when stream=false) or an async iterator of completion chunks (when stream=true)
Make chat completion for a given chat messages.
OAI-compatible chat completion options
OAI-compatible chat completion response (only the final result when stream=false) or an async iterator of completion chunks (when stream=true)
Make chat completion for a given chat messages.
OAI-compatible chat completion options
OAI-compatible chat completion response (only the final result when stream=false) or an async iterator of completion chunks (when stream=true)
Make (raw) completion for a given text.
OAI-compatible completion options
OAI-compatible completion response (stream=false), void when done (stream=true + onData), or async iterator (stream=true, no onData)
Make (raw) completion for a given text.
OAI-compatible completion options
OAI-compatible completion response (stream=false), void when done (stream=true + onData), or async iterator (stream=true, no onData)
Make (raw) completion for a given text.
OAI-compatible completion options
OAI-compatible completion response (stream=false), void when done (stream=true + onData), or async iterator (stream=true, no onData)
Calculate embedding vector for a given text or multimodal input. By default, BOS and EOS tokens will be added automatically. You can use the "skipBOS" and "skipEOS" option to disable it.
OAI-compatible embedding creation options
OAI-compatible embedding response
Start a realtime audio session. The model must support it, see LoadedContextInfo.has_audio_stream.
It holds one slot until close() is called.
session options
the session
Rerank a list of documents against a query. Requires the model to be loaded with embeddings: true and pooling_type: 'rank'.
Reranking options (query, documents, top_n)
Reranking response with relevance scores sorted highest first
Generate speech from text. The model must have audio output, see LoadedContextInfo.has_audio_output.
speech options
the whole audio (stream=false) or an async iterator of audio chunks (stream=true). A stream starts its request on the first read, so errors come from the iterator
Generate speech from text. The model must have audio output, see LoadedContextInfo.has_audio_output.
speech options
the whole audio (stream=false) or an async iterator of audio chunks (stream=true). A stream starts its request on the first read, so errors come from the iterator
Answer typed questions about a state, using a decision model (TypeSafe-compatible System One API). Image input is not supported yet.
The state and the questions
The answer of each question
Transcribe an audio file. The model must have audio input, see supportInputModality().
OAI-compatible transcription options
OAI-compatible transcription response (stream=false), void when done (stream=true + onData), or async iterator (stream=true, no onData)
Transcribe an audio file. The model must have audio input, see supportInputModality().
OAI-compatible transcription options
OAI-compatible transcription response (stream=false), void when done (stream=true + onData), or async iterator (stream=true, no onData)
Transcribe an audio file. The model must have audio input, see supportInputModality().
OAI-compatible transcription options
OAI-compatible transcription response (stream=false), void when done (stream=true + onData), or async iterator (stream=true, no onData)
Get the jinja chat template comes with the model. It only available if the original model (before converting to gguf) has the template in tokenizer_config.json
NOTE: This can only being used after loadModel is called.
the jinja template. null if there is no template in gguf
Get token ID associated to token used by decoder, to start generating output sequence(only usable for encoder-decoder architecture). In other words, encoder uses normal BOS and decoder uses this token.
NOTE: This can only being used after loadModel is called.
-1 if the model is not loaded.
Get model hyper-parameters and metadata
NOTE: This can only being used after loadModel is called.
ModelMetadata
Load model from a given list of Blob.
You can pass multiple buffers into the function (in case the model contains multiple shards).
Can be either list of Blobs (in case you use local file), or a Model object (in case you use ModelManager)
LoadModelParams
Load model from a given Hugging Face model ID and file path.
Load model from a given URL (or a list of URLs, in case the model is splitted into smaller files)
downloadModel()), then we will use the cached modelRestore the state of a slot from the output of saveSlotState().
The next request that starts with the same prompt reuses it.
The saved state
ID of the slot, between 0 and n_parallel - 1
number of tokens in the restored prompt
Save the state of a slot (the processed prompt), to restore it later with restoreSlotState().
The state can only be restored with the same model and the same wllama version.
ID of the slot, between 0 and n_parallel - 1
the state, you can store it (for example in OPFS)
Set compatibility options for Wllama.
Set to null to disable compatibility, or 'default' to use the default compat resources from CDN.
'safari' by default; If set to 'firefox_safari', the compat mode will also be enabled on Firefox, which will significantly degrade the performance but allow using WebGPU on Firefox.
[FOR DEBUGGING ONLY] Run ggml backend ops tests without loading any model.
Initializes the wasm runtime, executes test-backend-ops with the given args, then shuts down.
For more info, please refer to guides/debug.md
Arguments forwarded to test-backend-ops (e.g. ["-o", "ADD"])
retcode (0 = all tests passed) and success flag
Staticget
get debug info