Written in Rust · Zero dependencies · Self-hosted

LLM inference at bare metal.

SeraphByte bypasses every Python abstraction between your hardware and your model. Custom tokenizer, quantized weight loading, WebSocket streaming - all in Rust.

Explore the engine Request access

tokens / sec

GTX 1050 Ti · q4_K_M

first token latency

0ms

avg over 50 runs

peak VRAM

0MB

7B model · 4-bit quant

What's inside

Every layer, hand-built.

No framework shims, no vendored wheels. If it touches the model, it was written from scratch.

Custom Tokenizer

Binary BPE tokenizer built from scratch. Parses model vocab files directly with no HuggingFace dependency.

Quantized Weight Loader

Memory-maps GGUF files and dequantizes on-the-fly. Keeps VRAM footprint minimal for constrained hardware.

WebSocket Streaming

Token-by-token streaming over WebSocket with sub-millisecond network overhead. Seraphine runs on it live.

Fine-tune Pipeline

End-to-end fine-tuning support. Seraphine's tsundere persona is baked into weights, not runtime prompts.

Self-hosted by Default

Zero cloud. Runs entirely on-prem with no outbound telemetry. Your model, your hardware, your data.

Rust All The Way Down

No FFI, no WASM boundaries, no scripting layer. Pure Rust from inference loop to WebSocket frame serialization.

Architecture

The stack, top to bottom.

Every layer is owned. No black boxes, no vendor SDKs hiding the hot path.

Discord / WebSocket client ws

SeraphByte WS server rust

Inference loop rust

Attention + KV cache rust

Quantized weight loader (GGUF) rust

BPE tokenizer rust

GPU / CPU (consumer hw) hw

            
            
            
            seraphbyte/src/main.rs
          
// spin up the inference server
let engine = Engine::load(Config {
    model: "./models/seraphine-q4.gguf",
    quant: Quant::Q4_K_M,
    ctx_len: 4096,
    threads: 8,
})?;

let server = WsServer::bind("0.0.0.0:9001")?;

for conn in server.incoming() {
    let eng = engine.clone();
    tokio::spawn(async move {
        eng.stream_to(conn).await
    });
}

✓ Streams tokens as they're generated

✓ Multiple concurrent connections

✓ KV cache shared across sessions

✓ No Python runtime in the process tree

Contact

Let's make something worth keeping.

SeraphByte is in active development. Reach out if you want early access, want to collaborate, or just want to talk Rust.

← Back to Vaelora