We build open-weight foundation models from scratch. Combining content-conditional modulation with proprietary memory-factored optimization to unlock deep semantic generalization early in pretraining.
Dense 36-layer decoder-only transformer combining pre-norm SwiGLU foundations with content-conditional positional modulation.
| Hyperparameter | Value | Role |
|---|---|---|
| Total Parameters | 1,122,571,848 | 1.12B total capacity |
| Backbone Params | 1,021,910,088 | Non-embedding dense capacity |
| Depth & Width | 36 Layers · d_model 1536 | 24 heads (head dim 64) |
| MLP Block | SwiGLU (ff = 4,096) | No bias, gated expansion |
| Context & RoPE | 4,096 tokens · θ = 1,000,000 | NTK-scaled frequency |
| Vocabulary | 65,535 (uint16 tied) | Custom Byte-Level BPE |
Modulates positional fields dynamically via sequence hidden states for input-sensitive attention routing.
Learned block-level density profiles softly gate attention residuals on outlier or novel tokens.
Dimension-wise input gating before token embeddings enter the 36-layer dense stack.
Katana 1.1B pretrains entirely from scratch within a single consumer 12GB GPU with the complete optimizer state resident. The mechanisms that make this possible are proprietary and intentionally not detailed here.
Our breakthrough matrix-factorized AdamW variant cuts optimizer VRAM by up to 99.99%. Now with unified multi-precision: Lossless 4-Bit, 8-Bit, FP16, BF16, and FP32. Authored by Jassim Ibrahim Farooqi.
Custom variable-density gradient and activation scheduling locks peak VRAM to ~9.1 GB inside the 12GB envelope with zero host-RAM offloading.
Katana processes 32,768 tokens per step in ~14.37s per step, sustaining ~2,280 tokens/sec continuously on a single consumer RTX 3060 GPU.
Structured for coding, math, and agentic workflows. Over 52% of the dataset consists of code repositories and mathematical proofs.
Katana 1.1B open-weights base release on December 1, 2026. Continuous pretraining and finetuned (Coder-Instruct) weights will follow sequentially.
Foundational dense transformer for coding and agentic workflows. 36 layers with content-conditional position modulation.
Vision-Language multimodal adaptation combining Katana's syntactic core with visual token encoders.
Next-gen edge architecture for sub-10ms agentic decision loops and local function execution.
Deep reasoning architecture for mathematical verification, theorem synthesis, and complex AST refactoring.
Flagship sovereign open-weights model family. Scaled across dedicated cluster compute.
Long-horizon multi-agent synthesis system with real-time recursive self-verification.
Trained 1.12B parameters from scratch with full optimizer state resident on a single 12GB GPU at 2.28k tok/s.
157 GB token array with 52% code/math composition built at 16.6M tok/s.
Field report documenting semanticization through ~2.4B tokens: DOM ASTs, Unity il2cpp & reflection dialects, code-gate mapping, and temperature→genre phase maps. Read Report →
Clean local inference surface — katana-130k-Q8_0.gguf serving the on-box test harness with a clean GGUF export for iRun Studio / llama.cpp.
Public release on Hugging Face (OpenBlade/Katana-1.1b-base) and GGUF format. Base model trained through ~22.45B tokens (exactly 20 tokens/param, Chinchilla-optimal regime).
Sequential releases of advanced checkpoints and specialized coding/instruction-tuned weights.
Comparing the pretraining loss trajectory and early syntactic acquisition of Katana 1.1B against an industry-average 1.1B dense transformer.
Hover over Katana milestone points to inspect semantic acquisition states. Points ≤ 70.5k: logged moving-average; later points: 32-block held-out eval snapshots.
At under 0.9% of corpus (~737M tokens) Katana 1.1B learned syntax, code idioms, and domain vocabulary well before factual memorization — and by step 120k (~2.4B tokens) it was sustaining niche registered dialects: il2cpp decompiler text, Mono.Cecil reflection APIs, isomorphic DOM ASTs, and theology prose. Below, the on-device field log.
At Step 120k the code dialect fires at a lowered sweet-temperature (~0.85) and holds real Unity type names —
Type_t4DA9…, NullCheck,
VirtualFuncInvoke, System.Array`2<UnityEngine.UIElements…> —
persisting past the collapse window that ended earlier checkpoints' runs.
.RuntimeArray_t*)String_0;
_t* L_30 = V_0;
L_39 = Type_t4DA9F9F55A0C4A0B7A8D41C8A565F1;
NullCheck(L_8);
// System.Array`2<UnityEngine.UIElements.StylePropertyAnimationSystem…>
// … the il2cpp decompiler register, live at Step 120k.A high-performance macOS app on the Mac App Store. Built exclusively for Apple Silicon with custom Metal acceleration for zero-latency private execution.
Optimized for M-series unified memory, sustaining blazing on-device inference.
Seamlessly loads Katana 1.1B GGUF weights, image diffusion, and vision models.
Zero telemetry. Run local code assistants and agents completely offline.
OpenBlade Dense Foundation Model
OpenBlade is an independent ML research foundry in the EU. Hardware partnerships and academic compute grants directly accelerate open foundation models for science.
Compute partnerships unlock multi-node cluster scaling, moving Katana from single-GPU validation to frontier convergence.
Accelerates parallel training across Katana-VLM, Dagger, Reaper, and the sovereign Titan model.
All checkpoints, alignment finetunes, and research evaluations are released openly on Hugging Face.
Katana 1.1B base checkpoint (unlocking December 1, 2026):
from transformers import AutoModelForCausalLM, AutoTokenizer
repo_id = "OpenBlade/Katana-1.1b-base"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype="auto")
inputs = tokenizer("<html><body>", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100, temperature=0.9)
print(tokenizer.decode(outputs[0]))Academic BibTeX citation for Katana 1.1B technical report:
@techreport{farooqi2026katana,
title = {Katana 1.1B: Early Semanticization and Discrete Optimization in Open-Weight Transformers},
author = {Jassim Ibrahim Farooqi},
year = {2026},
institution = {OpenBlade EU Foundry},
url = {https://www.openblade.org/report},
note = {Ongoing research}
}