Katana 1.1B: Vertical Co-Design, Discrete Optimization, and Early Semanticization in Open-Weight Transformers

01. Executive Summary

Katana 1.1B is a 1.12-billion-parameter dense decoder-only transformer (1,122,571,848 total export parameters) trained completely from scratch on a bespoke 84.42-billion-token curated corpus. Engineered as the foundational release of the OpenBlade model hierarchy, it targets high-precision coding, formal mathematical synthesis, and agentic reasoning workloads, with a multimodal vision-language (VLM) variant scheduled downstream.

Five core engineering and empirical pillars define the Katana 1.1B breakthrough:

• 1. Consumer Hardware Resident Pretraining (~9.1 GB / 12 GB VRAM): The complete 1.12B parameter architecture trains from scratch on a single consumer NVIDIA GeForce RTX 3060 12GB GPU ($300 consumer silicon) with the full reference optimizer state resident directly on-device. Peak memory footprint remains locked at ~9.1 GiB, leaving a ~2.9 GiB safety headroom with zero CPU host-memory offloading and zero PCIe bandwidth bottlenecks.
• 2. High-Throughput Step Dynamics (32,768 Tokens/Step @ ~14.37s): The training engine scales to 32,768 tokens per step (effective batch 8 × 4,096 tokens context chunk) executing in ~14.37 seconds per step, sustaining a steady ~2,280 tokens/second throughput on consumer silicon. Over ~197 million tokens are processed every 24 hours on a single desktop card.
• 3. The Vertical Co-Design Paradigm ("Custom Everything"): Katana replaces off-the-shelf glue with in-house vertical engineering: a 36-layer deep hierarchy with content-conditional positional modulation and dynamic activation gating; a custom 84.42B-token dataset rich in formal syntax; an in-house 65,535-token uint16 byte-level BPE tokenizer cutting storage by 50%; and a bespoke memory-factored optimizer (GPUFactorizedAdamV2) maintaining the entire 1.12B parameter optimizer state in just ~10 MB.
• 4. Shattering Accuracy and Perplexity Baselines: At Step 70.5k (~737M tokens, <0.87% corpus exposure), validation perplexity drops to 61.0 with Top-1 accuracy (Acc@1) reaching 30.9% (Acc@5: 50.4%). At Step 116.4k (~2.23B tokens), held-out evaluation reaches Loss 4.0624 and Perplexity 58.12, continuing to Perplexity ~54.97 at the live Step 130.5k baseline.
• 5. Early Structural Emergence Preceding Memorization: By Step 70.5k the model generates non-collapsing multi-line Bootstrap 5 DOM markup, C/Rust signatures, and DataFrame AST structures. By Step 120k it holds the specialized Unity il2cpp decompiler dialect and systematic theology prose at lowered sweet-temperatures.

02. The Vertical Co-Design Advantage: Why Katana Outperforms Regular Models

Most 1B-scale models in circulation are either scaled-down clones of standard enterprise recipes (e.g. Pythia-1B, TinyLlama-1.1B, OpenLLaMA-1B) or pruned derivatives of larger foundations. They inherit shallow layer configurations, uncurated web dumps, generic tokenizers, and standard AdamW optimizers requiring enterprise compute clusters.

Katana 1.1B rejects the assembly of off-the-shelf components. Instead, every layer of the pipeline was co-engineered as a unified engine:

03. The Technical Magnitude of Katana: Why It Is a Big Deal

Katana 1.1B establishes that sovereign, high-grade model pretraining is no longer restricted to multi-million-dollar server clusters. The engineering breakthrough is defined by three interconnected operational realities:

Overcoming the Memory Barrier: On standard training frameworks, attempting to train 1.12B parameters on a 12GB GPU triggers an immediate Out-Of-Memory (OOM) abort during the first backward pass. Katana keeps 100% of weights and optimizer states resident within ~9.1 GiB of VRAM, leaving ~2.9 GiB of safety headroom. There is zero host-RAM offloading over PCIe, eliminating the bus saturation that cripples standard small-lab training setups.

Crushing Perplexity on a 65,535 Vocabulary: In statistical language modeling, perplexity scales with vocabulary size—predicting over 65,535 classes is substantially harder than predicting over 32,000 classes. Conventional 1B models at equivalent token exposure (<1% to 2.5% of corpus) typically exhibit perplexities in the 120–250+ range and Acc@1 below 20–24%. Katana achieves PPL 61.0 at 737M tokens and PPL 58.12 at 2.23B tokens (continuing to PPL ~54.97 past Step 130k), with Top-1 accuracy saturating above 30.9%.

04. Comprehensive Head-to-Head Comparative Benchmark

The table below contrasts Katana 1.1B against industry-standard 1B-class transformer pretraining runs under equivalent training regimes:

05. Model Architecture Specification

Katana 1.1B uses a dense residual pre-norm transformer backbone. The parameter budget is allocated across 36 dense layers, SwiGLU feed-forward networks, and tied byte-level BPE embeddings.

Proprietary Block Mechanisms: Katana blocks carry content-conditional positional modulation and learned dynamic activation gating. Novel or outlier activations receive calibrated scaling relative to expected layer manifolds. These operate as static weights at inference, adding zero inference-time latency penalties.

06. The Bespoke Training Harness: Engine Mechanics

Training a 1.12B model from scratch within 12GB of on-device VRAM required a custom training harness (fuser):

• ● GPUFactorizedAdamV2: Decomposes second-moment matrices for 2D weights into rank-1 row and column vectors, reducing second-moment memory from O(d_out * d_in) to O(d_out + d_in). The complete optimizer state consumes just ~10 MB across the entire 1.12B parameter model.
• ● Chunked Uncheckpointed Head Loss: Cross-entropy loss across the 65,535 vocabulary is computed sequentially in 512-token chunks without checkpointing the linear projection, saving an entire backward recompute pass over the language model head.
• ● Variable-Density Gradient Checkpointing: Auto-probing benchmarks third, half, and two-third densities on physical hardware at step 0, identifying third density as the optimal Pareto frontier for ~9.1 GiB peak VRAM.
• ● Asynchronous Pinned Prefetcher: Dedicated background worker streams pinned memory blocks directly to VRAM via non-blocking CUDA streams, completely hiding data loading latency.

07. The 84.42B-Token Curated Corpus & uint16 Tokenizer

The dataset was engineered with a heavy focus on formal structure (code and mathematics account for 52.1% of the tokens):

08. Empirical Evidence: Early Semanticization & Longitudinal Milestones

At under 0.9% of corpus exposure (~737M cumulative tokens) Katana 1.1B demonstrated structured acquisition progressing in strict order: Fluency → Formatting → Syntax → Structure → Broad Semantics. Continuing through ~2.4B tokens (Step 120k) and live Step 130k+, the model stabilized niche registered dialects: Unity decompiler bytecode, Mono.Cecil reflection APIs, and systematic theology prose.

^† Rows ≥ 98k are 32-block held-out eval snapshots recorded under the batch-8 configuration (32,768 tokens/step). Earlier rows are 20-step moving averages logged at save points. ^‡ indicates live post-resume telemetry.

09. Qualitative Probe Corpus: Full-Length Best Samples

The decisive evidence of structural acquisition lies in raw, unedited generative decodes. Sampling used GGUF (Q8_0) on-box via llama.cpp with top_k=40.

Sample A — Unity il2cpp Dialect, Step 120k, T=0.85, ~192 tokens (Full Hold):

.RuntimeArray_t*)String_0;
		_t* L_30 = V_0;
		bool L_7;
		L_39 = Type_t4DA9F9F55A0C4A0B7A8D41C8A565F1;
		memset((int32_t)il2cpp_codegen_subtract(L_7, L_6, NULL);
		NullCheck(L_8);
		L_26 = VirtualFuncInvoke(4< bool, IComparer_t5C5A8D4E4A8C013C1E5BAB0C323_0CAB0C512CBA08F, const RuntimeObject*, 1;
// System.Array`2<UnityEngine.UIElements.StylePropertyAnimationSystem.InputState>>::IsNullOrEmpty,T>,System.Single>,UnityEngine.StyleProperty`1,System.Visual(Lights, L_5, L_4;
}
// System.VoidArray`1<UnityEngine.UIElements.UIElements.StylePropertyAnimationSystem/ElementPropertyAnimationSystem.KeyValuePair,UnityEngine.UIElements.StylePropertyAnimationSystem/Values`1>,UnityEngine.UIElements.UIElements.StylePropertyAnimation>

// SystemEngine.UIElements.StyleProperty

Sample B — Contemporary Web DOM Scaffold, Step 70.5k, T=0.9, 100 tokens:

<div class="comment mb-4_4">9</div class="muted mb-0" data-0 data-id="d6j">
    <p>
    <div class="byline text-0 pt-muted mb-0"><a href="javascript:;" class="score">
      <span class="badge badge-secondary">1</span></a>&nbsp;&nbsp;&nbsp;<a class="author">

Sample C — Systematic Theology Register, Step 116k, T=1.0, 100 tokens (Verbatim):

"what is the Bible" →
 of the Jewish of the Gospel, and the Bible of God.
Jesus is a message of God, the Holy Spirit in the Holy Spirit, will bless the Spirit, and the Son, the Holy Spirit to the Father.
The Lord will be a beautiful reminder, to the Spirit.
The Bible is a priest. The Bible is a strong faith, and the church, the church of the Holy of the Church.
Jesus is a very well.
The church was bapt, the Baptist

Sample D — Android Java-to-XML Layout Transition, Step 109k, T=1.2:

getComponent(R) { ... if (m4 == null) {
   case "id" : ... <template> <state> <int key="3" /> <Value value="12"> ...

10. Register Taxonomy & Temperature Phase Maps

Sampling temperature acts as an architectural genre selector, revealing discrete attractors in the parameter space rather than smooth interpolation:

The Sweet-Temperature Law: The optimal temperature for complex code dialects descends as training deepens (Unity il2cpp: T=1.15 @ 98k → T=0.90 @ 109k → T=0.85 @ 120k), confirming growing model confidence in stored syntactic clusters.

11. Diagnostic Boundaries & Honest Negative Controls

Negative results are documented alongside wins to establish honest capability boundaries:

12. Release Milestones & Open Weights Commitment

OpenBlade is committed to open weights by default. All checkpoints will be released openly on Hugging Face:

• ● GGUF Export Pipeline (Shipped): Model-local exports (katana-130k-f16.gguf, katana-130k-Q8_0.gguf) serving on-box testing via llama.cpp.
• ● December 1, 2026 Public Base Release: Public drop of Katana 1.1B base weights on Hugging Face (OpenBlade/Katana-1.1b-base, trained through ~22.45B tokens, 20 tok/param) and GGUF export for local on-device inference via iRun Studio.
• ● Continuous Pretraining & Finetunes: Sequential checkpoint releases and aligned coder-instruct models.
• ● Future Model Ladder: Katana-VLM (multimodal vision), Dagger (ultra-low-latency edge), Reaper (deep reasoning), Titan (sovereign foundation), and Excalibur.

13. Research Citation

@techreport{openblade2026katana,
  title       = {Katana 1.1B: Early Semanticization and Discrete Optimization in Open-Weight Transformers},
  author      = {OpenBlade ML Research Foundry},
  year        = {2026},
  institution = {OpenBlade EU Foundry},
  url         = {https://www.openblade.org/report}
}