C-LPNCM-Tiny-124M
Calibrated Looped-Pointer Next-Concept Model (123.6M · TinyStories)
Pure-Transformer Hierarchical Concept Planning · Weight-Tied Equilibrium Refinement · Dual-Path Pointer Decoding
entity2260/cl-pncm-tiny-124m is a compact 123.6M-parameter pure-Transformer Calibrated Looped-Pointer Next-Concept Model (C-LPNCM) trained on TinyStories (hazyresearch/ncm-datasets) for fast single-GPU experimentation (fits easily on a single Tesla T4 / Colab GPU).
Looking for the larger flagship checkpoint? See
entity2260/cl-pncm-2.8b(2.89Bphysical /3.60Beffective parameters) in the C-LPNCM Collection.
Highlights
- Zero Diffusion, 100% Pure Transformer — Shifts generation from flat token-by-token autoregression to discrete multiscale quantization (
Cosine VQ + 3× FSQ) and parallel span decoding. - Leak-Free Dual-Path Surface Decoding — Path A decodes strictly from quantized representations $E_{\text{quant}} = U_0 + \sum_{\ell=2}^{4} \hat{z}^{(\ell)}$, while Path B queries an unquantized $(K_{\text{raw}}, V_{\text{raw}})$ skip bank strictly at prior tokens $j < t_{\text{start}}(m)$ (with
<NULL_PTR>) to preserve proper nouns (e.g.,"Tom") verbatim. - Guarded XSA & Banach Contraction Loops — Module 3 iterates
3weight-tied Transformer layers up to6times with anchor re-injection ($U_0$) and Banach contraction regularization. - Strictly Proper Brier Epistemic Calibration — Module 2 detaches backbone features before $q_\phi$ and optimizes Brier loss $(\hat{p}_{\text{conf}} - \mathbb{I}[\text{correct}])^2$ for honest uncertainty-driven routing.
Architecture & Model Family
General C-LPNCM three-tower / four-module architecture (shown with 2.8B dimensions; exact 124M hyperparameters are listed below).
C-LPNCM-Tiny-124M Configuration
| Specification | Value |
|---|---|
| Total Parameters | 123.6M |
Vocabulary Size (vocab_size) |
8,192 (custom BPE, bpe_vocab_8192.pickle) |
Hidden Dimensions (d_enc / d_core / d_dec) |
512 / 768 / 512 |
Layers (Enc / Core / Loop / Dec) |
4 / 8 / 3 (×6 loops) / 4 (+ 2-layer Boundary Refiner) |
Attention Heads (Enc / Core / Loop / Dec) |
8Q/2KV · 12Q/4KV · 8 · 8 |
| Coarse Concept Codebook ($K_1$) | 4,096 Cosine VQ codes with EMA updates |
| Fine Residual Hierarchy ($L_{\text{FSQ}}$) | 3 cascaded FSQ levels ($z^{(2..4)}$) × [8, 8, 8, 6, 5] |
| Dynamic Concept Patch Size ($S_m$) | $1 \le S_m \le 8$ (split at $\tau_H = 2.4\text{ bits}$, window $W=128$) |
| Epistemic Routing Thresholds | $\tau_{\text{high}} = 0.85$ (Fast) · $\tau_{\text{low}} = 0.40$ (Abstain) · 4 deliberative beams |
C-LPNCM Family Comparison
| Model Variant | Checkpoint / Config | Total Params | $d_{\text{enc}} / d_{\text{core}} / d_{\text{dec}}$ | Layers (Enc / Core / Loop / Dec) |
Coarse VQ | Target Hardware |
|---|---|---|---|---|---|---|
| C-LPNCM-Tiny-124M (this repo) | entity2260/cl-pncm-tiny-124m |
123.6M | 512 / 768 / 512 |
4 / 8 / 3 (×6) / 4 |
4,096 |
1× Tesla T4 (16 GB) |
| C-LPNCM-660M (Base) | CLPNCMConfig.base() |
659.6M | 1024 / 1536 / 1024 |
4 / 12 / 4 (×6) / 4 |
4,096 |
1× L4 / A100 |
| C-LPNCM-1B | configs/cl_pncm_1b_a100.yml |
1.04B | 1024 / 2048 / 1024 |
6 / 12 / 4 (×6) / 6 |
16,384 |
1× A100 (40/80 GB) |
| C-LPNCM-2.8B (flagship) | entity2260/cl-pncm-2.8b |
2.89B (3.60B eff.) |
1536 / 3072 / 1536 |
6 / 20 / 4 (×8) / 6 |
16,384 |
1× L4 / T4 (inf.) · A100 |
| C-LPNCM-8B (Prod) | configs/cl_pncm_8b.yml |
7.85B | 2048 / 4096 / 2048 |
6 / 24 / 4 (×8) / 6 |
16,384 |
4×–8× A100/H100 |
Training Metrics & Qualitative Output
Trained on hazyresearch/ncm-datasets (tinystories/train_bpe_8192.bin, 542.7M tokens, custom BPE vocabulary $V=8{,}192$):
| Metric | Warmup Progression (1,000 steps) |
|---|---|
| Stage 1 Surface Token Reconstruction CE Loss | 9.0726 → 1.2899 |
| Stage 1 Auxiliary Entropy Head Loss | 9.1262 → 4.1266 |
| Stage 2 Coarse Next-Concept CE Loss | 8.5003 → 6.1182 |
| Stage 2 Multi-Exit FSQ Equilibrium Loss | 28.0830 → 2.7818 |
| Stage 2 Banach Contraction Penalty | 0.0105 → 0.0014 |
Qualitative Example (1,000-Step Warmup Checkpoint)
| Stage | Output |
|---|---|
| Input Prompt (16 tokens) | Once upon a time, there was a boy named Tom and |
| Concept-Bottleneck Reconstruction | Once upon a time, there was a boy very Tom and (15/16 tokens exact, proper noun "Tom" preserved) |
| Generated Next-Concept Continuation | He was happy then found laughing and (6 concepts → variable span lengths [1, 1, 2, 1, 2, 1]) |
Quickstart Usage
Option A: Load Merged Checkpoint Directly with CLPNCM
import pickle
import torch
from huggingface_hub import hf_hub_download
from looped_dit.cl_pncm import CLPNCM, CLPNCMConfig
repo_id = "entity2260/cl-pncm-tiny-124m"
device = "cuda" if torch.cuda.is_available() else "cpu"
ckpt_file = hf_hub_download(repo_id=repo_id, filename="cl_pncm_merged.pt")
vocab_file = hf_hub_download(repo_id=repo_id, filename="bpe_vocab_8192.pickle")
ckpt = torch.load(ckpt_file, map_location=device, weights_only=True)
with open(vocab_file, "rb") as f:
bpe_vocab = pickle.load(f)
cfg = CLPNCMConfig.from_dict(ckpt["config"])
model = CLPNCM(cfg).to(device)
model.load_state_dict(ckpt["model"])
model.eval()
prompt_ids = torch.tensor([[1, 412, 895, 19, 334, 28, 12, 601]], device=device)
gen_out = model.generate(prompt_ids, max_concepts=6)
print("Generated Token IDs:", gen_out["tokens"])
Citation
If you use this model or build upon Next-Concept Modeling, please cite the foundational Next-Concept Modeling (NCM) paper and this checkpoint:
Next-Concept Modeling (NCM) Paper
@misc{kumbong2026ncm,
title = {Next-Concept Modeling: Fast, Parallel Coarse-to-Fine Text Generation},
author = {Hermann Kumbong and Ishan S. Khare and Iddah Mlauzi and Andrew Woen and Charles Ding and E. Kelly Buchanan and Simran Arora and Azalia Mirhoseini and Christopher R{\'e}},
year = {2026},
eprint = {2610.14162},
archivePrefix = {arXiv},
primaryClass = {cs.CL},
url = {https://arxiv.org/abs/2610.14162}
}
C-LPNCM-Tiny-124M Model
@misc{entity2260_clpncm_tiny_124m_2026,
title = {C-LPNCM-Tiny-124M: Calibrated Looped-Pointer Next-Concept Model},
author = {Ranjith Kumar (entity2260)},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/entity2260/cl-pncm-tiny-124m}}
}
License
This model and its accompanying code are released under the MIT License.
- Downloads last month
- 176
Dataset used to train entity2260/cl-pncm-tiny-124m
Collection including entity2260/cl-pncm-tiny-124m
Paper for entity2260/cl-pncm-tiny-124m
Evaluation results
- Stage 1 Token Reconstruction CE Loss (1k steps) on TinyStories (hazyresearch/ncm-datasets)self-reported1.290
- Stage 2 Multi-Exit FSQ Equilibrium Loss on TinyStories (hazyresearch/ncm-datasets)self-reported2.782