NeoLLM

NeoLLM is a 85.13 M parameter decoder-only language model trained from scratch on FineWeb-Edu with BF16 compute, completing training in approximately ~1h 19m on NVIDIA RTX PRO 6000 Blackwell Server Edition. It integrates a collection of recently published attention and normalization techniques into a single architecture, with the goal of studying how they interact during pretraining. The model is actively being developed and the current checkpoint represents an intermediate training state.

Author / contact: @Kyokopom on X Repository: KitsuVp/NeoLLM

---

Architecture

NeoLLM is a decoder-only transformer with the following configuration:

Parameter	Value
Hidden size	512
Layers	12
Attention heads	8
KV heads (GQA)	4
Head dim	64
Intermediate size	1536
Vocabulary	LiquidAI/LFM2.5-1.2B-Thinking tokenizer (64,402 tokens)
Context length	512 tokens
Leviathan	enabled=True; d_seed=128, modes=8, knots=16
JTok / JTok-M	enabled=False / False; Torch continuous surfaces, experts=5, top_k=2
DAGFormer	enabled=False; {'predictor_dim': 256, 'predictor_layers': 2, 'predictor_heads': 4, 'local_dim': 128, 'input_source': 'tokens'}
Linear Depth Memory	enabled=False; groups=8, key dimension=8; shared branch history when combined with SiameseNorm
Virtual Head Attention	enabled=False; query/KV heads follow the configured geometry
Copy-Paste LM	enabled=True; causal vocabulary/pointer mixture

Parameter breakdown

Parameter bucket	Count
Total parameters	85.13M (85,126,072)
Embedding parameters (tied)	32.97M (32,973,824)
Non-embedding parameters	52.15M (52,152,248)
Effective trainable parameters	85.13M (85,126,072)

Weight tying is enabled: the input embedding matrix and the language-model head share the same parameters, so the effective trainable budget is total − embed = 52.15M.

Integrated techniques

NeoLLM combines architecture modules, optional auxiliary objectives, and training-time optimizer/stability components from the following papers.

Embedding and token representation

• Learnable Multipliers (arXiv:2601.04890) — Adds per-row and per-column learnable scalar parameters to selected matrix layers and, when enabled, embeddings.
• Leviathan (arXiv:2601.22040) — Optional continuous token embedding generator that can replace the discrete input lookup table.
• Leviathan-JTok / JTok-M (arXiv:2602.00800) — Optional continuous adaptation of JTok over the shared Leviathan coordinate. JTok applies one normalized surface as a multiplicative MLP gate; JTok-M routes a Sigmoid+TopK mixture of continuous surfaces and adds the paper's depth-scaled residual. Runtime jtok_kernel_backend selects the PyTorch reference or the CCE Triton adapter; the JTok surfaces remain outside MXFP8 conversion.
• KHRONOS (arXiv:2505.13315) — Kernel/basis reference used by the Leviathan continuous token generator implementation.
• Spelling Bee Embeddings (arXiv:2601.18030) — Augments token embeddings with character-level spelling information.
• Token Embedding Manifold analysis (arXiv:2504.01002) — Reference motivation for treating token embeddings as structured objects rather than unconstrained lookup rows.

Attention, positions, and output projection

• Virtual Head Attention (arXiv:2610.17959) — Optional group-specific query premix and low-rank head postmix. NeoLLM shares its gated query projection across corresponding head slots in KV groups and applies postmix after the gate; inference weights are not automatically folded into standard GQA projections.
• Copy-Paste LM (arXiv:2610.19247) — Optional causal pointer and sink mixed with the vocabulary distribution. CUDA training uses CCE's sparse Copy-Paste kernels; inference returns dense mixed logits.
• FAN (arXiv:2502.21309) — Fourier Analysis Networks. A portion of the projection channels are dedicated to periodic cosine/sine features.
• MEA (arXiv:2601.19611) — Explicit Multi-head Attention. Adds small learnable interaction matrices between attention heads for K and V.
• LUCID (arXiv:2602.10410) — Applies a learned lower-triangular preconditioner to V before attention, decorrelating value representations across positions.
• Affine-Scaled Attention (arXiv:2602.23057) — Adds two learnable per-head scalars (α and β) to the softmax weights: [α·softmax(QKᵀ) + β]·V.
• XSA (arXiv:2603.09078) — Exclusive Self Attention. After computing attention, removes the component of the output aligned with the token's own value vector.
• Directional Routing (arXiv:2603.14923) — Each head learns K=4 directions in the output space; a learned router suppresses the attention output along each direction per input.
• Gated Attention (arXiv:2505.06708) — A sigmoid gate is applied to the attention output before the output projection, introducing non-linearity and preventing attention sinks.
• Momentum Attention (arXiv:2411.03884) — Modifies Q and K by subtracting a fraction of the previous position's Q and K values (causal first-difference).
• Interleaved Head Attention / IHA (arXiv:2602.21371) — Builds pseudo-heads from learned cross-head mixtures to create multiple attention patterns per original head.
• REPO (arXiv:2512.14391) — Context re-positioning module that learns contextual position coordinates above a configurable start layer.
• GRAPE (arXiv:2512.07805) — Group representational position encoding used by the REPO-GRAPE positional path.
• GOAT priors (arXiv:2601.15380) — Optional factorized attention log-prior channels inspired by trainable attention priors.
• Hadamard output projection (arXiv:2603.08343) — Replaces dense attention output projection with a structured Hadamard transform plus lightweight scaling.

Normalization, residual flow, and MLP

• SeeDNorm (arXiv:2510.22777) — Applied to Q and K projections. Dynamically rescales normalization from the input's own statistics.
• LayerNorm Scaling / LNS (arXiv:2502.05795) — Each layer's output is scaled by 1/√ℓ where ℓ is the layer index.
• GPAS (arXiv:2506.22049) — Gradient-Preserving Activation Scaling for residual junctions.
• PolyNorm (arXiv:2602.04902) — Replaces the standard MLP activation with normalized linear, quadratic, and cubic branches.
• SimpleGPT (arXiv:2602.01212) — Second-order geometry-inspired normalization strategy applied inside MLP projections.
• StackMemory / STACKTRANS (NeurIPS 2025) — Optional differentiable hidden-state stack between decoder layers.
• Attention Residuals / AttnRes (arXiv:2603.15031) — Optional learned depth-wise aggregation over previous layer outputs or block summaries.
• LAUREL (arXiv:2411.07501) — Optional learned augmented residual layer with residual-weight and low-rank variants.

Training objectives and training-time regularizers

• Cut Cross Entropy (Apple repository) — Memory-efficient next-token loss that avoids materializing the full token-by-vocabulary logits tensor. NeoLLM remains compatible with the upstream package when the extensions below are disabled.
• MiLe Loss (arXiv:2310.19531) — Optional detached, mean-normalized predictive-entropy weighting of token losses, implemented inside the extended CCE path.
• Output Embedding Centering / mu-loss (arXiv:2601.02031) — Optional lambda * ||mean(output_embeddings)||^2 regularizer for output-logit stability.
• MEAP (arXiv:2502.07490) — Optional training-only input corruption that masks a fixed fraction of eligible tokens while preserving clean next-token labels, causal attention, and the inference path.
• TWEO (arXiv:2511.23225) — Optional Transformers Without Extreme Outliers activation regularizer for FP8/low-bit-friendly training.
• NITP (arXiv:2605.24956) — Optional Next Implicit Token Prediction auxiliary objective using shallow-layer implicit token targets and a cosine loss.
• NextLat (arXiv:2511.05963) — Optional next-latent prediction objective using latent dynamics, Smooth L1 supervision, and frozen-head KL.

Optional extended-CCE configuration

Feature	Enabled	Value
MiLe Loss	True	gamma=1.0
mu-loss	True	lambda=0.0001
MEAP	True	ratio=0.15

Copy-Paste uses CCE sparse mixture statistics for CUDA training and computes mu-loss in Torch. MEAP and patch training also use the extended cut-cross-entropy package. Kitsunp/ml-cross-entropy supplies the extended kernels. Copy-Paste exposes three compact scalars: unweighted NTP cross entropy, the MiLe reweighting delta, and the mu-loss penalty. Their sum reconstructs ntp_loss exactly. MEAP reports its eligible and selected counts from the masking kernel; the trainer logs the selected count and exact fraction. Copy-Paste diagnostics use sparse source-token candidates and CCE vocabulary reductions.

Optimizer and training stability

• Adam-family optimizer mode — Active variant: AdEMAMix. AdEMAMix (arXiv:2409.03137) contributes the optional beta3 slow-memory branch when ADEMAMIX=True. DeltaMomentum (arXiv:2608.19491) optionally replaces only the fast beta1 first-moment update on hidden Linear weights when DELTA=True; beta2 remains Adam's ordinary squared-gradient EMA in every mode.
• BF16 stochastic rounding (arXiv:2502.20566) — Optional master-free BF16 model update with stochastic rather than nearest rounding; the same write-back path is shared by AdamW, DeltaAdamW, AdEMAMix, and DeltaAdEMAMix.
• Cautious Weight Decay (arXiv:2510.12402) — Sign-selective weight decay variant used by the custom optimizer logic.
• Correction of Decoupled Weight Decay (arXiv:2512.08217) — Adapts decoupled weight decay during learning-rate decay.
• AdamHD (arXiv:2511.14721) — Decoupled Huber decay regularization reference used by the optimizer.
• Gradient conditioning — CAST active; history-adaptive spectral/norm policies by optimizer group; 5208 committed updates. References: CAST, GradientStabilizer.

---

Training

Setting	Value
Dataset	FineWeb-Edu (sample-10BT)
Tokens seen	~0.51B (15,625 steps × batch 64 × length 512)
Precision	BF16 model state (bfloat16: 85,123,896 elements, float32: 2,176 elements) + AdEMAMix FP32 master active; stochastic rounding disabled
Optimizer	AdEMAMix
Gradient transform	CAST active; history-adaptive spectral/norm policies by optimizer group; 5208 committed updates
Learning rate	1e-03 with 4,000-step linear warmup
Patch transition	Reset optimizer/conditioner at step 10,417; per-phase LR warmup; restarted AdEMAMix warmup fitted to each phase
Weight decay	coefficients [0.005, 0.01, 0.1]; WD 5/5 groups; CWD 2; Huber 0; Correction 0
Training time	~1h 19m
Hardware	NVIDIA RTX PRO 6000 Blackwell Server Edition

Training curve — total objective

Total includes MiLe when enabled, plus all active auxiliary objectives and regularizers. Without MiLe means model total minus its MiLe delta on the same forward; this is not a separate training run without MiLe.

Step	Train total	Val total	Train without MiLe	Val without MiLe
1	12.1091	—	12.1091	—
250	7.48817	—	7.48006	—
500	7.38205	—	7.36692	—
750	7.31452	—	7.29579	—
1,000	7.27752	—	7.25599	—
1,250	7.24217	—	7.21878	—
1,500	7.21533	—	7.18957	—
1,750	7.1794	—	7.15178	—
2,000	7.1502	—	7.1206	—
2,250	7.12798	—	7.09679	—
2,500	7.09162	—	7.05962	—
2,750	7.05831	—	7.02534	—
3,000	7.03258	—	6.99856	—
3,250	7.00834	—	6.97288	—
3,500	6.99144	—	6.95501	—
3,750	6.97047	—	6.93302	—
4,000	6.95205	—	6.91422	—
4,250	6.93665	—	6.89784	—
4,500	6.92121	—	6.88164	—
4,750	6.90024	—	6.86	—
5,000	6.87952	6.97561	6.83771	6.91933
5,250	6.86819	—	6.82568	—
5,500	6.85762	—	6.81425	—
5,750	6.84215	—	6.79739	—
6,000	6.82791	—	6.78216	—
6,250	6.81792	—	6.77247	—
6,500	6.80506	—	6.7599	—
6,750	6.80079	—	6.75407	—
7,000	6.78446	—	6.73771	—
7,250	6.78035	—	6.7324	—
7,500	6.77858	—	6.73004	—
7,750	6.76912	—	6.72007	—
8,000	6.76376	—	6.7138	—
8,250	6.75292	—	6.70173	—
8,500	6.74686	—	6.69614	—
8,750	6.74092	—	6.68849	—
9,000	6.72985	—	6.67761	—
9,250	6.71476	—	6.66062	—
9,500	6.706	—	6.65167	—
9,750	6.70201	—	6.64772	—
10,000	6.68574	6.81049	6.62962	6.74182
10,250	6.67942	—	6.62312	—
10,500	6.51852	—	6.41485	—
10,750	5.68894	—	5.38453	—
11,000	5.55274	—	5.21286	—
11,250	5.48644	—	5.12636	—
11,500	5.44022	—	5.06406	—
11,750	5.40189	—	5.01526	—
12,000	5.37264	—	4.97773	—
12,250	5.34265	—	4.93939	—
12,500	5.31439	—	4.90518	—
12,750	5.30246	—	4.88358	—
13,000	5.27841	—	4.85558	—
13,250	5.2676	—	4.83947	—
13,500	5.24949	—	4.81518	—
13,750	5.25181	—	4.81448	—
14,000	5.23557	—	4.79578	—
14,250	5.22921	—	4.78735	—
14,500	5.22425	—	4.77866	—
14,750	5.20891	—	4.75632	—
15,000	5.20171	4.8294	4.74737	4.27092
15,250	5.17208	—	4.71224	—
15,500	5.15728	—	4.69331	—
15,625	5.13852	4.77795	4.67192	4.20901

Cross-entropy with and without MiLe

CE without MiLe is unweighted CE; CE with MiLe adds the reweighting delta. Both exclude mu and auxiliary terms. ntp_loss includes mu when enabled.

Step	Train CE without MiLe	Val CE without MiLe	Train CE with MiLe	Val CE with MiLe
1	11.1197	—	11.1197	—
250	7.2798	—	7.2879	—
500	6.98502	—	7.00015	—
750	6.87304	—	6.89176	—
1,000	6.80336	—	6.82489	—
1,250	6.74976	—	6.77315	—
1,500	6.70907	—	6.73483	—
1,750	6.67012	—	6.69774	—
2,000	6.64233	—	6.67192	—
2,250	6.62434	—	6.65553	—
2,500	6.59703	—	6.62903	—
2,750	6.57489	—	6.60786	—
3,000	6.55921	—	6.59323	—
3,250	6.5437	—	6.57916	—
3,500	6.52882	—	6.56525	—
3,750	6.51063	—	6.54809	—
4,000	6.49316	—	6.53099	—
4,250	6.48127	—	6.52008	—
4,500	6.46815	—	6.50771	—
4,750	6.44927	—	6.48952	—
5,000	6.43098	6.34724	6.47279	6.40352
5,250	6.42022	—	6.46272	—
5,500	6.41084	—	6.45422	—
5,750	6.39614	—	6.4409	—
6,000	6.38525	—	6.431	—
6,250	6.3778	—	6.42325	—
6,500	6.36691	—	6.41207	—
6,750	6.36165	—	6.40838	—
7,000	6.34741	—	6.39416	—
7,250	6.34213	—	6.39008	—
7,500	6.33801	—	6.38655	—
7,750	6.32711	—	6.37616	—
8,000	6.3207	—	6.37066	—
8,250	6.3071	—	6.35829	—
8,500	6.30117	—	6.35189	—
8,750	6.29348	—	6.34591	—
9,000	6.28187	—	6.33411	—
9,250	6.2649	—	6.31905	—
9,500	6.25528	—	6.30961	—
9,750	6.25096	—	6.30526	—
10,000	6.23301	6.15348	6.28913	6.22215
10,250	6.22786	—	6.28416	—
10,500	5.98752	—	6.09119	—
10,750	4.99507	—	5.29948	—
11,000	4.83642	—	5.17629	—
11,250	4.74897	—	5.10905	—
11,500	4.68281	—	5.05897	—
11,750	4.63018	—	5.01682	—
12,000	4.58826	—	4.98317	—
12,250	4.54662	—	4.94988	—
12,500	4.50783	—	4.91703	—
12,750	4.48224	—	4.90113	—
13,000	4.45022	—	4.87305	—
13,250	4.43033	—	4.85846	—
13,500	4.40326	—	4.83758	—
13,750	4.39858	—	4.83591	—
14,000	4.37625	—	4.81604	—
14,250	4.36535	—	4.8072	—
14,500	4.35465	—	4.80025	—
14,750	4.33096	—	4.78355	—
15,000	4.32014	3.87222	4.77448	4.4307
15,250	4.28448	—	4.74433	—
15,500	4.26551	—	4.72949	—
15,625	4.24431	3.81132	4.7109	4.38026

Observed loss components — weighted contributions

Only contributions actually added to the objective are shown; raw values and subcomponents remain in training metrics. regularization_loss is the subtotal of mu, weighted TWeo, JTok-M and StackMemory entropy, excluding NITP/NextLat; do not add it again to those components. JTok-M auxiliary loss is training-only.

Step	Phase	mu	TWeo	NITP	Reg. subtotal
1	train	3.15217e-10	8.85082e-07	0.989407	8.85397e-07
250	train	3.73292e-06	0.00123591	0.199023	0.00123965
500	train	8.67309e-06	0.000489802	0.381402	0.000498475
750	train	1.46084e-05	0.000645963	0.422095	0.000660571
1,000	train	2.19271e-05	0.000853222	0.45175	0.000875149
1,250	train	3.00306e-05	0.00107691	0.467909	0.00110694
1,500	train	3.78419e-05	0.00137081	0.479092	0.00140865
1,750	train	4.50641e-05	0.00167613	0.479939	0.0017212
2,000	train	5.17696e-05	0.00195085	0.47627	0.00200262
2,250	train	5.86737e-05	0.00217424	0.470216	0.00223291
2,500	train	6.65358e-05	0.0022861	0.460232	0.00235263
2,750	train	7.55516e-05	0.00232904	0.448046	0.00240459
3,000	train	8.65665e-05	0.00233349	0.436928	0.00242006
3,250	train	0.000101063	0.00227577	0.426803	0.00237683
3,500	train	0.00011896	0.00214756	0.423927	0.00226652
3,750	train	0.000140306	0.002038	0.420206	0.00217831
4,000	train	0.000165066	0.00194413	0.418951	0.0021092
4,250	train	0.000194861	0.0017438	0.414625	0.00193866
4,500	train	0.000232431	0.00167878	0.411586	0.00191121
4,750	train	0.000278571	0.0015199	0.408931	0.00179847
5,000	train	0.000333497	0.00136128	0.405027	0.00169478
5,000	validation	0.000364807	0.00445405	0.567271	0.00481886
5,250	train	0.000397402	0.00126008	0.403805	0.00165748
5,500	train	0.000464144	0.00113831	0.401809	0.00160246
5,750	train	0.000533964	0.00105294	0.399665	0.0015869
6,000	train	0.0006026	0.000934447	0.395382	0.00153705
6,250	train	0.000667589	0.000857554	0.393142	0.00152514
6,500	train	0.000727096	0.000791855	0.391478	0.00151895
6,750	train	0.000783413	0.000742837	0.390892	0.00152625
7,000	train	0.000839761	0.000718544	0.38874	0.0015583
7,250	train	0.000892616	0.000736747	0.388637	0.00162936
7,500	train	0.000940995	0.000717072	0.390372	0.00165807
7,750	train	0.000980058	0.000689226	0.391288	0.00166928
8,000	train	0.00101025	0.000673191	0.39142	0.00168344
8,250	train	0.00103412	0.000637854	0.392951	0.00167197
8,500	train	0.00105137	0.000579414	0.39334	0.00163079
8,750	train	0.00106911	0.000537723	0.393394	0.00160683
9,000	train	0.00108875	0.000505026	0.394151	0.00159378
9,250	train	0.00110437	0.00047421	0.394137	0.00157859
9,500	train	0.00111756	0.000446753	0.394824	0.00156432
9,750	train	0.00112671	0.00041636	0.395211	0.00154307
10,000	train	0.0011306	0.00038917	0.395094	0.00151977
10,000	validation	0.00113129	0.00109173	0.586116	0.00222302
10,250	train	0.00113103	0.000368104	0.393761	0.00149914
10,500	train	0.00113422	0.000535544	0.42566	0.00166977
10,750	train	0.00115112	0.000529416	0.387781	0.00168053
11,000	train	0.00113757	0.000359662	0.374953	0.00149723
11,250	train	0.00111374	0.00029127	0.37598	0.00140501
11,500	train	0.00108693	0.00025111	0.379912	0.00133804
11,750	train	0.00106742	0.000223842	0.383783	0.00129126
12,000	train	0.00104938	0.000209719	0.388213	0.0012591
12,250	train	0.00102569	0.000207818	0.391538	0.0012335
12,500	train	0.00100141	0.000200826	0.396153	0.00120224
12,750	train	0.000984945	0.000199516	0.400149	0.00118446
13,000	train	0.000973787	0.000204766	0.404183	0.00117855
13,250	train	0.00096588	0.000214577	0.407961	0.00118046
13,500	train	0.000960328	0.000234133	0.410719	0.00119446
13,750	train	0.00095725	0.000249427	0.41469	0.00120668
14,000	train	0.000963165	0.000276272	0.418295	0.00123944
14,250	train	0.000975121	0.000304745	0.420725	0.00127987
14,500	train	0.00098893	0.000333744	0.422681	0.00132267
14,750	train	0.00100775	0.00037004	0.423986	0.00137779
15,000	train	0.00102777	0.000389596	0.425814	0.00141737
15,000	validation	0.00103705	0.000423607	0.397243	0.00146065
15,250	train	0.00104482	0.000394994	0.426315	0.00143981
15,500	train	0.00105605	0.000392557	0.426342	0.00144861
15,625	train	0.00106163	0.000381018	0.426175	0.00144265
15,625	validation	0.00106291	0.000419964	0.396201	0.00148288

Tables use exact logged steps; missing values are — without carrying values forward. Repeated evaluations at one step use the latest logged value. Components average batch scalars weighted by examples; CE/PPL token metrics retain their valid-target weighting. Weight decay/CWD act in the optimizer, outside these forward losses.

---

Limitations

• Token budget — ~1.5 B tokens seen; below estimated optimum. Knowledge-intensive tasks will improve with more training.
• Gradient spike at step 40k — Reorganized the attention pattern in layer 9 that previously captured long-range token correlations. A checkpoint from ~step 38k is expected to have better aggregate benchmark scores.
• PolyNorm exclusivity — The quadratic branch has become partially redundant with the linear branch. Will be corrected in the next training run.
• Base model only — Not instruction-tuned or aligned; purely a next-token-prediction base model.

---

References

All papers whose techniques are integrated into NeoLLM's architecture, training objective, or training stack:

Area	Technique	Paper title	Reference
Embeddings	Learnable Multipliers	Freeing the Scale of Language Model Matrix Layers	arXiv:2601.04890
Embeddings	Leviathan	A Separable Architecture for Continuous Token Representation in Language Models	arXiv:2601.22040
Embeddings / residual modulation	JTok / JTok-M	JTok: On Token Embedding as another Axis of Scaling Law via Joint Token Self-modulation	arXiv:2602.00800
Embeddings	KHRONOS	KHRONOS: a Kernel-Based Neural Architecture for Rapid, Resource-Efficient Scientific Computation	arXiv:2505.13315
Embeddings	Spelling Bee	Spelling Bee Embeddings for Language Modeling	arXiv:2601.18030
Embeddings	Token embedding analysis	Token Embeddings Violate the Manifold Hypothesis	arXiv:2504.01002
Attention / positions	FAN	Fourier Analysis Networks	arXiv:2502.21309
Attention / positions	MEA	Explicit Multi-head Attention for Inter-head Interaction in Large Language Models	arXiv:2601.19611
Attention / positions	LUCID	Attention with Preconditioned Representations	arXiv:2602.10410
Attention / positions	Affine-Scaled Attention	Affine-Scaled Attention: Towards Flexible and Stable Transformer Attention	arXiv:2602.23057
Attention / positions	XSA	Exclusive Self Attention	arXiv:2603.09078
Attention / positions	Directional Routing	Directional Routing in Transformers	arXiv:2603.14923
Attention / positions	Gated Attention	Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free	arXiv:2505.06708
Attention / positions	Momentum Attention	Momentum Attention	arXiv:2411.03884
Attention / positions	IHA	Interleaved Head Attention	arXiv:2602.21371
Attention / positions	REPO	Language Models with Context Re-Positioning	arXiv:2512.14391
Attention / positions	GRAPE	Group Representational Position Encoding	arXiv:2512.07805
Attention / positions	GOAT priors	You Need Better Attention Priors	arXiv:2601.15380
Attention / positions	Hadamard o_proj	Rethinking Attention Output Projection: Structured Hadamard Transforms for Efficient Transformers	arXiv:2603.08343
Residual / normalization	SeeDNorm	Self-Rescaled Dynamic Normalization	arXiv:2510.22777
Residual / normalization	LNS	The Curse of Depth in LLMs	arXiv:2502.05795
Residual / normalization	GPAS	Gradient-Preserving Activation Scaling	arXiv:2506.22049
Residual / normalization	PolyNorm	PolyNorm / PolyCom	arXiv:2602.04902
Residual / normalization	SimpleGPT	SimpleGPT	arXiv:2602.01212
Residual / normalization	StackMemory / STACKTRANS	Recursive Transformer: Boosting Reasoning Ability with State Stack	NeurIPS 2025
Residual / normalization	Attention Residuals	Attention Residuals	arXiv:2603.15031
Residual / normalization	Linear Depth Memory	Depth Memory	arXiv:2610.18809
Residual / normalization	LAUREL	LAUREL: Learned Augmented Residual Layer	arXiv:2411.07501
Objectives	TWEO	Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies	arXiv:2511.23225
Objectives	NITP	Next Implicit Token Prediction for LLM Pre-training	arXiv:2605.24956
Objectives	NextLat	Next-Latent Prediction Transformers Learn Compact World Models	arXiv:2511.05963
Optimizer / training	AdEMAMix	The AdEMAMix Optimizer: Better, Faster, Older	arXiv:2409.03137
Optimizer / training	DeltaMomentum	DeltaMomentum: A Key-Value based Anisotropic Momentum Update via Delta Rule	arXiv:2608.19491
Optimizer / training	BF16 + SR	Stochastic Rounding for LLM Training: Theory and Practice	arXiv:2502.20566
Optimizer / training	CWD	Cautious Weight Decay	arXiv:2510.12402
Optimizer / training	WD correction	Correction of Decoupled Weight Decay	arXiv:2512.08217
Optimizer / training	AdamHD	AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training	arXiv:2511.14721
Optimizer / training	GradientStabilizer	GradientStabilizer	arXiv:2502.17055

---

Citation

@misc{neollm2026,
  title  = {NeoLLM: A Research Language Model Integrating Recent Attention and Normalization Techniques},
  author = {KitsuVp},
  year   = {2026},
  url    = {https://huggingface.co/KitsuVp/NeoLLM}
}

---

Author

@Kyokopom on X

---

License

Apache 2.0