COLM — Complex Oscillating Language Model

Paper: Zenodo (PDF) | Code: GitHub | Dataset: edeneldith/DCDM | Predecessor: WiggleGPT (Zenodo)

Author: Phillip C. O'Brien — ORCID 0009-0007-3961-1182

What is COLM?

COLM is a novel autoregressive language model that operates entirely in the complex number plane using oscillatory neurons. It replaces the transformer's quadratic-complexity self-attention with an O(N) causal recurrence driven by complex-valued gates, and replaces all learned linear transformations in its core blocks with fixed unitary rotations and element-wise complex oscillatory activations.

Zero nn.Linear layers in the processing blocks — all transformation is performed by the oscillating activation sin(W * Z + B) * tanh(Z) where W, B are complex-valued, routed through fixed energy-preserving complex mixers.

Key Results

Metric	Value
Learned parameter elements	498,214 (445,710 real + 52,504 complex)
Best validation loss	1.1449
Creativity score (GPT-5.4 blind eval)	4.83 / 10
Age group estimate	84% rated age 13-16
Training time	8.7 hours
Hardware	Single RTX 5060 Ti 16GB
Tokenizer	499 tokens: 396 word tokens, 99 single-character tokens (including whitespace), 4 special tokens
Domain	Theological-philosophical prose

At 498k parameters — roughly half the size of TinyStories' smallest coherent model — COLM generates thematically coherent philosophical prose at temperature 1 with no spell correction.

The parameter count follows the native training log: one complex-valued element counts as one parameter element. Counting its real and imaginary components separately gives 550,718 real scalar degrees of freedom. The 32 fixed mixer matrices contain another 3,359,232 complex elements, which are required model state but are not learned parameters. The SafeTensors weights contain only learned parameters; the original mixer values are preserved in model_buffers.npz.

Intended Use

COLM is an experimental research model for studying complex-valued oscillatory language modeling and generating theological-philosophical prose in its training domain. It is a native PyTorch architecture, loaded with the code in this repository. The text-generation tag describes its task; this release does not provide a Transformers AutoModel integration or hosted inference endpoint.

Architecture

Component	COLM
State	Native torch.cfloat throughout
Activation	sin(W * Z + B) * tanh(Z), complex W, B
Sequence routing	O(N) causal recurrence via torch.cumsum
MLP/FFN	Fixed unitary mixer -> Oscillator -> mixer -> Oscillator
Residual	Complex sinc resonance coupling
Normalisation	ComplexRMSNorm (phase-preserving)
Sparsity	Learnable sigmoidal gate on magnitude

Model Configuration

{
  "n_embd": 324,
  "n_layer": 16,
  "embed_dim": 66,
  "block_size": 128,
  "vocab_size": 499
}

These dimensions are recovered from colm_best_Final.pt and confirmed by the Final run's training log. The released best checkpoint is step 860,000, with validation loss 1.144872358441353. The complete training run lasted about 8.7 hours; that duration is not the time of the best checkpoint.

colm_config.json now matches this checkpoint, including batch size 4 in the logged training metadata. Before the 8 November 2026 packaging update, the downloadable config described a different 512-wide, 12-layer run even though the table above described the released 324-wide, 16-layer model. The corrected file records the checkpoint architecture, inference settings and confirmed training metadata; it is not a complete training-resume recipe.

Files

File	Description
model.safetensors	264 learned parameter tensors, preserving native float32 and complex64 values
model_buffers.npz	32 original fixed complex mixer matrices; required alongside the SafeTensors weights
colm_best_Final.pt	Original best checkpoint, including optimizer state; retained unchanged
colm_config.json	Released-checkpoint architecture, inference settings and confirmed training metadata
colm_tokenizer.json	499-token word+character hybrid tokenizer vocabulary
model.py	All nn.Module classes needed to load the model
tokenizer_train.py	Original tokenizer implementation; use COLMTokenizer.load with the saved vocabulary
colm_loader.py	Strict native loader for the SafeTensors package or original .pt checkpoint
export_safetensors.py	Reproduce and verify the lossless export from the original checkpoint on CPU
conversion_report.json	Source hash, file hashes, tensor counts and exact-state verification results

Usage

Use Python 3.11+, PyTorch, NumPy and SafeTensors 0.7.0 or newer (for native complex64 storage). The loader preserves complex dtypes; do not cast the model to a real dtype such as float16 or bfloat16.

Download the files together:

hf download edeneldith/COLM model.safetensors model_buffers.npz colm_config.json colm_tokenizer.json model.py colm_loader.py tokenizer_train.py --local-dir COLM
cd COLM

Load the native model and its existing tokenizer:

import torch
from colm_loader import load_colm
from tokenizer_train import COLMTokenizer

torch.set_num_threads(1)
model = load_colm(".")  # CPU loading only; no generation or GPU allocation
tokenizer = COLMTokenizer.load("colm_tokenizer.json")
prompt_ids = tokenizer.encode("The nature of consciousness")

# When ready to run inference on a free CUDA device, move the model explicitly:
# model = model.to(device="cuda")

Both export files are necessary. The loader restores all learned tensors and the saved fixed matrices, strips the _orig_mod. prefix when loading the original compiled-training checkpoint, and requires an exact match to the native model's state. It does not regenerate the trained checkpoint's mixer state, retrain the model, alter its tokenizer, or compile it automatically.

For Hub size indexing, the 16 scalar coupling parameters are stored as one-element tensors in SafeTensors. Their names are recorded in the file metadata, and the loader restores their original scalar shapes before loading the model. This changes only the storage representation, not the model or its values.

To use the original checkpoint instead, download colm_best_Final.pt too and call load_colm(".", weights_format="pt"). See the GitHub repository for full training, generation, and evaluation scripts.

Export Verification

The 8 November 2026 export was checked against the original published checkpoint (SHA-256 bee39cefd9b9426c0993ff9ce8b90296f9316c1d4b180ac0eb8fad32922b1e49). All 296 state tensors—264 learned parameter tensors and 32 fixed buffers—matched in name, shape, dtype and bytes after serialization and strict native loading. model.py and the tokenizer vocabulary are unchanged. Verification used one CPU thread, without initializing CUDA or running generation; this is an exact-state conversion check, not a new model evaluation. Full details are in conversion_report.json.

Training Data

Trained on the DCDM dataset — 47 million tokens of synthetic theological-philosophical prose generated from 93 public domain works through a locally-run Gemma 3 12B pipeline.

The Final training log records 47,044,316 tokens, split into 42,339,884 training and 4,704,432 validation tokens. Its reported validation loss and the GPT-5.4 blind creativity/age ratings above are different measurements; the latter are generated-text judgments described in the paper, not validation-set accuracy.

Limitations

• Spelling: The 499-token vocabulary contains 396 whole-word tokens covering common English and corpus-specific domain words; words outside this vocabulary require character-level assembly, producing spelling variation on out-of-vocabulary terms
• Single trained model: The released checkpoint has only generated text in the DCDM theological-philosophical register; cross-domain output from the trained model is untested. The data generation pipeline has been validated across approximately 894,000 tokens of private source material spanning archaeology, theology, mythology, philosophy, political history, intelligence studies, science fiction, and AI research.
• Batch size: Final run used batch_size=4 rather than intended 32 — results are a lower bound

Citation

@misc{obrien2026colm,
  author = {O'Brien, Phillip C.},
  title = {COLM: Complex Oscillating Language Model — Coherent Language from Sub-500k Parameter Oscillatory Models},
  year = {2026},
  publisher = {Zenodo},
  url = {https://github.com/Eden-Eldith/COLM}
}

Licence

MIT License. Copyright (c) 2025-2026 Phillip C. O'Brien.