classone-qwen3.5-4b β€” ClassOne System 1 Decision Model

devops-thiago/classone-qwen3.5-4b is an open-source System 1 decision model built on Qwen/Qwen3.5-4B using the ClassOne (Classifier One) architecture. The merged standalone backbone ships directly in this repository β€” it loads as a single model, with no adapter and no separate base-model download.

Instead of generating text token by token, ClassOne evaluates structured decisions in a single forward pass, returning typed, calibrated outputs with zero decoding overhead.

Self-Measured Benchmark Results

Self-Measured Evaluation Provenance: Executed locally on 2x NVIDIA RTX 5060 Ti 16 GB (Date: 2026-11-13, Commit: c375207). See Official Results (Benchmark Heaven) below for third-party sealed evaluation.

1. JevBench Public Multi-Tier Suite (231 Public Tasks, Self-Measured Standalone BF16)

Evaluated standalone (--bf16 --no-lora --pause-tokens 0) across all 231 public tasks in fstandhartinger/jevbench (heads SHA256 654DA038...).

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB Β· Date: 2026-11-13 Β· Commit: c375207

Tier Tasks Accuracy ECE Brier Score Median Latency (p50)
Easy 48 100.0% (48/48) 0.0266 0.0093 150.8 ms
Original 72 95.8% (69/72) 0.0385 0.0342 157.1 ms
Hard 111 52.3% (58/111) 0.3177 0.3451 475.6 ms
Overall Aggregate (Public) 231 75.8% (175/231) β€” β€” 157.1 ms
  • Easy Tier Sub-Breakdown: Choice: 100.0% (36/36); Noul: 100.0% (12/12).
  • Original Tier Sub-Breakdown: Noul: 100.0% (24/24); Score: 100.0% (12/12); Choice: 91.7% (33/36).
  • Hard Tier Sub-Breakdown: Noul: 55.3% (21/38); Choice: 52.2% (35/67); Score: 33.3% (2/6).

2. Decision Index Suite (Self-Run, 50 Tasks)

Self-run local evaluation on multimodalart/jev-decision-index (--quant none).

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB Β· Date: 2026-11-13 Β· Commit: c375207

Domain Weight Accuracy (%) Score
Knowledge & Verification 25.8% 100.0% (10/10) 100.00 pts
Language & Policy Contracts 25.8% 100.0% (10/10) 100.00 pts
Retrieval & Relevance Triage 20.0% 100.0% (10/10) 100.00 pts
Tools & Execution Guardrails 18.3% 100.0% (10/10) 100.00 pts
Rubric & Quality Scoring 10.1% 90.0% (9/10) 90.00 pts
Overall Weighted Score 100.0% 98.0% (49/50) 98.99 / 100.00 (0.0333 ECE, 178.7 ms p50)

3. RLCDAlignBench Alignment & Safety Suite (100 Instances, Self-Measured Standalone BF16)

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB Β· Date: 2026-11-13 Β· Commit: c375207

Failure Mode / Axis Samples (N) AUROC Accuracy (%) ECE Latency (p50)
Power Seeking 6 0.889 66.7% 0.2187 1036.6 ms
Faithfulness 9 0.900 77.8% 0.2913 1093.4 ms
Honesty (Deception) 11 0.833 63.6% 0.2473 1109.4 ms
Refusal (Jailbreaks) 11 0.600 45.5% 0.3237 1373.8 ms
Overall Balanced Accuracy 100 0.572 51.5% 0.3385 1051.6 ms

4. Tool-Routing & Safety Guardrail Suites (Self-Measured Standalone)

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB Β· Date: 2026-11-13 Β· Commit: c375207

Benchmark Suite Split / Samples Accuracy Secondary Metric Calibration ECE Latency (p50)
WildGuardTest (walledai/WildGuardTest) test (100 stratified, 4-bit) 89.00% (88.94% BalAcc) 0.9335 AUROC (91.07% Vanilla, 86.36% Adversarial) 0.0840 231.9 ms
Deepset Prompt Injections (deepset/prompt-injections) test (116 full, BF16) 57.76% (67/116) 0.7619 AUROC (100.0% Benign Spec, 18.3% Recall) 0.3825 141.1 ms
NVIDIA When2Call (nvidia/When2Call) mcq (100 stratified, BF16) 35.00% 88.2% tool_call (30/34), 11.9% cannot_answer 0.4188 546.4 ms

Official Results (Benchmark Heaven)

Official third-party JevBench leaderboards are hosted at Benchmark Heaven (checked 2026-11-13, JevBench v1.8.10 / method v1.7, published 2026-11-12; see the live entry for classone-qwen3.5-9b). Official Benchmark Heaven scores differ from local self-measurements because the official harness evaluates private sealed items in addition to the public suite, applies a public/sealed generalization gap penalty, enforces a strict 10-second cutoff rule, and measures latency/cost on server-grade GPUs rather than local workstation hardware (2x NVIDIA RTX 5060 Ti 16 GB).

Edge vs Cloud Latency (Self-Measured Comparison)

Self-measured on 2x NVIDIA RTX 5060 Ti 16 GB Β· Date: 2026-11-13 Β· Commit: c375207

Measured against a remote cloud LLM decision API round-trip:

  • ClassOne (Local RTX 5060 Ti): 52.49 ms mean latency (19.1 req/s, $0.00 inference cost, 100% private)
  • Cloud API Baseline: 329.90 ms mean latency (3.0 req/s)
  • Edge Speedup: 6.3Γ— faster than cloud API round-trip latency

Decision Primitives

  • Noul β€” Boolean check returning a calibrated probability P(true) ∈ [0, 1]
  • Choice β€” Categorical selection over 2–255 dynamic options with full probability distribution
  • Score β€” Continuous ordinal rubric rating over 2–10 levels (expected value)

All outputs are calibrated with a combined focal NLL (Ξ³=2.0) + normalized Brier + contrastive margin loss and post-hoc temperature scaling.

Quickstart

pip install classone
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

from classone.modeling.modeling_classone import ClassOneModel
from classone.schemas import NoulQuestion, ChoiceQuestion, ScoreQuestion
from classone.tokenizer import ClassOnePromptBuilder

REPO_ID = "devops-thiago/classone-qwen3.5-4b"

# 1. Load the ClassOne model (weights + tokenizer are fully self-contained here)
tokenizer = AutoTokenizer.from_pretrained(REPO_ID)
builder = ClassOnePromptBuilder(tokenizer)
model = ClassOneModel.from_backbone(
    base_model_name_or_path=REPO_ID,
    tokenizer=tokenizer,
    device="cuda",
    torch_dtype=torch.bfloat16,
)

# 2. Load the trained decision heads
heads = torch.load(hf_hub_download(REPO_ID, "classone_heads.pt"), map_location="cuda")
model.noul_head.load_state_dict(heads["noul_head"], strict=False)
model.choice_head.load_state_dict(heads["choice_head"], strict=False)
model.score_head.load_state_dict(heads["score_head"], strict=False)
model.eval()

# 3. Pack state + questions and run a single forward pass
packed = builder.pack(
    state={"customer": "Alex", "message": "I was charged twice for order #123."},
    questions={
        "refund": NoulQuestion(instructions="Is the user requesting a refund?"),
        "dept":   ChoiceQuestion(
                      instructions="Route to team:",
                      criteria={"billing": "Payment issues", "tech": "Technical bugs"}
                  ),
        "anger":  ScoreQuestion(
                      instructions="Dissatisfaction level:",
                      criteria=["satisfied", "neutral", "dissatisfied", "churning"]
                  ),
    }
)
results = model.evaluate_packed(packed)

print("Refund P(true):", results["refund"].noul)
print("Department:    ", results["dept"].choice, "β€”", results["dept"].probabilities)
print("Anger score:   ", results["anger"].score)

Repository Files

File Description
model.safetensors (or sharded) Merged ClassOne standalone backbone weights
config.json Model configuration
tokenizer.json, tokenizer_config.json Tokenizer with ClassOne delimiter tokens
classone_heads.pt Trained Noul / Choice / Score head weights + calibrated temperatures

Certified classone_heads.pt SHA256: 654DA038EA5883916F3323F1A5BE64DEEFADB864B2EAE48DF9CD915D9C8A8F66

Citation

@misc{classone2026,
  title={ClassOne: A Fast Single-Pass Decision Architecture for Language Models},
  author={Thiago Gonzaga},
  year={2026},
  url={https://github.com/devops-thiago/class-one},
}

Attribution & Legal

Downloads last month
290
Safetensors
Model size
5B params
Tensor type
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for devops-thiago/classone-qwen3.5-4b

Finetuned
Qwen/Qwen3.5-4B
Finetuned
(1,479)
this model

Space using devops-thiago/classone-qwen3.5-4b 1

Collection including devops-thiago/classone-qwen3.5-4b