Instructions to use devops-thiago/classone-gemma4-e2b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use devops-thiago/classone-gemma4-e2b with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="devops-thiago/classone-gemma4-e2b")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("devops-thiago/classone-gemma4-e2b") model = AutoModel.from_pretrained("devops-thiago/classone-gemma4-e2b", device_map="auto") - Notebooks
- Google Colab
- Kaggle
Download README.md from devops-thiago/classone-gemma4-e2b: direct link, hf CLI and curl.
- Browser
- Download file 13 kB
-
https://huggingface.co/devops-thiago/classone-gemma4-e2b/resolve/main/README.md
- Command line
-
hf download hf://devops-thiago/classone-gemma4-e2b/README.md
-
curl -L -o README.md https://huggingface.co/devops-thiago/classone-gemma4-e2b/resolve/main/README.md
license: apache-2.0
base_model: google/gemma-4-e2b-it
library_name: transformers
tags:
- decision-model
- system-1
- rlcd
- proper-scoring-rules
- gemma4
- classone
- classification
- lm-letter-readout
pipeline_tag: text-classification
devops-thiago/classone-gemma4-e2b — v1.1 Repair Release
No New Training / No Tuning:
v1.1pairs the unmodified basegoogle/gemma-4-e2b-it(@3e22461f65e89153144f8adb70e3b8c2cc9845a7) BF16 text and Per-Layer Embedding (PLE) weights (Gemma4TextModel, vision and audio towers stripped) with the unchangedv1.0decision heads (classone_heads.pt,SHA256: F6A6E42365158D99807B8995C800429A0BD2B12755EDC1DEC5823400D35F337D) and Stage 2 single-token LM-letter readout (M2 K >= 64, top_k = 16). Zero new training, LoRA adaptation, or hyperparameter tuning were performed.Self-Measured Evaluation Provenance: Executed locally on
2x NVIDIA GeForce RTX 5060 Ti 16 GB· Date:2026-11-21· Repository Commit:d615e28(feat/stage2-lm-letter-readout). See Official Results (Benchmark Heaven) for third-party sealed evaluation context.
Packaging Bug in v1.0 & Repair in v1.1
What Was Wrong in v1.0 (revision="v1.0", Commit 77178b3b)
The v1.0 upload of devops-thiago/classone-gemma4-e2b (tagged v1.0) had a packaging bug: its Hub safetensors shipped with [262144, 1536] token embeddings and [262144, 8960] PLE tables, omitting trained embedding rows 262144..262155 for custom Stage 1 delimiter tokens (<|state_start|>..). Consequently, v1.0 model-card numbers (e.g., JevBench 69.3%) did not reproduce from the Hub weights: evaluating the published v1.0 Hub checkpoint under SDK v0.2.0+ (<unused0>..<unused12> mapping) yields 57.58% on JevBench Public (231), 13.56 Official DI on Decision Index 0.3 (sample-0.3-unseen), and 11.87 Official DI on sample-0.3-unseen-v2.
How v1.1 Fixes It (revision="v1.1")
- Unmodified Base Text + PLE Backbone (
Gemma4TextModel): Strips unusedvision_tower,audio_tower,embed_vision, andembed_audiofromgoogle/gemma-4-e2b-it(@3e22461f), reducing disk footprint to8.663 GiB(9.302 GB) and loaded GPU VRAM to8.644 GiBwith0.0hidden-state drift. - FP32 Numerics Preservation (
modeling_classone.py): Keeps Gemma 4 RoPErotary_emb.*_inv_freqbuffers and the unchangedv1.0decision heads (noul_head,choice_head,score_head) intorch.float32at inference runtime, eliminating BF16 RoPE truncation drift. - Stage 2 LM-Letter Readout +
M2Shortlist (stage2_readout.py): Scores categorical options through a deterministic180-entry single-token round-trip codebook (gemma4_e2b_codebook_v1.json,SHA256: 692C78FBF001671D0C3D9E712594280F7300F3416615BF36D8F7D8EB79C7DCAE) withfinal_logit_softcapping = 30.0,M2top-16 second-pass rerouting whenK >= 64, and FP32 heads forNoulandScore.
Self-Measured Benchmark Results (v1.1 Repair vs. v1.0 Hub Weights)
All metrics below are self-measured locally on 2x NVIDIA GeForce RTX 5060 Ti 16 GB (Date: 2026-11-21, Commit: d615e28). 95% confidence intervals and paired (v1.1 - v1.0) bootstrap differences use B = 2,000 stratified resamples (seed = 20261120).
1. Public Benchmark Suites (95% Bootstrap CIs)
| Public Benchmark / Suite | v1.1 Repair (95% CI) |
v1.0 Hub Re-Evaluated (95% CI) |
Paired v1.1 - v1.0 (95% CI) |
|---|---|---|---|
JevBench v1.6.1 Public (231 tasks) |
57.58% [51.52, 63.20] |
57.58% [52.38, 63.20] |
+0.01% [-5.19, +5.21] |
RLCDAlignBench Balanced Accuracy (100 instances) |
60.68% [51.54, 69.75] |
47.90% [37.94, 57.94] |
+12.80% [-1.06, +25.85] |
OOS-320 Macro (AG News / BoolQ / Emotion / SciQ) |
77.19% [72.50, 81.25] |
69.38% [64.69, 74.06] |
+7.71% [+3.12, +12.20] |
OOS-1000 Macro (AG News / BoolQ / Emotion / SciQ) |
77.00% [74.50, 79.40] |
69.70% [67.20, 72.20] |
+7.29% [+4.80, +9.80] |
ARC+OBQA Macro (1,667 tasks: ARC-Challenge + OpenBookQA) |
72.13% [69.79, 74.56] |
50.49% [47.97, 53.14] |
+21.67% [+18.87, +24.41] |
JevBench v1.6.1Public Tier Breakdown (231tasks):Easy87.50%(42/48,[77.08, 95.83]),Original63.89%(46/72,[52.78, 75.00]),Hard40.54%(45/111,[32.41, 49.55]) vs.v1.0Hub re-evaluated (Easy91.67%,Original72.22%,Hard33.33%). Note that the historicalv1.0card number (69.3%) could not be reproduced from the packagedv1.0Hub weights due to the missing rows262144..262155.
2. Decision Index 0.3 — Local Measurement (Local Scorer, 95% Bootstrap CIs)
Local Scorer Disclosure: The Decision Index 0.3 figures below are local self-measurements reported by our local Decision Index 0.3 scorer on
2x NVIDIA GeForce RTX 5060 Ti 16 GB(Date:2026-11-21, Commit:d615e28), not official third-party submissions.
| Decision Index 0.3 Suite / Metric (Local Measurement) | v1.1 Repair (95% CI) |
v1.0 Hub Re-Evaluated (95% CI) |
Paired v1.1 - v1.0 (95% CI) |
|---|---|---|---|
Decision Index 0.3 Official (sample-0.3-unseen-v2) |
28.38 [25.68, 31.44] |
11.87 [10.31, 15.18] |
+15.86 [+12.64, +18.91] |
Decision Index 0.3 Disjoint-36 (sample-0.3-unseen-v2) |
26.48 [23.71, 30.30] |
10.50 [8.95, 15.21] |
+15.21 [+11.43, +18.68] |
Decision Index 0.3 Official (sample-0.3-unseen) |
27.22 [25.45, 30.92] |
13.56 [11.80, 16.47] |
+14.09 [+11.20, +17.00] |
Decision Index 0.3 Disjoint-36 (sample-0.3-unseen) |
25.79 [23.93, 30.43] |
13.76 [11.81, 17.25] |
+12.57 [+9.60, +15.75] |
Decision Index 0.3 Official (sample-0.3) |
25.67 [23.11, 28.41] |
15.95 [13.81, 18.82] |
+9.46 [+6.47, +12.46] |
Decision Index 0.3 Disjoint-36 (sample-0.3) |
24.34 [21.70, 27.43] |
17.63 [14.82, 20.77] |
+6.72 [+3.50, +10.11] |
#62 When2Call (nvidia/When2Call, sample-0.3-unseen) |
39.56 [25.33, 55.56] |
4.00 [0.00, 18.23] |
+34.42 [+17.77, +51.54] |
- Private Gating Suite Note: Release gating also confirmed paired-bootstrap improvement (
95%lower bound> 0) on the internalHeld-Out-1K v2suite (private evaluation set; data and detailed scores not published). - Standalone Export Verification (
2x NVIDIA GeForce RTX 5060 Ti 16 GB, singlecuda:0):- 50-Request Server
/v1/decideE2E (sample-0.3-unseen):201/201questions (100.0%agreement,max_prob_diff = 0.0);8.644 GiBloaded allocated VRAM,9.671 GiBpeak allocated VRAM;84.94 msmedian 1-pass latency (K < 64),169.05 msmedian M2 latency (K >= 64),163.11 msoverall median. - 200-Request
sample-0.3-unseenParity Check:945/945questions (100.0%agreement,max_prob_diff = 0.0) and bit-identical local Decision Index 0.3 scores (0.0diff across all42evaluated benchmarks).
- 50-Request Server
Official Results (Benchmark Heaven)
Official third-party leaderboards are hosted at Benchmark Heaven. Self-measured local JevBench numbers (57.58% on 2x NVIDIA RTX 5060 Ti 16 GB) differ from official Benchmark Heaven leaderboard scores because:
- Sealed vs. Public Split: Local evaluations run only on the 231 public JevBench tasks, whereas Benchmark Heaven evaluates both public items and private sealed items.
- Public/Sealed Generalization Gap Penalty: Benchmark Heaven penalizes divergence between public and sealed accuracy.
- 10-Second Cutoff & Server Reference Hardware: Tasks exceeding 10 seconds receive zero credit, and speed/cost metrics are measured on server-grade GPUs rather than local workstation hardware.
Quickstart & SDK Usage (v1.1)
pip install classone
Python SDK (v1.1 Standalone Stage 2 Readout)
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
from classone.modeling.modeling_classone import ClassOneModel
from classone.schemas import ChoiceQuestion, NoulQuestion, ScoreQuestion
from classone.stage2_readout import ChatDecisionPromptBuilder, evaluate_stage2_request
REPO_ID = "devops-thiago/classone-gemma4-e2b"
REVISION = "v1.1" # or "main"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision=REVISION, trust_remote_code=True)
codebook_path = hf_hub_download(REPO_ID, "gemma4_e2b_codebook_v1.json", revision=REVISION)
chat_builder = ChatDecisionPromptBuilder(tokenizer, codebook_path=codebook_path)
model = ClassOneModel.from_backbone(
REPO_ID,
revision=REVISION,
tokenizer=tokenizer,
device="cuda:0",
torch_dtype=torch.bfloat16,
)
heads = torch.load(
hf_hub_download(REPO_ID, "classone_heads.pt", revision=REVISION),
map_location="cuda:0",
weights_only=True,
)
model.noul_head.load_state_dict(heads["noul_head"], strict=False)
model.choice_head.load_state_dict(heads["choice_head"], strict=False)
model.score_head.load_state_dict(heads["score_head"], strict=False)
model.eval()
answers, total_tokens = evaluate_stage2_request(
model,
chat_builder,
state={"customer": "Alex", "message": "I was charged twice for order #123."},
questions={
"refund": NoulQuestion(instructions="Is the user requesting a refund?"),
"dept": ChoiceQuestion(
instructions="Route to team:",
criteria={"billing": "Payment issues", "tech": "Technical bugs"},
),
"anger": ScoreQuestion(
instructions="Dissatisfaction level:",
criteria=["satisfied", "neutral", "dissatisfied", "churning"],
),
},
m2_k_threshold=64,
m2_top_k=16,
)
print("Refund P(true):", answers["refund"].noul)
print("Department: ", answers["dept"].choice, "—", answers["dept"].probabilities)
print("Anger score: ", answers["anger"].score)
FastAPI Server (v1.1)
config.json enables Stage 2 (eval_mode = "stage2", m2_k_threshold = 64, m2_top_k = 16, codebook_file = "gemma4_e2b_codebook_v1.json") automatically when served via classone.server.app:
export CLASSONE_BASE_MODEL="devops-thiago/classone-gemma4-e2b"
uvicorn classone.server.app:app --host 0.0.0.0 --port 8000
Pinning v1.0 (revision="v1.0")
To reproduce the original v1.0 upload prior to the v1.1 repair release, pin revision="v1.0":
import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer
from classone.modeling.modeling_classone import ClassOneModel
from classone.tokenizer import ClassOnePromptBuilder
REPO_ID = "devops-thiago/classone-gemma4-e2b"
tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision="v1.0", trust_remote_code=True)
builder = ClassOnePromptBuilder(tokenizer)
model = ClassOneModel.from_backbone(
REPO_ID,
revision="v1.0",
tokenizer=tokenizer,
device="cuda:0",
torch_dtype=torch.bfloat16,
)
heads = torch.load(
hf_hub_download(REPO_ID, "classone_heads.pt", revision="v1.0"),
map_location="cuda:0",
weights_only=True,
)
model.noul_head.load_state_dict(heads["noul_head"], strict=False)
model.choice_head.load_state_dict(heads["choice_head"], strict=False)
model.score_head.load_state_dict(heads["score_head"], strict=False)
model.eval()
Or when launching the FastAPI server:
export CLASSONE_BASE_MODEL="devops-thiago/classone-gemma4-e2b"
export CLASSONE_REVISION="v1.0"
Repository Files (v1.1)
| File | Description |
|---|---|
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json |
Unmodified BF16 Gemma4TextModel text + PLE weights (google/gemma-4-e2b-it @3e22461f, multimodal towers stripped) |
config.json |
Text/PLE configuration (final_logit_softcapping = 30.0) + classone_config (eval_mode = "stage2", m2_k_threshold = 64, m2_top_k = 16, codebook_file = "gemma4_e2b_codebook_v1.json") |
tokenizer.json, tokenizer_config.json, chat_template.jinja |
Gemma 4 tokenizer (vocab_size = 262,144) with ClassOne delimiter mapping (<unused0>..) |
gemma4_e2b_codebook_v1.json |
Versioned 180-entry single-token round-trip codebook (SHA256: 692C78FB...) |
classone_heads.pt |
Unchanged v1.0 calibrated Noul / Choice / Score heads (SHA256: F6A6E423...) |
Attribution & Legal
- Base backbone:
google/gemma-4-e2b-it(Google) — Apache License 2.0 - Architecture & evaluation code:
devops-thiago/class-one— Apache License 2.0