classone-gemma4-e2b / README.md
devops-thiago's picture
v1.1: repair release — fix packaging, Stage 2 readout
2ba24ff verified
|
Raw History Blame Contribute Delete
13 kB
metadata
license: apache-2.0
base_model: google/gemma-4-e2b-it
library_name: transformers
tags:
  - decision-model
  - system-1
  - rlcd
  - proper-scoring-rules
  - gemma4
  - classone
  - classification
  - lm-letter-readout
pipeline_tag: text-classification

devops-thiago/classone-gemma4-e2b — v1.1 Repair Release

No New Training / No Tuning: v1.1 pairs the unmodified base google/gemma-4-e2b-it (@3e22461f65e89153144f8adb70e3b8c2cc9845a7) BF16 text and Per-Layer Embedding (PLE) weights (Gemma4TextModel, vision and audio towers stripped) with the unchanged v1.0 decision heads (classone_heads.pt, SHA256: F6A6E42365158D99807B8995C800429A0BD2B12755EDC1DEC5823400D35F337D) and Stage 2 single-token LM-letter readout (M2 K >= 64, top_k = 16). Zero new training, LoRA adaptation, or hyperparameter tuning were performed.

Self-Measured Evaluation Provenance: Executed locally on 2x NVIDIA GeForce RTX 5060 Ti 16 GB · Date: 2026-11-21 · Repository Commit: d615e28 (feat/stage2-lm-letter-readout). See Official Results (Benchmark Heaven) for third-party sealed evaluation context.


Packaging Bug in v1.0 & Repair in v1.1

What Was Wrong in v1.0 (revision="v1.0", Commit 77178b3b)

The v1.0 upload of devops-thiago/classone-gemma4-e2b (tagged v1.0) had a packaging bug: its Hub safetensors shipped with [262144, 1536] token embeddings and [262144, 8960] PLE tables, omitting trained embedding rows 262144..262155 for custom Stage 1 delimiter tokens (<|state_start|>..). Consequently, v1.0 model-card numbers (e.g., JevBench 69.3%) did not reproduce from the Hub weights: evaluating the published v1.0 Hub checkpoint under SDK v0.2.0+ (<unused0>..<unused12> mapping) yields 57.58% on JevBench Public (231), 13.56 Official DI on Decision Index 0.3 (sample-0.3-unseen), and 11.87 Official DI on sample-0.3-unseen-v2.

How v1.1 Fixes It (revision="v1.1")

  1. Unmodified Base Text + PLE Backbone (Gemma4TextModel): Strips unused vision_tower, audio_tower, embed_vision, and embed_audio from google/gemma-4-e2b-it (@3e22461f), reducing disk footprint to 8.663 GiB (9.302 GB) and loaded GPU VRAM to 8.644 GiB with 0.0 hidden-state drift.
  2. FP32 Numerics Preservation (modeling_classone.py): Keeps Gemma 4 RoPE rotary_emb.*_inv_freq buffers and the unchanged v1.0 decision heads (noul_head, choice_head, score_head) in torch.float32 at inference runtime, eliminating BF16 RoPE truncation drift.
  3. Stage 2 LM-Letter Readout + M2 Shortlist (stage2_readout.py): Scores categorical options through a deterministic 180-entry single-token round-trip codebook (gemma4_e2b_codebook_v1.json, SHA256: 692C78FBF001671D0C3D9E712594280F7300F3416615BF36D8F7D8EB79C7DCAE) with final_logit_softcapping = 30.0, M2 top-16 second-pass rerouting when K >= 64, and FP32 heads for Noul and Score.

Self-Measured Benchmark Results (v1.1 Repair vs. v1.0 Hub Weights)

All metrics below are self-measured locally on 2x NVIDIA GeForce RTX 5060 Ti 16 GB (Date: 2026-11-21, Commit: d615e28). 95% confidence intervals and paired (v1.1 - v1.0) bootstrap differences use B = 2,000 stratified resamples (seed = 20261120).

1. Public Benchmark Suites (95% Bootstrap CIs)

Public Benchmark / Suite v1.1 Repair (95% CI) v1.0 Hub Re-Evaluated (95% CI) Paired v1.1 - v1.0 (95% CI)
JevBench v1.6.1 Public (231 tasks) 57.58% [51.52, 63.20] 57.58% [52.38, 63.20] +0.01% [-5.19, +5.21]
RLCDAlignBench Balanced Accuracy (100 instances) 60.68% [51.54, 69.75] 47.90% [37.94, 57.94] +12.80% [-1.06, +25.85]
OOS-320 Macro (AG News / BoolQ / Emotion / SciQ) 77.19% [72.50, 81.25] 69.38% [64.69, 74.06] +7.71% [+3.12, +12.20]
OOS-1000 Macro (AG News / BoolQ / Emotion / SciQ) 77.00% [74.50, 79.40] 69.70% [67.20, 72.20] +7.29% [+4.80, +9.80]
ARC+OBQA Macro (1,667 tasks: ARC-Challenge + OpenBookQA) 72.13% [69.79, 74.56] 50.49% [47.97, 53.14] +21.67% [+18.87, +24.41]
  • JevBench v1.6.1 Public Tier Breakdown (231 tasks): Easy 87.50% (42/48, [77.08, 95.83]), Original 63.89% (46/72, [52.78, 75.00]), Hard 40.54% (45/111, [32.41, 49.55]) vs. v1.0 Hub re-evaluated (Easy 91.67%, Original 72.22%, Hard 33.33%). Note that the historical v1.0 card number (69.3%) could not be reproduced from the packaged v1.0 Hub weights due to the missing rows 262144..262155.

2. Decision Index 0.3 — Local Measurement (Local Scorer, 95% Bootstrap CIs)

Local Scorer Disclosure: The Decision Index 0.3 figures below are local self-measurements reported by our local Decision Index 0.3 scorer on 2x NVIDIA GeForce RTX 5060 Ti 16 GB (Date: 2026-11-21, Commit: d615e28), not official third-party submissions.

Decision Index 0.3 Suite / Metric (Local Measurement) v1.1 Repair (95% CI) v1.0 Hub Re-Evaluated (95% CI) Paired v1.1 - v1.0 (95% CI)
Decision Index 0.3 Official (sample-0.3-unseen-v2) 28.38 [25.68, 31.44] 11.87 [10.31, 15.18] +15.86 [+12.64, +18.91]
Decision Index 0.3 Disjoint-36 (sample-0.3-unseen-v2) 26.48 [23.71, 30.30] 10.50 [8.95, 15.21] +15.21 [+11.43, +18.68]
Decision Index 0.3 Official (sample-0.3-unseen) 27.22 [25.45, 30.92] 13.56 [11.80, 16.47] +14.09 [+11.20, +17.00]
Decision Index 0.3 Disjoint-36 (sample-0.3-unseen) 25.79 [23.93, 30.43] 13.76 [11.81, 17.25] +12.57 [+9.60, +15.75]
Decision Index 0.3 Official (sample-0.3) 25.67 [23.11, 28.41] 15.95 [13.81, 18.82] +9.46 [+6.47, +12.46]
Decision Index 0.3 Disjoint-36 (sample-0.3) 24.34 [21.70, 27.43] 17.63 [14.82, 20.77] +6.72 [+3.50, +10.11]
#62 When2Call (nvidia/When2Call, sample-0.3-unseen) 39.56 [25.33, 55.56] 4.00 [0.00, 18.23] +34.42 [+17.77, +51.54]
  • Private Gating Suite Note: Release gating also confirmed paired-bootstrap improvement (95% lower bound > 0) on the internal Held-Out-1K v2 suite (private evaluation set; data and detailed scores not published).
  • Standalone Export Verification (2x NVIDIA GeForce RTX 5060 Ti 16 GB, single cuda:0):
    • 50-Request Server /v1/decide E2E (sample-0.3-unseen): 201/201 questions (100.0% agreement, max_prob_diff = 0.0); 8.644 GiB loaded allocated VRAM, 9.671 GiB peak allocated VRAM; 84.94 ms median 1-pass latency (K < 64), 169.05 ms median M2 latency (K >= 64), 163.11 ms overall median.
    • 200-Request sample-0.3-unseen Parity Check: 945/945 questions (100.0% agreement, max_prob_diff = 0.0) and bit-identical local Decision Index 0.3 scores (0.0 diff across all 42 evaluated benchmarks).

Official Results (Benchmark Heaven)

Official third-party leaderboards are hosted at Benchmark Heaven. Self-measured local JevBench numbers (57.58% on 2x NVIDIA RTX 5060 Ti 16 GB) differ from official Benchmark Heaven leaderboard scores because:

  1. Sealed vs. Public Split: Local evaluations run only on the 231 public JevBench tasks, whereas Benchmark Heaven evaluates both public items and private sealed items.
  2. Public/Sealed Generalization Gap Penalty: Benchmark Heaven penalizes divergence between public and sealed accuracy.
  3. 10-Second Cutoff & Server Reference Hardware: Tasks exceeding 10 seconds receive zero credit, and speed/cost metrics are measured on server-grade GPUs rather than local workstation hardware.

Quickstart & SDK Usage (v1.1)

pip install classone

Python SDK (v1.1 Standalone Stage 2 Readout)

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

from classone.modeling.modeling_classone import ClassOneModel
from classone.schemas import ChoiceQuestion, NoulQuestion, ScoreQuestion
from classone.stage2_readout import ChatDecisionPromptBuilder, evaluate_stage2_request

REPO_ID = "devops-thiago/classone-gemma4-e2b"
REVISION = "v1.1"  # or "main"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision=REVISION, trust_remote_code=True)
codebook_path = hf_hub_download(REPO_ID, "gemma4_e2b_codebook_v1.json", revision=REVISION)
chat_builder = ChatDecisionPromptBuilder(tokenizer, codebook_path=codebook_path)

model = ClassOneModel.from_backbone(
    REPO_ID,
    revision=REVISION,
    tokenizer=tokenizer,
    device="cuda:0",
    torch_dtype=torch.bfloat16,
)

heads = torch.load(
    hf_hub_download(REPO_ID, "classone_heads.pt", revision=REVISION),
    map_location="cuda:0",
    weights_only=True,
)
model.noul_head.load_state_dict(heads["noul_head"], strict=False)
model.choice_head.load_state_dict(heads["choice_head"], strict=False)
model.score_head.load_state_dict(heads["score_head"], strict=False)
model.eval()

answers, total_tokens = evaluate_stage2_request(
    model,
    chat_builder,
    state={"customer": "Alex", "message": "I was charged twice for order #123."},
    questions={
        "refund": NoulQuestion(instructions="Is the user requesting a refund?"),
        "dept": ChoiceQuestion(
            instructions="Route to team:",
            criteria={"billing": "Payment issues", "tech": "Technical bugs"},
        ),
        "anger": ScoreQuestion(
            instructions="Dissatisfaction level:",
            criteria=["satisfied", "neutral", "dissatisfied", "churning"],
        ),
    },
    m2_k_threshold=64,
    m2_top_k=16,
)

print("Refund P(true):", answers["refund"].noul)
print("Department:    ", answers["dept"].choice, "—", answers["dept"].probabilities)
print("Anger score:   ", answers["anger"].score)

FastAPI Server (v1.1)

config.json enables Stage 2 (eval_mode = "stage2", m2_k_threshold = 64, m2_top_k = 16, codebook_file = "gemma4_e2b_codebook_v1.json") automatically when served via classone.server.app:

export CLASSONE_BASE_MODEL="devops-thiago/classone-gemma4-e2b"
uvicorn classone.server.app:app --host 0.0.0.0 --port 8000

Pinning v1.0 (revision="v1.0")

To reproduce the original v1.0 upload prior to the v1.1 repair release, pin revision="v1.0":

import torch
from huggingface_hub import hf_hub_download
from transformers import AutoTokenizer

from classone.modeling.modeling_classone import ClassOneModel
from classone.tokenizer import ClassOnePromptBuilder

REPO_ID = "devops-thiago/classone-gemma4-e2b"

tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision="v1.0", trust_remote_code=True)
builder = ClassOnePromptBuilder(tokenizer)
model = ClassOneModel.from_backbone(
    REPO_ID,
    revision="v1.0",
    tokenizer=tokenizer,
    device="cuda:0",
    torch_dtype=torch.bfloat16,
)
heads = torch.load(
    hf_hub_download(REPO_ID, "classone_heads.pt", revision="v1.0"),
    map_location="cuda:0",
    weights_only=True,
)
model.noul_head.load_state_dict(heads["noul_head"], strict=False)
model.choice_head.load_state_dict(heads["choice_head"], strict=False)
model.score_head.load_state_dict(heads["score_head"], strict=False)
model.eval()

Or when launching the FastAPI server:

export CLASSONE_BASE_MODEL="devops-thiago/classone-gemma4-e2b"
export CLASSONE_REVISION="v1.0"

Repository Files (v1.1)

File Description
model-00001-of-00002.safetensors, model-00002-of-00002.safetensors, model.safetensors.index.json Unmodified BF16 Gemma4TextModel text + PLE weights (google/gemma-4-e2b-it @3e22461f, multimodal towers stripped)
config.json Text/PLE configuration (final_logit_softcapping = 30.0) + classone_config (eval_mode = "stage2", m2_k_threshold = 64, m2_top_k = 16, codebook_file = "gemma4_e2b_codebook_v1.json")
tokenizer.json, tokenizer_config.json, chat_template.jinja Gemma 4 tokenizer (vocab_size = 262,144) with ClassOne delimiter mapping (<unused0>..)
gemma4_e2b_codebook_v1.json Versioned 180-entry single-token round-trip codebook (SHA256: 692C78FB...)
classone_heads.pt Unchanged v1.0 calibrated Noul / Choice / Score heads (SHA256: F6A6E423...)

Attribution & Legal