--- license: apache-2.0 base_model: google/gemma-4-e2b-it library_name: transformers tags: - decision-model - system-1 - rlcd - proper-scoring-rules - gemma4 - classone - classification - lm-letter-readout pipeline_tag: text-classification --- # `devops-thiago/classone-gemma4-e2b` — `v1.1` Repair Release > **No New Training / No Tuning:** **`v1.1`** pairs the unmodified base [`google/gemma-4-e2b-it`](https://huggingface.co/google/gemma-4-e2b-it) (`@3e22461f65e89153144f8adb70e3b8c2cc9845a7`) BF16 text and Per-Layer Embedding (PLE) weights (`Gemma4TextModel`, vision and audio towers stripped) with the **unchanged `v1.0` decision heads** (`classone_heads.pt`, `SHA256: F6A6E42365158D99807B8995C800429A0BD2B12755EDC1DEC5823400D35F337D`) and **Stage 2 single-token LM-letter readout** (`M2 K >= 64, top_k = 16`). Zero new training, LoRA adaptation, or hyperparameter tuning were performed. > > **Self-Measured Evaluation Provenance:** Executed locally on `2x NVIDIA GeForce RTX 5060 Ti 16 GB` · Date: `2026-11-21` · Repository Commit: `d615e28` ([`feat/stage2-lm-letter-readout`](https://github.com/devops-thiago/class-one)). See [Official Results (Benchmark Heaven)](#official-results-benchmark-heaven) for third-party sealed evaluation context. --- ## Packaging Bug in `v1.0` & Repair in `v1.1` ### What Was Wrong in `v1.0` (`revision="v1.0"`, Commit `77178b3b`) The `v1.0` upload of `devops-thiago/classone-gemma4-e2b` (tagged **`v1.0`**) had a packaging bug: its Hub safetensors shipped with `[262144, 1536]` token embeddings and `[262144, 8960]` PLE tables, omitting trained embedding rows `262144..262155` for custom Stage 1 delimiter tokens (`<|state_start|>..`). Consequently, **`v1.0` model-card numbers (e.g., JevBench `69.3%`) did not reproduce from the Hub weights**: evaluating the published `v1.0` Hub checkpoint under SDK `v0.2.0+` (`..` mapping) yields `57.58%` on JevBench Public (`231`), `13.56` `Official DI` on `Decision Index 0.3` (`sample-0.3-unseen`), and `11.87` `Official DI` on `sample-0.3-unseen-v2`. ### How `v1.1` Fixes It (`revision="v1.1"`) 1. **Unmodified Base Text + PLE Backbone (`Gemma4TextModel`):** Strips unused `vision_tower`, `audio_tower`, `embed_vision`, and `embed_audio` from `google/gemma-4-e2b-it` (`@3e22461f`), reducing disk footprint to `8.663 GiB` (`9.302 GB`) and loaded GPU VRAM to `8.644 GiB` with `0.0` hidden-state drift. 2. **FP32 Numerics Preservation (`modeling_classone.py`):** Keeps Gemma 4 RoPE `rotary_emb.*_inv_freq` buffers and the unchanged `v1.0` decision heads (`noul_head`, `choice_head`, `score_head`) in `torch.float32` at inference runtime, eliminating BF16 RoPE truncation drift. 3. **Stage 2 LM-Letter Readout + `M2` Shortlist (`stage2_readout.py`):** Scores categorical options through a deterministic `180`-entry single-token round-trip codebook (`gemma4_e2b_codebook_v1.json`, `SHA256: 692C78FBF001671D0C3D9E712594280F7300F3416615BF36D8F7D8EB79C7DCAE`) with `final_logit_softcapping = 30.0`, `M2` top-16 second-pass rerouting when `K >= 64`, and FP32 heads for `Noul` and `Score`. --- ## Self-Measured Benchmark Results (`v1.1` Repair vs. `v1.0` Hub Weights) *All metrics below are self-measured locally on `2x NVIDIA GeForce RTX 5060 Ti 16 GB` (Date: `2026-11-21`, Commit: `d615e28`). `95%` confidence intervals and paired `(v1.1 - v1.0)` bootstrap differences use `B = 2,000` stratified resamples (`seed = 20261120`).* ### 1. Public Benchmark Suites (95% Bootstrap CIs) | Public Benchmark / Suite | `v1.1` Repair *(95% CI)* | `v1.0` Hub Re-Evaluated *(95% CI)* | Paired `v1.1 - v1.0` *(95% CI)* | | :--- | :---: | :---: | :---: | | **`JevBench v1.6.1` Public (`231` tasks)** | **`57.58%`** `[51.52, 63.20]` | `57.58%` `[52.38, 63.20]` | **`+0.01%`** `[-5.19, +5.21]` | | **`RLCDAlignBench` Balanced Accuracy (`100` instances)** | **`60.68%`** `[51.54, 69.75]` | `47.90%` `[37.94, 57.94]` | **`+12.80%`** `[-1.06, +25.85]` | | **`OOS-320` Macro (`AG News / BoolQ / Emotion / SciQ`)** | **`77.19%`** `[72.50, 81.25]` | `69.38%` `[64.69, 74.06]` | **`+7.71%`** `[+3.12, +12.20]` | | **`OOS-1000` Macro (`AG News / BoolQ / Emotion / SciQ`)** | **`77.00%`** `[74.50, 79.40]` | `69.70%` `[67.20, 72.20]` | **`+7.29%`** `[+4.80, +9.80]` | | **`ARC+OBQA` Macro (`1,667` tasks: `ARC-Challenge` + `OpenBookQA`)** | **`72.13%`** `[69.79, 74.56]` | `50.49%` `[47.97, 53.14]` | **`+21.67%`** `[+18.87, +24.41]` | - **`JevBench v1.6.1` Public Tier Breakdown (`231` tasks):** `Easy` **`87.50%`** (`42/48`, `[77.08, 95.83]`), `Original` **`63.89%`** (`46/72`, `[52.78, 75.00]`), `Hard` **`40.54%`** (`45/111`, `[32.41, 49.55]`) vs. `v1.0` Hub re-evaluated (`Easy` `91.67%`, `Original` `72.22%`, `Hard` `33.33%`). Note that the historical `v1.0` card number (`69.3%`) could not be reproduced from the packaged `v1.0` Hub weights due to the missing rows `262144..262155`. ### 2. Decision Index 0.3 — Local Measurement (Local Scorer, 95% Bootstrap CIs) > **Local Scorer Disclosure:** The Decision Index 0.3 figures below are **local self-measurements reported by our local Decision Index 0.3 scorer** on `2x NVIDIA GeForce RTX 5060 Ti 16 GB` (Date: `2026-11-21`, Commit: `d615e28`), not official third-party submissions. | Decision Index 0.3 Suite / Metric *(Local Measurement)* | `v1.1` Repair *(95% CI)* | `v1.0` Hub Re-Evaluated *(95% CI)* | Paired `v1.1 - v1.0` *(95% CI)* | | :--- | :---: | :---: | :---: | | **`Decision Index 0.3` `Official` (`sample-0.3-unseen-v2`)** | **`28.38`** `[25.68, 31.44]` | `11.87` `[10.31, 15.18]` | **`+15.86`** `[+12.64, +18.91]` | | **`Decision Index 0.3` `Disjoint-36` (`sample-0.3-unseen-v2`)** | **`26.48`** `[23.71, 30.30]` | `10.50` `[8.95, 15.21]` | **`+15.21`** `[+11.43, +18.68]` | | **`Decision Index 0.3` `Official` (`sample-0.3-unseen`)** | **`27.22`** `[25.45, 30.92]` | `13.56` `[11.80, 16.47]` | **`+14.09`** `[+11.20, +17.00]` | | **`Decision Index 0.3` `Disjoint-36` (`sample-0.3-unseen`)** | **`25.79`** `[23.93, 30.43]` | `13.76` `[11.81, 17.25]` | **`+12.57`** `[+9.60, +15.75]` | | **`Decision Index 0.3` `Official` (`sample-0.3`)** | **`25.67`** `[23.11, 28.41]` | `15.95` `[13.81, 18.82]` | **`+9.46`** `[+6.47, +12.46]` | | **`Decision Index 0.3` `Disjoint-36` (`sample-0.3`)** | **`24.34`** `[21.70, 27.43]` | `17.63` `[14.82, 20.77]` | **`+6.72`** `[+3.50, +10.11]` | | **`#62` When2Call (`nvidia/When2Call`, `sample-0.3-unseen`)** | **`39.56`** `[25.33, 55.56]` | `4.00` `[0.00, 18.23]` | **`+34.42`** `[+17.77, +51.54]` | - **Private Gating Suite Note:** Release gating also confirmed paired-bootstrap improvement (`95%` lower bound `> 0`) on the internal `Held-Out-1K v2` suite (private evaluation set; data and detailed scores not published). - **Standalone Export Verification (`2x NVIDIA GeForce RTX 5060 Ti 16 GB`, single `cuda:0`):** - **50-Request Server `/v1/decide` E2E (`sample-0.3-unseen`):** `201/201` questions (`100.0%` agreement, `max_prob_diff = 0.0`); `8.644 GiB` loaded allocated VRAM, `9.671 GiB` peak allocated VRAM; `84.94 ms` median 1-pass latency (`K < 64`), `169.05 ms` median M2 latency (`K >= 64`), `163.11 ms` overall median. - **200-Request `sample-0.3-unseen` Parity Check:** `945/945` questions (`100.0%` agreement, `max_prob_diff = 0.0`) and bit-identical local Decision Index 0.3 scores (`0.0` diff across all `42` evaluated benchmarks). --- ## Official Results (Benchmark Heaven) Official third-party leaderboards are hosted at **[Benchmark Heaven](https://benchmarkheaven.com/jev-models)**. Self-measured local JevBench numbers (`57.58%` on `2x NVIDIA RTX 5060 Ti 16 GB`) differ from official Benchmark Heaven leaderboard scores because: 1. **Sealed vs. Public Split:** Local evaluations run only on the **231 public JevBench tasks**, whereas Benchmark Heaven evaluates both public items and private **sealed items**. 2. **Public/Sealed Generalization Gap Penalty:** Benchmark Heaven penalizes divergence between public and sealed accuracy. 3. **10-Second Cutoff & Server Reference Hardware:** Tasks exceeding 10 seconds receive zero credit, and speed/cost metrics are measured on server-grade GPUs rather than local workstation hardware. --- ## Quickstart & SDK Usage (`v1.1`) ```bash pip install classone ``` ### Python SDK (`v1.1` Standalone Stage 2 Readout) ```python import torch from huggingface_hub import hf_hub_download from transformers import AutoTokenizer from classone.modeling.modeling_classone import ClassOneModel from classone.schemas import ChoiceQuestion, NoulQuestion, ScoreQuestion from classone.stage2_readout import ChatDecisionPromptBuilder, evaluate_stage2_request REPO_ID = "devops-thiago/classone-gemma4-e2b" REVISION = "v1.1" # or "main" tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision=REVISION, trust_remote_code=True) codebook_path = hf_hub_download(REPO_ID, "gemma4_e2b_codebook_v1.json", revision=REVISION) chat_builder = ChatDecisionPromptBuilder(tokenizer, codebook_path=codebook_path) model = ClassOneModel.from_backbone( REPO_ID, revision=REVISION, tokenizer=tokenizer, device="cuda:0", torch_dtype=torch.bfloat16, ) heads = torch.load( hf_hub_download(REPO_ID, "classone_heads.pt", revision=REVISION), map_location="cuda:0", weights_only=True, ) model.noul_head.load_state_dict(heads["noul_head"], strict=False) model.choice_head.load_state_dict(heads["choice_head"], strict=False) model.score_head.load_state_dict(heads["score_head"], strict=False) model.eval() answers, total_tokens = evaluate_stage2_request( model, chat_builder, state={"customer": "Alex", "message": "I was charged twice for order #123."}, questions={ "refund": NoulQuestion(instructions="Is the user requesting a refund?"), "dept": ChoiceQuestion( instructions="Route to team:", criteria={"billing": "Payment issues", "tech": "Technical bugs"}, ), "anger": ScoreQuestion( instructions="Dissatisfaction level:", criteria=["satisfied", "neutral", "dissatisfied", "churning"], ), }, m2_k_threshold=64, m2_top_k=16, ) print("Refund P(true):", answers["refund"].noul) print("Department: ", answers["dept"].choice, "—", answers["dept"].probabilities) print("Anger score: ", answers["anger"].score) ``` ### FastAPI Server (`v1.1`) `config.json` enables Stage 2 (`eval_mode = "stage2"`, `m2_k_threshold = 64`, `m2_top_k = 16`, `codebook_file = "gemma4_e2b_codebook_v1.json"`) automatically when served via `classone.server.app`: ```bash export CLASSONE_BASE_MODEL="devops-thiago/classone-gemma4-e2b" uvicorn classone.server.app:app --host 0.0.0.0 --port 8000 ``` ### Pinning `v1.0` (`revision="v1.0"`) To reproduce the original `v1.0` upload prior to the `v1.1` repair release, pin `revision="v1.0"`: ```python import torch from huggingface_hub import hf_hub_download from transformers import AutoTokenizer from classone.modeling.modeling_classone import ClassOneModel from classone.tokenizer import ClassOnePromptBuilder REPO_ID = "devops-thiago/classone-gemma4-e2b" tokenizer = AutoTokenizer.from_pretrained(REPO_ID, revision="v1.0", trust_remote_code=True) builder = ClassOnePromptBuilder(tokenizer) model = ClassOneModel.from_backbone( REPO_ID, revision="v1.0", tokenizer=tokenizer, device="cuda:0", torch_dtype=torch.bfloat16, ) heads = torch.load( hf_hub_download(REPO_ID, "classone_heads.pt", revision="v1.0"), map_location="cuda:0", weights_only=True, ) model.noul_head.load_state_dict(heads["noul_head"], strict=False) model.choice_head.load_state_dict(heads["choice_head"], strict=False) model.score_head.load_state_dict(heads["score_head"], strict=False) model.eval() ``` Or when launching the FastAPI server: ```bash export CLASSONE_BASE_MODEL="devops-thiago/classone-gemma4-e2b" export CLASSONE_REVISION="v1.0" ``` --- ## Repository Files (`v1.1`) | File | Description | |---|---| | `model-00001-of-00002.safetensors`, `model-00002-of-00002.safetensors`, `model.safetensors.index.json` | Unmodified BF16 `Gemma4TextModel` text + PLE weights (`google/gemma-4-e2b-it` `@3e22461f`, multimodal towers stripped) | | `config.json` | Text/PLE configuration (`final_logit_softcapping = 30.0`) + `classone_config` (`eval_mode = "stage2"`, `m2_k_threshold = 64`, `m2_top_k = 16`, `codebook_file = "gemma4_e2b_codebook_v1.json"`) | | `tokenizer.json`, `tokenizer_config.json`, `chat_template.jinja` | Gemma 4 tokenizer (`vocab_size = 262,144`) with ClassOne delimiter mapping (`..`) | | `gemma4_e2b_codebook_v1.json` | Versioned `180`-entry single-token round-trip codebook (`SHA256: 692C78FB...`) | | `classone_heads.pt` | Unchanged `v1.0` calibrated Noul / Choice / Score heads (`SHA256: F6A6E423...`) | --- ## Attribution & Legal - Base backbone: [`google/gemma-4-e2b-it`](https://huggingface.co/google/gemma-4-e2b-it) (Google) — Apache License 2.0 - Architecture & evaluation code: [`devops-thiago/class-one`](https://github.com/devops-thiago/class-one) — Apache License 2.0