👋 Open to Work

Mateusz Piesiak PRO

Mati83moni

9 22 6

mati83mon

AI & ML interests

Hybrid LLMs, MoE, Mamba, multilingual NLP, biomedical reasoning, efficient pretraining

Recent Activity

updated a Space about 8 hours ago

Mati83moni/TTS_Limits_Free

published a Space about 8 hours ago

Mati83moni/TTS_Limits_Free

upvoted an article 1 day ago

Evaluation Awareness: One Direction, or Eight?

View all activity

Organizations

None yet

Is "evaluation awareness" in a language model one direction, or eight?

We held the task fixed, toggled the eight cue factors of EvalAwareBench one at a time, and looked at the residual stream of Qwen3 (0.6B–32B), Gemma 3 and PLLuM-12B, plus a first Polish cross-lingual pilot.

What we found (descriptive, same instrument and author throughout):
- Cue directions overlap at about 1/5–1/4 of what two estimates of the same cue reach: not one shared axis.
- The standard all-cues direction is mostly an explicit-disclosure detector (cosine 0.81–0.89 with that cue).
- It transfers poorly to stimulus artificiality (held-out AUROC ≈ 0.5–0.6), though the models do represent that cue.
- No statistically detectable MMLU framing effect in the tested settings. That does not establish absence of an effect or of sandbagging.
- The Polish pilot is inconclusive: the formal verdict says so, and its own length control was not met.

The article carries its corrections (three early findings were withdrawn at scale; one traced to a bug in our hook code), an open validation problem for the add interventions, and all 23 figures with sources.

Full article: https://huggingface.co/blog/Mati83moni/is-evaluation-awareness-one-direction-or-eight

Spaces:
Mati83moni/evaluation-awareness-one-direction-or-eight
Mati83moni/evaluation-awareness-pllum-12b