Hugging Face's logo
R's picture

R PRO

Twu31
13
ErenAta00's profile picture masterset-ai's profile picture jackhooke's profile picture
·

AI & ML interests

None yet

Recent Activity

repliedto jiteshkprasad's post about 9 hours ago
We tested 27 small open language models, all under 400M parameters, in the same way. Same harness, same settings, and the model versions pinned. I think small models are really hard to compare from their own cards, so we put the whole thing out. Board: https://huggingface.co/spaces/udgamlabs/small-model-board Dataset: https://huggingface.co/datasets/udgamlabs/small-model-board-scores Write-up on the Udgam Labs blog: https://udgamlabs.com/blog/small-model-board The most surprising part for me was MMLU. In the usual A, B, C, D format, just answering "D" every time gets 26.9%, and 23 of these 27 models score at or below that. So at this size the letter format basically tests if a model can pick a letter, not what it knows. When we asked the same questions in cloze format, where the answer text itself is scored, all 23 did better. Our own models are in there like everyone else. Laghu-Base is 5th of 19 models up to 200M, and Gargi-Base is 7.3 points behind SmolLM2-360M. So we are not on top, and that is fine. What I wanted was a fair comparison. Do you compare small models on letter or cloze MMLU?
repliedto crp4222's post about 9 hours ago
I trained a small PII detector for PrivAiTe, my local proxy that masks personal data before a prompt reaches an LLM: crp4222/privaite-pii-minilm. It is a multilingual MiniLM (117M parameters, 15 types from names and contacts to IDs, bank details and secrets), trained only on datasets whose licenses allow commercial use, released under BSD 3-Clause, with PyTorch, ONNX fp32 and int8, and Core ML builds. On 9 public test sets it never saw, it masks as many person names as the best of GLiNER PII, NVIDIA GLiNER-PII, GLiNER2, Piiranha and OpenAI privacy-filter, or more, on 6 of them, usually with higher precision. It trails GLiNER PII on short lowercase voice requests and NVIDIA on court rulings. It is also 2.5 to 7 times smaller and much faster: 9 ms per 1,000 characters on an M1 Pro Neural Engine and 23 ms on CPU in int8, against 186 to 621 ms for the others on the same machine and documents. Numbers, test sets and limits are on the card: https://huggingface.co/crp4222/privaite-pii-minilm
reacted to dleemiller's post with 👍 about 9 hours ago
Join the competition! We have 45 models scored (and growing), across 44 public benchmarks. My goal for this space is to push the limits on NLI and the breadth of tasks these models can perform in. Contributions are encouraged, as well as suggestions for additional benchmarks. Right now, all models are cross encoders (Sentence Transformers 🥰), but LLMs are welcome too! https://huggingface.co/spaces/dleemiller/open-nli-leaderboard
View all activity

Organizations

None yet