Villanova2
Collection
Collection of all models in the Villanova2 family. • 10 items • Updated • 2
Villanova2-2B-Base
2.4B dense parameters · 4.7T tokens · 131,072-token context · five European languages · open weight & data
A compact foundation model, trained from scratch on English, Italian, German, Spanish and French. Built in EU, for EU.
DISCLAIMER: This is a base model, not instruction-tuned. It is intended as a foundation for downstream fine-tuning and alignment.
| Property | Value |
|---|---|
| Model type | Dense decoder-only Transformer, Qwen3-style backbone |
| Total parameters | 2,411,190,272 (2.411B) |
| Non-embedding parameters | 2,076,301,312 (2.076B) |
| Vocabulary | 131,072-token BPE tokenizer · 130,816 embedding rows, tied input/output |
| Context window | 131,072 tokens |
| Openness | open weight & data: model weights under Apache 2.0; training data documented source by source, each dataset under its own terms |
| Languages | English, Italian, German, Spanish, French (partial support beyond these) |
| License | Apache 2.0 |
Quick start
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "VillanovaAI/Villanova2-2B-Base"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
model_id,
dtype=torch.bfloat16,
device_map="auto",
)
prompt = "L'Italia è una penisola che"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
output = model.generate(
**inputs,
max_new_tokens=128,
do_sample=True,
temperature=0.7,
top_p=0.9,
)
print(tokenizer.decode(output[0, inputs.input_ids.shape[1]:], skip_special_tokens=True))
Prompting a base model means continuing it, not asking. Few-shot prompts, prefixes and document continuations work; "Can you explain…?" does not. For chat, use the instruction-tuned variant Villanova2-2B.
Intended use
| Good fit | Not a good fit without extra work |
|---|---|
| Continued pre-training and domain adaptation | Direct deployment as a chat assistant |
| SFT / preference tuning into assistants and task models | Safety-critical or high-stakes decisions |
| Few-shot text completion in EN · IT · DE · ES · FR | Reliable arithmetic or multi-step math |
| Research on small, multilingual, long-context models | Languages outside the five core ones, where quality is not guaranteed |
The family
- Villanova2-15B-A3B-Base · sparse mixture of experts, 15.5B total / 3.9B active
- Villanova2-15B-A3B · instruction-tuned
- Villanova2-15B-A3B-VL · vision-language
- Villanova2-2B-Base · dense, 2.4B (this model)
- Villanova2-2B · instruction-tuned
- Villanova2-2B-VL · vision-language
Both base models share the same 4.7T-token pre-training curriculum, the same tokenizer and the same training recipe; they differ in scale, topology and hardware budget.
Villanova-2B-Base-2603, from the inaugural generation, was a strong first step; Villanova2-2B-Base is a substantial step forward.
| Villanova-2B-Base-2603 | Villanova2-2B-Base | |
|---|---|---|
| Training tokens | 4.4T | 4.7T |
| Context window | 32,768 | 131,072 |
| Tokenizer | Salamandra, 256k | Villanova, 131k |
| 39-benchmark avg (5-shot) | 52.28 | 61.56 |
| RULER @ 4k | 73.3 | 83.0 |
| RULER @ 16k | 52.8 | 70.0 |
| RULER @ 32k | 47.0 | 60.1 |
The results published in the 2603 model card might differ because it comes from a different, smaller 18-task suite and is not directly comparable; Villanova-2B-Base-2603 is re-evaluated on the 39-task suite used for this model.
Architecture
| Architectural feature | Value |
|---|---|
| Backbone | Qwen3-style dense decoder |
| Layers | 22 |
| Hidden dimension | 2,560 |
| Attention | GQA, 20 Q heads / 4 KV groups (5:1 compression) |
| Head dimension | 128 |
| FFN activation | SwiGLU |
| FFN intermediate dimension | 10,240 (4× expansion) |
| RoPE | Full, applied to all head dimensions |
| Embedding rows | 130,816 (tokenizer vocabulary: 131,072) |
| Tied embeddings | Yes |
| Total parameters | 2,411,190,272 |
| Active parameters per token | 2,411,190,272 |
Pre-training
One run, four stages, no restarts
The learning rate is cooled to zero once, over the whole run: the cooldown is the context extension, with no restart and no fresh optimizer for long context. Adam's first and second moments are carried across every stage boundary.
| Stage | Tokens (cumulative) | Seq. length | Global batch | Learning rate |
|---|---|---|---|---|
| Stable pre-training | 0 → 4.0T | 4,096 | 3,072 | 5.00 × 10⁻⁴ (constant) |
| Annealing | 4.0T → 4.6T | 4,096 | 3,072 | 5.00 → 1.25 × 10⁻⁴ |
| Long context · 32k | 4.6T → 4.65T | 32,768 | 384 | 1.25 → 0.63 × 10⁻⁴ |
| Long context · 128k | 4.65T → 4.7T | 131,072 | 96 | 0.63 → 0 × 10⁻⁴ |
| Total | 4.7T |
The batch stays at 3,072 sequences: tokens per optimizer step are constant at 12.58M (3,072 × 4,096 = 384 × 32,768 = 96 × 131,072), about 373,500 steps in total.
The annealing cooldown is linear against a virtual 800B-token horizon ending at 4.8T, so training stops early at 4.6T with 25% of the pre-training learning rate left for the long-context tiers.
Compute
- 48 NVIDIA B200 GPUs,
423 hours each: **20,300 GPU-hours**, about 17.6 days wall-clock. - ≈ 6.8 × 10²² training FLOPs (6ND estimate).
- 4.7T tokens on a 2.411B model: ~1,950 tokens per parameter.
Data
All sources are converted to text and filtered before mixing: boilerplate removal, repetition and length heuristics, classifier scores where available, near- and exact deduplication, then tokenization with the Villanova tokenizer.
Stable phase · 4.0T tokens
| Source family | Sources | Role |
|---|---|---|
| English educational web | FineWeb-Edu, DCLM-Edu | Backbone of the mixture |
| PDF documents | FinePDFs | Long-form, structured text |
| Multilingual web | FineWeb-2, FineWeb2-HQ | Stable share for the four non-English languages |
| Encyclopedic | FineWiki | World knowledge |
| Mathematics | FineMath, Nemotron-CC-Math | Quantitative reasoning |
| Code | Stack-Edu | Code and technical text |
| Science | Common Pile | Scientific literature |
| Synthetic | Nemotron-CC, Nemotron-Pretraining | QA, code and reasoning data |
| Parallel text | Parallel FineWeb-Edu | Translate-aligned segments for IT/ES/DE/FR |
The mixture is dominated by English educational web and by the four major non-English languages. Each of Italian, Spanish, German and French draws on three complementary sources (FineWeb-2, FineWeb2-HQ and FinePDFs), so their share stays stable across the whole 4T-token phase.
Annealing · 600B tokens
Curated material replaces most of the long-tail web. New entries: FinePDFs-Edu (educational PDFs in the four languages), Dolma3 (code and mathematics), Dolmino-Mix (FLAN-style instruction text), Common Pile StackExchange (technical Q&A), GitHub issues and Kaggle notebooks (code-adjacent text) and the top mathematical tiers (FineMath-4+, InfiWebMath-4+). FineWeb-Edu, DCLM-Edu, FinePDFs, FineWeb2-HQ and FineWiki are up-weighted, with the web sources restricted to their highest-quality tiers.
Long context · 100B tokens
Two 50B-token tiers. The 32k tier mixes the annealing mixture with long DCLM-Edu documents and 12k+ parallel FineWeb-Edu segments; the 128k tier adds pre-1929 books, long scientific PDFs from Dolma 3 and a 128k+ educational slice. Short documents preserve what the previous stages learned.
Tokenizer
Villanova2 ships a 131,072-token BPE tokenizer built from scratch around the five project languages and code, instead of inheriting a 256k vocabulary spread across 35+ European languages.
| Parameter | Value |
|---|---|
| Algorithm | Byte-Pair Encoding (BPE) |
| Pre-tokenization | Llama-3 rules |
| Vocabulary | 131,072 tokens (model embedding: 130,816 rows) |
| Max token length | 128 |
| Minimum frequency | 8 |
| Training corpus | ~127 GB · 49 languages · 14 programming languages |
| Language mix | English ~44% · five core languages ~83% · Python ~2.2% |
| Training | 112 CPUs, 2 TB RAM, ~2 hours |
The design follows Llama-3 pre-tokenization: English clitics stay separate, digits are grouped in blocks of at most three (so large numbers are never opaque tokens), and indentation and line breaks survive. Against the previous Salamandra tokenizer, it improves fertility by 12% on the core languages and code and by more than 150% on non-Latin scripts, with half the vocabulary.
Evaluation
The protocol is 39 benchmarks, five languages and six reasoning domains, scored 0-shot and 5-shot. Every number is an unweighted flat average over the 39 tasks, using normalized accuracy where available. Per-language aggregates cover all 14 English tasks (GSM8K included), 7 Italian, and 6 each for German, Spanish and French. In the comparison tables, bold marks the best score in each column and underline the best among the open weight & data EU models.
The full 39-benchmark suite
- English (14): ARC-Challenge, ARC-Easy, Belebele (eng), CommonsenseQA, Global MMLU (en), Global PIQA (en), GSM8K, HellaSwag, MMLU, OpenBookQA, PIQA, SciQ, WinoGrande, XNLI (en).
- Italian (7): ARC-IT, Belebele (ita), EVALITA MP-MC, Global MMLU (it), Global PIQA (it), HellaSwag-IT, XCOPA-IT.
- German (6): ARC-DE, Belebele (deu), Global MMLU (de), Global PIQA (de), HellaSwag-DE, XNLI (de).
- Spanish (6): ARC-ES, Belebele (spa), Global MMLU (es), Global PIQA (es), HellaSwag-ES, XNLI (es).
- French (6): ARC-FR, Belebele (fra), Global MMLU (fr), Global PIQA (fr), HellaSwag-FR, XNLI (fr).
Overall: the strongest European model strictly below 3B
| Model | Openness | Origin | Params | 5-shot | 0-shot | EN | IT | DE | ES | FR |
|---|---|---|---|---|---|---|---|---|---|---|
| Ministral-3-3B-Base-2512 | weight | EU | 3.4B | 69.26 | 63.94 | 73.67 | 68.61 | 65.94 | 67.37 | 64.95 |
| Mistral-7B-v0.3 | weight | EU | 7.2B | 65.05 | 58.26 | 70.52 | 63.01 | 60.99 | 62.90 | 60.85 |
| DeepSeek-V2-Lite | weight | non-EU | 15.7B / 2.4B | 61.73 | 55.08 | 68.22 | 59.52 | 56.41 | 59.59 | 56.66 |
| Villanova2-2B-Base | weight & data | EU | 2.4B | 61.56 | 56.16 | 63.91 | 61.47 | 59.42 | 61.15 | 58.73 |
| OLMo-3-7B | weight & data | non-EU | 7.3B | 61.13 | 54.37 | 71.73 | 53.87 | 54.40 | 56.97 | 55.74 |
| salamandra-7b | weight & data | EU | 7.8B | 58.13 | 53.10 | 59.79 | 58.84 | 56.07 | 58.22 | 55.40 |
| Villanova-2B-Base-2603 | weight & data | EU | 2.4B | 52.28 | 46.99 | 55.09 | 52.43 | 50.47 | 51.24 | 48.38 |
| Minerva-7B-base | weight & data | EU | 7.4B | 50.30 | 47.36 | 59.58 | 55.99 | 37.18 | 43.09 | 42.32 |
| EuroLLM-1.7B | weight & data | EU | 1.7B | 44.70 | 43.54 | 47.03 | 44.49 | 44.19 | 43.54 | 41.23 |
| salamandra-2b | weight & data | EU | 2.3B | 43.83 | 42.76 | 47.54 | 42.66 | 42.22 | 41.70 | 40.30 |
Openness: weight = open weights only; weight & data = open weights plus documented training data. EN · IT · DE · ES · FR are the per-language 5-shot averages.
In-context learning is worth +5.40 points for Villanova2-2B-Base (56.16 → 61.56).
Per-language profile
| Language | 5-shot | Δ vs English |
|---|---|---|
| English | 63.91 | n/a |
| Italian | 61.47 | −2.44 |
| Spanish | 61.15 | −2.76 |
| German | 59.42 | −4.49 |
| French | 58.73 | −5.18 |
The five languages sit within 5.2 points of each other: English first, then Italian, Spanish, German and French.
Italian is the strongest non-English language at 61.47, only 2.44 points below English and the axis where the project's data investment is most visible: 44.44% on EVALITA MP-MC.
Long context: a window of 131,072, a measured curve
| Model | 4k | 8k | 16k | 32k | 64k | 128k | ECE |
|---|---|---|---|---|---|---|---|
| Llama-3.1-8B | 93.1 | 91.2 | 90.3 | 84.7 | 80.9 | 66.4 | 16,384 |
| Qwen3-8B-Base | 93.0 | 92.7 | 91.2 | 86.7 | — | — | 32,768 |
| Qwen3-4B-Base | 92.5 | 91.7 | 86.4 | 81.3 | — | — | 16,384 |
| Mistral-7B-v0.3 | 92.1 | 87.8 | — | — | — | — | 8,192 |
| OLMo-3-7B | 91.6 | 85.2 | 77.6 | 70.7 | — | — | 4,096 |
| EuroLLM-22B | 91.0 | 85.2 | 80.7 | 73.2 | — | — | 4,096 |
| Qwen3.5-9B-Base | 89.9 | 87.7 | 86.9 | 87.6 | — | — | 32,768 |
| Villanova2-15B-A3B-Base | 92.1 | 90.1 | 86.6 | 79.6 | 71.0 | 58.5 | 16,384 |
| Villanova2-2B-Base | 83.0 | 76.5 | 70.0 | 60.1 | 53.4 | 37.3 | < 4k |
| Villanova-2B-Base-2603 | 73.3 | 63.6 | 52.8 | 47.0 | — | — | — |
Effective context length (ECE) is the longest length at which the mean RULER score stays at or above the RULER pass mark, the 4K score of the Llama-2-7B reference model (85.6 here). By that strict definition, Villanova2-2B-Base does not have an effective 4k context: its 4k score is 83.0. What the curve does show is a monotone, smooth decay rather than a cliff: single-needle retrieval is solved at 4k–8k (98–100%), multi-key retrieval and code-word extraction are where the capacity runs out, and at 4k/8k/16k the model is 9.7 / 12.9 / 17.2 points above the previous generation. Treat 4k–8k as the reliable zone, 16k–32k as usable with care, and 128k as an available-but-lossy window.
Fine-tuning
- Start from this checkpoint for continued pre-training, domain adaptation, or SFT.
- Full fine-tuning of 2.4B parameters is feasible on a single 8×GPU node with BF16 and activation checkpointing; LoRA/QLoRA is enough for style and task adaptation.
- Use the Villanova tokenizer, not a foreign one: replacing it invalidates the embedding budget and the fertility gains.
- Recommended SFT sequence length: 4k–8k. If you need longer, fine-tune at the target length rather than extrapolating from 4k.
- The model was trained in MXFP8; BF16 fine-tuning is numerically safe and is the recommended default.
License, citation, acknowledgements
License. Apache 2.0.
Citation. If you use Villanova2-2B-Base, please cite our work
@techreport{villanova2_2026,
title = {Villanova2 --- Technical Report},
author = {{Villanova.AI}},
institution = {Villanova.AI},
year = {2026},
note = {Coming soon}
}
Acknowledgements. This work is part of the Villanova project, financed by IPCEI-CIS, Prog. n. SA. 102519 — CUP B29J24000850005.
Villanova2-2B-Base · built in EU, for EU.
- Downloads last month
- 9