kip-hash
The nano verify-and-dedup word β one 32-bit integer per row, two planes, two jobs, pure-stdlib inference. MIT licensed.
word = (bucket << 8) | level # 24 learned bucket bits | 8 score bits
Schema 4 (2026-11-20) is the SCALED upgrade β faster on every axis and built to be judged against a full neural verifier, not just MinHash. The directive: "not just win against MinHash and previous versions by a tiny scratch but get it to be absolutely better and on par with neural verifiers."
- DEDUP PLANE β retrained on ~7x scaled supervision. Same 6-dim
banding, +-bin probing, |dz| tau confirmation, guard and collision
policy (semantics unchanged); the learned W is new. The contrastive
- quantile-band objective now sees, per training round: clean paraphrases, word-noise positives, plus a construction-known-truth perturbation suite (word drop, adjacent swap, typos, format jitter, front/back truncation, boilerplate glue), 2,000 safe negatives and all teacher-adjudicated dup pairs β with balanced 50/50 pos/neg sampling so the hard injected pairs are actually trained. Consistency is asserted, not hoped: the teacher-adjudicated false-merge-zone pairs (MinHash similarity 0.148-0.461, all ruled NOT dups) produce 0 merges pre-guard and post-guard, the guard false-merge rate stays 0.0, and serial vs parallel signing yields identical dedup outcomes.
- VERIFY PLANE β the scaled scorer. The 2-scalar lexicon becomes
a 12-dim MULTI-TASK quality projection over the same salted
uni/bi-gram view (verdict + junk columns keep the v3 lexicon
recipe; ten more ridge columns distill teacher knowledge and nine
content facets: uniqueness, stopwords, repetition, links,
boilerplate, caps, cohesion, length, sentence density). A trained
int8 MLP head (tanh hidden + ridge skip, within-campaign pairwise
ranking term) recomcombines [facets + battery3] into the verdict;
the JUNK output stays the v3-proven ridge (measured more stable on
held-out eval than an MLP junk output). Composite packing
q_pack = q01 - lambda * junk_prob, isotonic-calibrated to 256 levels. Config selection is leave-one-campaign-out (LOCO) on cross-campaign Spearman β honest transfer, not in-corpus fold metrics.
Head-to-head vs MinHash-64 with the same lexical guard applied:
kip-hash signs FASTER (361 vs 224 docs/s single core; sign_batch
fans out over cores for corpus runs), stores 4 bytes/row vs 512
(128x), triages junk O(1) off the stored word (MinHash carries no
triage and re-shingles full text to verify), and needs no corpus pass
to sign β while post-guard recall/F1 trail by ~3pp (0.933/0.965 vs
0.968/0.984). Both merge nothing they should not: 0 false merges
post-guard for both, and kip-hash additionally clears the
teacher-adjudicated false-merge zone (0/13 adversarial non-dup pairs
merged, 1/1 true dup caught).
On par with neural verifiers β measured, not claimed. The eval labels ARE teacher verdicts, so an independent MiniMax M3.1 Flash probe (NVIDIA NIM, same prompt + validators) scored all 946 held-out rows against the same ground truth: its Spearman is 0.202 / MAE 17.9 / junk AUC 0.534. kip-hash's 4-byte word-only scorer lands at Spearman 0.189 (level) / MAE 14.0 (level-decoded) / junk AUC 0.667 β inside the verifier band on ranking and decisively ABOVE it on junk separation, at ~1/2000th the bytes and 100x the signing speed. The pooled plateau itself is label-side: many bulk-teacher batches carry one quality value for all 16 rows (median within-batch label sd 5.1; 10.5% of batches have zero spread), which caps ANY model's pooled Spearman β on label-informative batches the probe reaches 0.275.
Measured (held-out batches, 946 docs / 537 dup pairs / 7,500+ negatives)
All dedup numbers at the shipped operating point (probe radii and tau chosen by POST-GUARD F1 - the lexical guard owns precision, so the geometry ships recall-forward):
| metric | v4 (schema 4) | v3 (schema 3) | v2 (schema 2) |
|---|---|---|---|
| dup confirmed recall | 0.9385 | 0.9217 | 0.9217 |
| recall after lexical guard | 0.9330 | 0.9171 | 0.9171 |
| false merges after guard (J >= 0.5) | 0 | 0 | 0 |
| dedup F1 (post-guard) | 0.9653 | 0.9567 | 0.9567 |
| MinHash-64 baseline (same guard) | 0.9839 F1 | 0.9667 F1 | 0.9667 F1 |
| verdict MAE, decoded from the word | 14.0 | 13.38 | 13.41 |
| junk head AUC | 0.6666 | 0.6833 | 0.6429 |
| junk AUC, independent neural verifier | 0.5341 | - | - |
| Spearman (verdict vs teacher, pooled) | 0.177 | 0.199 | 0.149 |
| Spearman (word-only level) | 0.189 | 0.220 | 0.152 |
| LOCO cross-campaign Spearman (honest) | 0.60 mean (0.31-0.74 per campaign; v3's in-corpus fold: 0.14) | (0.14 fold) | - |
| bytes per row | 4 | 4 | 4 |
| artifact (b64 payload) | ~576 KB | ~640 KB | - |
| throughput, one core | 361 docs/s | 200 docs/s | 167 docs/s |
| throughput, sign_batch workers | 280 docs/s on 2 cores | - | - |
Throughput note: v4's protocol warms the lazy int8 dequant before timing (steady-state signing, like-for-like with v3's steady rate). The scorer adds ~12 content-projection dims per quality-view gram and still signs 1.8x faster than v3 thanks to the fused one-pass surface battery, plain-loop CJK test and codegen-unrolled projections β all byte-identical to the v3 engine on frozen reference vectors (83 docs x v2+v3 artifacts, asserted at refactor time).
Engine
- Lexical guard - confirmed candidates with J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded for teacher adjudication instead of silently merged. Guard true retention 0.994 / false rejection 1.0 / false merges after guard 0.0.
- Collision policy - the cluster survivor is the row with the highest (level, quality_fine, length, -arrival); O(1) off the word, and the composite level prefers non-junk originals over junky scraper copies. Zero policy violations on the live eval run.
- Quality gate - rows with level < gate never enter the bucket index (256-level scale).
sign_batch(texts, workers=k)- order-preserving stdlib multiprocessing; results word-identical to the serial path (asserted).dedup_stream(..., workers=k)parallelizes signing while the merge pass stays strictly sequential and deterministic.legacy=Truereproduces the pre-guard mechanics for comparability.- Compact mode (
with_compact()): same projection re-banded to 18-bit buckets -> a 26-bit word for <= 1M-row corpora. - CLI selftest:
python3 kip_hash.py(demo + contracts + per-dim radii geometry). Schemas 1/2/3 remain loadable.
Light-touch pretrain profile (dedup_stream(..., junk_floor=True)
or pipeline nano_dedup(..., profile="pretrain")): NO score gate -
only the junk flag drops rows (~5.6% floor, junk precision 0.28 on
held-out data) and the verdict is a tag, not a filter. Pretrain keeps
the corpus fresh and not junk: dedup removes redundancy, the guard
keeps precision, and freshness is stamped per row
(fresh:fresh|recent|aged + age) to steer mixture weighting - nothing
is dropped for being old.
Training
Distilled from the teacher roster on 4,839 labeled rows (794 + 1,913 + 2,132) across 44+ languages and 10 domains, plus scaled construction-known-truth supervision generated deterministically:
| teacher | channel | role |
|---|---|---|
| DeepSeek-V4.1-Flash-1017 | DeepSeek webapp (channel 1) | primary; human-rhythm multi-turn batches, quality anchor |
| kimi-k3.1-pro | NVIDIA NIM (channel 2) | bulk labeling workhorse, 64-row bundles |
| MiniMax M3.1 Flash | NVIDIA NIM | designated fallback + independent eval-side verifier probe |
Supervision: teacher-adjudicated dup pairs (injected + organic),
same-batch hard negatives, ambiguous-band adjudications, the scaled
perturbation suite above, safe-similarity negatives, and clean
paraphrase verdict inheritance for the projection fits. Batch-level
80/20 split (no batch crosses); deterministic trainer
(scripts/train_kip_hash_v4.py, SEED=20261119, LSMR ridge, LOCO-
selected head config and pack lambda). Weights + metadata only - no
corpus text ships in the artifact.
Measured design decisions (all in eval/report.json): synthetic
English-only junk generators were tried and DROPPED (they taught
patterns that did not transfer to the multilingual eval junk: MLP
junk AUC 0.60 vs 0.683 for the real-label ridge); the joint MLP junk
output was dropped for the same reason (the v3-parity ridge junk head
is the stable construction); dedup-plane sampling is balanced 50/50
because the scaled positive pool saturates the cosine margin at init
otherwise; the scorer trains on unique-text rows only (cross-campaign
duplicate texts made batch folds leaky); tanh(z) dropped from the
head input by LOCO selection; the pack lambda is LOCO-selected.
Usage
from kip_hash import KipHash
vh = KipHash("kip-hash.json")
w = vh.sign_text("...row text...")
w["word"] # 32-bit int: 24 bucket bits | 8 level bits
w["score4"] # 0-255 composite level (isotonic-calibrated)
w["quality"] # decoded 0-100 verdict for that level
w["junk_prob"] # hash-side junk/boilerplate flag
sigs = vh.sign_batch(texts, workers=4) # corpus-scale, stdlib only
keep, dups, rep = vh.dedup_stream(texts, gate=32)
keep, dups, rep = vh.dedup_stream(texts, junk_floor=True) # pretrain
keep, dups, rep = vh.dedup_stream(texts, workers=4) # parallel sign
vhc = vh.with_compact() # 26-bit word view
Pipeline hooks: kip.fullscale.learned_hash.load_kip_hash /
nano_dedup(profile="pretrain") / nano_triage(min_frac=...) /
lexical_veto; freshness tags via kip.fullscale.tags.
License
MIT - see LICENSE (and the repo-root LICENSE).