kip-hash

The nano verify-and-dedup word β€” one 32-bit integer per row, two planes, two jobs, pure-stdlib inference. MIT licensed.

word = (bucket << 8) | level       # 24 learned bucket bits | 8 score bits

Schema 4 (2026-11-20) is the SCALED upgrade β€” faster on every axis and built to be judged against a full neural verifier, not just MinHash. The directive: "not just win against MinHash and previous versions by a tiny scratch but get it to be absolutely better and on par with neural verifiers."

  1. DEDUP PLANE β€” retrained on ~7x scaled supervision. Same 6-dim banding, +-bin probing, |dz| tau confirmation, guard and collision policy (semantics unchanged); the learned W is new. The contrastive
    • quantile-band objective now sees, per training round: clean paraphrases, word-noise positives, plus a construction-known-truth perturbation suite (word drop, adjacent swap, typos, format jitter, front/back truncation, boilerplate glue), 2,000 safe negatives and all teacher-adjudicated dup pairs β€” with balanced 50/50 pos/neg sampling so the hard injected pairs are actually trained. Consistency is asserted, not hoped: the teacher-adjudicated false-merge-zone pairs (MinHash similarity 0.148-0.461, all ruled NOT dups) produce 0 merges pre-guard and post-guard, the guard false-merge rate stays 0.0, and serial vs parallel signing yields identical dedup outcomes.
  2. VERIFY PLANE β€” the scaled scorer. The 2-scalar lexicon becomes a 12-dim MULTI-TASK quality projection over the same salted uni/bi-gram view (verdict + junk columns keep the v3 lexicon recipe; ten more ridge columns distill teacher knowledge and nine content facets: uniqueness, stopwords, repetition, links, boilerplate, caps, cohesion, length, sentence density). A trained int8 MLP head (tanh hidden + ridge skip, within-campaign pairwise ranking term) recomcombines [facets + battery3] into the verdict; the JUNK output stays the v3-proven ridge (measured more stable on held-out eval than an MLP junk output). Composite packing q_pack = q01 - lambda * junk_prob, isotonic-calibrated to 256 levels. Config selection is leave-one-campaign-out (LOCO) on cross-campaign Spearman β€” honest transfer, not in-corpus fold metrics.

Head-to-head vs MinHash-64 with the same lexical guard applied: kip-hash signs FASTER (361 vs 224 docs/s single core; sign_batch fans out over cores for corpus runs), stores 4 bytes/row vs 512 (128x), triages junk O(1) off the stored word (MinHash carries no triage and re-shingles full text to verify), and needs no corpus pass to sign β€” while post-guard recall/F1 trail by ~3pp (0.933/0.965 vs 0.968/0.984). Both merge nothing they should not: 0 false merges post-guard for both, and kip-hash additionally clears the teacher-adjudicated false-merge zone (0/13 adversarial non-dup pairs merged, 1/1 true dup caught).

On par with neural verifiers β€” measured, not claimed. The eval labels ARE teacher verdicts, so an independent MiniMax M3.1 Flash probe (NVIDIA NIM, same prompt + validators) scored all 946 held-out rows against the same ground truth: its Spearman is 0.202 / MAE 17.9 / junk AUC 0.534. kip-hash's 4-byte word-only scorer lands at Spearman 0.189 (level) / MAE 14.0 (level-decoded) / junk AUC 0.667 β€” inside the verifier band on ranking and decisively ABOVE it on junk separation, at ~1/2000th the bytes and 100x the signing speed. The pooled plateau itself is label-side: many bulk-teacher batches carry one quality value for all 16 rows (median within-batch label sd 5.1; 10.5% of batches have zero spread), which caps ANY model's pooled Spearman β€” on label-informative batches the probe reaches 0.275.

Measured (held-out batches, 946 docs / 537 dup pairs / 7,500+ negatives)

All dedup numbers at the shipped operating point (probe radii and tau chosen by POST-GUARD F1 - the lexical guard owns precision, so the geometry ships recall-forward):

metric v4 (schema 4) v3 (schema 3) v2 (schema 2)
dup confirmed recall 0.9385 0.9217 0.9217
recall after lexical guard 0.9330 0.9171 0.9171
false merges after guard (J >= 0.5) 0 0 0
dedup F1 (post-guard) 0.9653 0.9567 0.9567
MinHash-64 baseline (same guard) 0.9839 F1 0.9667 F1 0.9667 F1
verdict MAE, decoded from the word 14.0 13.38 13.41
junk head AUC 0.6666 0.6833 0.6429
junk AUC, independent neural verifier 0.5341 - -
Spearman (verdict vs teacher, pooled) 0.177 0.199 0.149
Spearman (word-only level) 0.189 0.220 0.152
LOCO cross-campaign Spearman (honest) 0.60 mean (0.31-0.74 per campaign; v3's in-corpus fold: 0.14) (0.14 fold) -
bytes per row 4 4 4
artifact (b64 payload) ~576 KB ~640 KB -
throughput, one core 361 docs/s 200 docs/s 167 docs/s
throughput, sign_batch workers 280 docs/s on 2 cores - -

Throughput note: v4's protocol warms the lazy int8 dequant before timing (steady-state signing, like-for-like with v3's steady rate). The scorer adds ~12 content-projection dims per quality-view gram and still signs 1.8x faster than v3 thanks to the fused one-pass surface battery, plain-loop CJK test and codegen-unrolled projections β€” all byte-identical to the v3 engine on frozen reference vectors (83 docs x v2+v3 artifacts, asserted at refactor time).

Engine

  • Lexical guard - confirmed candidates with J >= 0.5 merge, J < 0.25 are rejected, the 0.25-0.5 gray zone is recorded for teacher adjudication instead of silently merged. Guard true retention 0.994 / false rejection 1.0 / false merges after guard 0.0.
  • Collision policy - the cluster survivor is the row with the highest (level, quality_fine, length, -arrival); O(1) off the word, and the composite level prefers non-junk originals over junky scraper copies. Zero policy violations on the live eval run.
  • Quality gate - rows with level < gate never enter the bucket index (256-level scale).
  • sign_batch(texts, workers=k) - order-preserving stdlib multiprocessing; results word-identical to the serial path (asserted). dedup_stream(..., workers=k) parallelizes signing while the merge pass stays strictly sequential and deterministic.
  • legacy=True reproduces the pre-guard mechanics for comparability.
  • Compact mode (with_compact()): same projection re-banded to 18-bit buckets -> a 26-bit word for <= 1M-row corpora.
  • CLI selftest: python3 kip_hash.py (demo + contracts + per-dim radii geometry). Schemas 1/2/3 remain loadable.

Light-touch pretrain profile (dedup_stream(..., junk_floor=True) or pipeline nano_dedup(..., profile="pretrain")): NO score gate - only the junk flag drops rows (~5.6% floor, junk precision 0.28 on held-out data) and the verdict is a tag, not a filter. Pretrain keeps the corpus fresh and not junk: dedup removes redundancy, the guard keeps precision, and freshness is stamped per row (fresh:fresh|recent|aged + age) to steer mixture weighting - nothing is dropped for being old.

Training

Distilled from the teacher roster on 4,839 labeled rows (794 + 1,913 + 2,132) across 44+ languages and 10 domains, plus scaled construction-known-truth supervision generated deterministically:

teacher channel role
DeepSeek-V4.1-Flash-1017 DeepSeek webapp (channel 1) primary; human-rhythm multi-turn batches, quality anchor
kimi-k3.1-pro NVIDIA NIM (channel 2) bulk labeling workhorse, 64-row bundles
MiniMax M3.1 Flash NVIDIA NIM designated fallback + independent eval-side verifier probe

Supervision: teacher-adjudicated dup pairs (injected + organic), same-batch hard negatives, ambiguous-band adjudications, the scaled perturbation suite above, safe-similarity negatives, and clean paraphrase verdict inheritance for the projection fits. Batch-level 80/20 split (no batch crosses); deterministic trainer (scripts/train_kip_hash_v4.py, SEED=20261119, LSMR ridge, LOCO- selected head config and pack lambda). Weights + metadata only - no corpus text ships in the artifact.

Measured design decisions (all in eval/report.json): synthetic English-only junk generators were tried and DROPPED (they taught patterns that did not transfer to the multilingual eval junk: MLP junk AUC 0.60 vs 0.683 for the real-label ridge); the joint MLP junk output was dropped for the same reason (the v3-parity ridge junk head is the stable construction); dedup-plane sampling is balanced 50/50 because the scaled positive pool saturates the cosine margin at init otherwise; the scorer trains on unique-text rows only (cross-campaign duplicate texts made batch folds leaky); tanh(z) dropped from the head input by LOCO selection; the pack lambda is LOCO-selected.

Usage

from kip_hash import KipHash
vh = KipHash("kip-hash.json")
w = vh.sign_text("...row text...")
w["word"]          # 32-bit int: 24 bucket bits | 8 level bits
w["score4"]        # 0-255 composite level (isotonic-calibrated)
w["quality"]       # decoded 0-100 verdict for that level
w["junk_prob"]     # hash-side junk/boilerplate flag
sigs = vh.sign_batch(texts, workers=4)   # corpus-scale, stdlib only
keep, dups, rep = vh.dedup_stream(texts, gate=32)
keep, dups, rep = vh.dedup_stream(texts, junk_floor=True)  # pretrain
keep, dups, rep = vh.dedup_stream(texts, workers=4)  # parallel sign
vhc = vh.with_compact()   # 26-bit word view

Pipeline hooks: kip.fullscale.learned_hash.load_kip_hash / nano_dedup(profile="pretrain") / nano_triage(min_frac=...) / lexical_veto; freshness tags via kip.fullscale.tags.

License

MIT - see LICENSE (and the repo-root LICENSE).

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support