tinystories-50m / tokenizer.json

Commit History

  1. v2: 12288-vocab BPE tokenizer (was 8192) (#5)⁠

    3ebcad6

  1. Add BPE-8192 tokenizer

    Verified signature
    c55f7f9