Post
2,326
We just released Drishya Tiny, a 212M-parameter video generator trained from scratch on open-domain video, using the frozen Wan2.2 VAE and the Flan-T5-Base text encoder. It turns a text prompt, or a photo plus a prompt, into a 2-second 256×256 clip. The training run was 15 hours on one L40S, and the model’s own path cost about $21 of GPU time.
On VBench (our own sampling and scoring with the official code, not a certified entry) it scores 75.6 on quality, level with VideoCrafter-0.9 (74.9) and close to ModelScope (78.1). On semantic it’s 32.3 against their 65-67, so following the prompt is the weak part. In a controlled test, better captions gave +3.0, and adding 30% photos on top gave about 1.6 more, all of it in what objects are and none in composition. I think at this size, looking real is cheap and understanding the prompt is what costs.
It’s a research release, so the weights, code, logs, the full cost ($57.43 with every failed run) and the negative results are all out. Weights are CC BY-NC 4.0 because the training clips are research-only; the code is Apache-2.0.
And it runs right in your browser with WebGPU, from text or a photo, on your own device. The browser version uses the tiny TAEW2.2 decoder by Ollin Boer Bohan, so its clips are a bit softer, and a clip takes a few seconds on a recent Mac.
Run it in your browser: https://udgamlabs.com/models/drishya-tiny
Space: udgamlabs/Drishya-Tiny
Model: udgamlabs/Drishya-Tiny
Technical report: https://udgamlabs.com/research/drishya-tiny
Write-up: https://udgamlabs.com/blog/drishya-tiny
Code: https://github.com/udgamlabs/drishya-tiny
On VBench (our own sampling and scoring with the official code, not a certified entry) it scores 75.6 on quality, level with VideoCrafter-0.9 (74.9) and close to ModelScope (78.1). On semantic it’s 32.3 against their 65-67, so following the prompt is the weak part. In a controlled test, better captions gave +3.0, and adding 30% photos on top gave about 1.6 more, all of it in what objects are and none in composition. I think at this size, looking real is cheap and understanding the prompt is what costs.
It’s a research release, so the weights, code, logs, the full cost ($57.43 with every failed run) and the negative results are all out. Weights are CC BY-NC 4.0 because the training clips are research-only; the code is Apache-2.0.
And it runs right in your browser with WebGPU, from text or a photo, on your own device. The browser version uses the tiny TAEW2.2 decoder by Ollin Boer Bohan, so its clips are a bit softer, and a clip takes a few seconds on a recent Mac.
Run it in your browser: https://udgamlabs.com/models/drishya-tiny
Space: udgamlabs/Drishya-Tiny
Model: udgamlabs/Drishya-Tiny
Technical report: https://udgamlabs.com/research/drishya-tiny
Write-up: https://udgamlabs.com/blog/drishya-tiny
Code: https://github.com/udgamlabs/drishya-tiny
