crazy theory/suggestion
what if for v7 you used a (v)LLM as the text encoder and a modern VAE like qwen image's (or even went without a VAE and went pixel-space)? i just wonder what would happen if you maxed out the "exterior" parts of the model with the state of the art methods, and then tried to make the most out of the smallest backbone. would it make a considerable difference, or would it still be held back by the small size and training data?
or make it autoregressive, that will surely work XD
to be honest i only know a very high level overview of how image models work, this is really cool though!
sorry for the late reply, HF just decided not to give me notifications on this org. this is a really interesting idea, but I probably would not combine all of it into v7. I have tried to make each PixelModel version test a reasonably clear change, so replacing the text encoder, VAE, and generation method simultaneously would make it impossible to know what actually improved the model.
The modern VAE idea is probably the best candidate for v7. I could keep the v6 backbone, data, and conditioning mostly unchanged and test whether a better latent representation improves quality or training efficiency. Pixel-space diffusion would be extremely expensive because it would increase the image token count from 256 to potentially thousands, while autoregressive generation would basically be an entirely different model family.
A larger LLM or VLM text encoder could improve prompt understanding, but it might also make the external frozen components several times larger than the actual PixelModel backbone. That would still be a useful experiment, but probably better as its own version so the result is actually interpretable.
@TobiasLogic when does v7 come? 🤩 I am so excited 😊
expect it to come out 2-3 weeks from now considering we're compute bound and focusing on other Modalities at the moment
Okay. Thanks. Looking forward to it 🤩