diffusion train

• RL for Consistency Models, Faster Reward Guided Text-to-Image Generation
  • to optimize for task specific rewards, enable fast training-inference, we propose fine-tuning via RL
  • Reinforcement Learning for Consistency Model (RLCM)
  • objectives challenging with prompting, like image compressibility and human feedback

• distributed-diffusion using hivemind (distributed training) vs Deepspeed
• COMPOSITIONAL DIFFUSION
• SiT discrete transformers

• 4, 8 bit models, Q-Diffusion insight reddit quantization
  • Memory-Efficient Personalization using Quantized Diffusion Model (enhancing it)
• Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models
  • align outputs of the quantized model and the full-precision model at different network granularity
• QuEST: Low-bit Diffusion Model Quantization via Efficient Selective Finetuning
  • finetuning the quantized model to better adapt to the activation distribution (mitigation)
• Task-Oriented Diffusion Model Compression
  • satisfactory output quality with 39.2% and 56.4% reduction in model footprint and 81.4% and 68.7%
  • applying it to InstructPix2Pix and StableSR

• GIT RE-BASIN: MERGING MODELS MODULO PERMUTATION SYMMETRIES
  • transfer knowledge between teacher to student model
• Idempotent Generative Network
  • f(f(z))=f(z), can generate an output in one step
  • step towards a “global projector” = projecting any input into a target data distribution

• uform: clip not required, trained in a day
• cloneofsimo: learning from the clip
  • wanna perform affordable kernel regression on l2-normalized data?
    • get yourself Spherical Random Features for Polynomial Kernels
    • relevant if you are aiming for large scale non-parametric regression on CLIP projected feature spaces

• Efficient Diffusion Training via Min-SNR Weighting Strategy
  • slow convergence due to conflicting optimization directions between timesteps, 3.4 times faster
• Imagen suggests that scaling the text encoder is much more impactful than scaling the UNet
  • at least for diffusion models
• mosaiclml: custom $50k stable diffusion training, reddit post
• Patch Diffusion: Faster and More Data-Efficient Training of Diffusion Models
• compressed-stable-diffusion 36% reduced parameters and latency
• Wuerstchen: Efficient Pretraining of Text-to-Image Models
  • 16 times faster to train, 2 times faster inference, , only 9200 GPU hours (42 time compression rate vs 8 of sd)
• DREAM: Diffusion Rectification and Estimation-Adaptive Models (requiring minimal code changes)
  • 2 to 3 times faster training convergence
• PERCEPTUAL LOSS best=

• faster using electric flow-charges https://www.assemblyai.com/blog/an-introduction-to-poisson-flow-generative-models/
  • better than inference: https://twitter.com/_akhaliq/status/1620958983639924736 https://arxiv.org/pdf/2302.00482.pdf
• Spectral Diffusion: slim Standard Diffusion, 20 times smaller in size
  • Wavelet diffusion code
    • Wavelet Diffusion Models are fast and scalable Image Generators
• Score-Based Diffusion Models as Principled Priors for Inverse Imaging (more complex priors)
• COMPOSITIONAL DIFFUSION DIFFUSION TRANSFORMER

• Shifted Diffusion =Corgi= for Text-to-image Generation: from clip straight to diffusion, =only 1.7 of the images required captions=
• Object Detection: CutLER
• D3S: Invariant Learning via Diffusion Dreamed Distribution Shifts, separating foreground-background
  • disentangling foreground from background by chopping-pasting them out in the synthetic training dataset
  • like SVDiff
• A Picture is Worth a Thousand Words: Principled Recaptioning Improves Image Generation
  • automatic captioning is better than crawled low quality captions
  • CapsFusion: Rethinking Image-Text Data at Scale
    • hindered by simplistic captioners, consolidate and refine information

I have recently written a paper on understanding transformer learning via the lens of coinduction & Hopf algebra. https://arxiv.org/abs/2302.01834

The learning mechanism of transformer models was poorly understood however it turns out that a transformer is like a circuit with a feedback.

I argue that autodiff can be replaced with what I call in the paper Hopf coherence which happens within the single layer as opposed to across the whole graph.

Furthermore, if we view transformers as Hopf algebras, one can bring convolutional models, diffusion models and transformers under a single umbrella.

I’m working on a next gen Hopf algebra based machine learning framework.

Join my discord if you want to discuss this further https://discord.gg/mr9TAhpyBW