[AI 0002] Improve your Standard Diffusion art with VAEs doing the exact same thing.
Fine tuning a model makes for magic.
Introduction
Every model comes with it's own Variational Autoencoder (VAE). However, there are sometimes oversights or places where the model wasn't tuned well and that is where custom VAEs come into play. Different VAEs do different things. Some can improve certain features such as the mouth or eyes and others can improve the color. In my example if you look at the above notice how they are the same image, but one of them looks washed out? Take a guess, which one do you think has the VAE applied and which one doesn't? If you guessed the left one then you are correct. When I was starting out with this model I realized that something was wrong with my images, but I couldn't put my finger on it. It wasn't until someone mentioned that my images look bad that I thought maybe I downloaded a bad model and went to do some research only to find out that I didn't apply the custom VAE that was tuned for this model.
What are VAEs?
They area family of probabilistic graph models and variational Bayesian metho---blah blah blah. Speak English!
I simply wants to see something interesting and novel. These images are possibly something no one has seen before ever and in the event that they aren't shared... never again.
Simply put, VAEs are middlemen that take little squiggly what-cha-ma-call-it thingy-ma-bobs called latent vectors and translates late them into pixels—the oh so pretty pixels—that we dare to place our gaze upon.
How to use Custom VAEs
There are two ways to use VAEs. You can tailor it to a model specifically or you set it up for general use:
To make it usable for a specific model:
You want to copy the VAE in to your models folder and change the name so it matches the model and then add a
.vae.ptas the extension.For example, if I have a model
appleSauce.ckptand I wanted to apply a VAE to that model specifically I would copymyCustomVae.vae.ptinto the model folder withappleSauce.ckptand then I would renamemyCustomVae.vae.pttoappleSauce.vae.pt.That’s all you need to do and after restarting A1111-webui the VAE will automatically be applied.
To make it usable for all models:
The other way to use VAE is to put the into the
./models/VAEfolder and then go into A1111-webui's settings and select the VAE. This will allow the VAE to be usable for any model. However, if you use many different models you may have unexpected results as the VAE is blanket applied on all your models. (Wonder why there isn’t a quick switch)
The one challenge with this is if for example you use generate both realistic and drawn characters then using a cookie cutter approach can yield unintended results and could cause issues on it’s own. That’s why I like to use the first method where I leave the built-in setting to auto and I create a new VAE for each model. It takes up more space, but I can ensure that I always know what I’m dealing with.
Conclusion
Just like that your creations have the potentially to magically improve. In my case, my images now have more depth and color to them—at least for drawn characters. However, neon hair now looks completely wild. As you can see on the left the green hair is now light saber green as opposed to a soft sea green. This is why I suggest that you tailor fit the VAE for the model you are using.