arrow
Volume 31, Issue 4
VAE-KRnet and Its Applications to Variational Bayes

Xiaoliang Wan & Shuangqing Wei

Commun. Comput. Phys., 31 (2022), pp. 1049-1082.

Published online: 2022-03

Export citation
  • Abstract

In this work, we have proposed a generative model, called VAE-KRnet, for density estimation or approximation, which combines the canonical variational autoencoder (VAE) with our recently developed flow-based generative model, called KRnet. VAE is used as a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. Using a linear model between the data and the latent variable, we show that VAE-KRnet can be more effective and robust than the canonical VAE. VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function (PDF) known up to a constant. VAE-KRnet is flexible in terms of dimensionality. When the number of dimensions is relatively small, KRnet can effectively approximate the distribution in terms of the original random variable. For high-dimensional cases, we may use VAE-KRnet to incorporate dimension reduction. One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution. The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler (KL) divergence between the model and the posterior. For high-dimensional distributions, it is very challenging to construct an accurate density model due to the curse of dimensionality, where extra assumptions are often introduced for efficiency. For instance, the classical mean-field approach assumes mutual independence between dimensions, which often yields an underestimated variance due to oversimplification. To alleviate this issue, we include into the loss the maximization of the mutual information between the latent random variable and the original random variable, which helps keep more information from the region of low density such that the estimation of variance is improved. Numerical experiments have been presented to demonstrate the effectiveness of our model.

  • AMS Subject Headings

62C10, 62G07, 65C20, 65C60

  • Copyright

COPYRIGHT: © Global Science Press

  • Email address
  • BibTex
  • RIS
  • TXT
@Article{CiCP-31-1049, author = {Wan , Xiaoliang and Wei , Shuangqing}, title = {VAE-KRnet and Its Applications to Variational Bayes}, journal = {Communications in Computational Physics}, year = {2022}, volume = {31}, number = {4}, pages = {1049--1082}, abstract = {

In this work, we have proposed a generative model, called VAE-KRnet, for density estimation or approximation, which combines the canonical variational autoencoder (VAE) with our recently developed flow-based generative model, called KRnet. VAE is used as a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. Using a linear model between the data and the latent variable, we show that VAE-KRnet can be more effective and robust than the canonical VAE. VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function (PDF) known up to a constant. VAE-KRnet is flexible in terms of dimensionality. When the number of dimensions is relatively small, KRnet can effectively approximate the distribution in terms of the original random variable. For high-dimensional cases, we may use VAE-KRnet to incorporate dimension reduction. One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution. The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler (KL) divergence between the model and the posterior. For high-dimensional distributions, it is very challenging to construct an accurate density model due to the curse of dimensionality, where extra assumptions are often introduced for efficiency. For instance, the classical mean-field approach assumes mutual independence between dimensions, which often yields an underestimated variance due to oversimplification. To alleviate this issue, we include into the loss the maximization of the mutual information between the latent random variable and the original random variable, which helps keep more information from the region of low density such that the estimation of variance is improved. Numerical experiments have been presented to demonstrate the effectiveness of our model.

}, issn = {1991-7120}, doi = {https://doi.org/10.4208/cicp.OA-2021-0087}, url = {http://global-sci.org/intro/article_detail/cicp/20376.html} }
TY - JOUR T1 - VAE-KRnet and Its Applications to Variational Bayes AU - Wan , Xiaoliang AU - Wei , Shuangqing JO - Communications in Computational Physics VL - 4 SP - 1049 EP - 1082 PY - 2022 DA - 2022/03 SN - 31 DO - http://doi.org/10.4208/cicp.OA-2021-0087 UR - https://global-sci.org/intro/article_detail/cicp/20376.html KW - Deep learning, variational Bayes, uncertainty quantification, Bayesian inverse problems, generative modeling. AB -

In this work, we have proposed a generative model, called VAE-KRnet, for density estimation or approximation, which combines the canonical variational autoencoder (VAE) with our recently developed flow-based generative model, called KRnet. VAE is used as a dimension reduction technique to capture the latent space, and KRnet is used to model the distribution of the latent variable. Using a linear model between the data and the latent variable, we show that VAE-KRnet can be more effective and robust than the canonical VAE. VAE-KRnet can be used as a density model to approximate either data distribution or an arbitrary probability density function (PDF) known up to a constant. VAE-KRnet is flexible in terms of dimensionality. When the number of dimensions is relatively small, KRnet can effectively approximate the distribution in terms of the original random variable. For high-dimensional cases, we may use VAE-KRnet to incorporate dimension reduction. One important application of VAE-KRnet is the variational Bayes for the approximation of the posterior distribution. The variational Bayes approaches are usually based on the minimization of the Kullback-Leibler (KL) divergence between the model and the posterior. For high-dimensional distributions, it is very challenging to construct an accurate density model due to the curse of dimensionality, where extra assumptions are often introduced for efficiency. For instance, the classical mean-field approach assumes mutual independence between dimensions, which often yields an underestimated variance due to oversimplification. To alleviate this issue, we include into the loss the maximization of the mutual information between the latent random variable and the original random variable, which helps keep more information from the region of low density such that the estimation of variance is improved. Numerical experiments have been presented to demonstrate the effectiveness of our model.

Xiaoliang Wan & Shuangqing Wei. (2022). VAE-KRnet and Its Applications to Variational Bayes. Communications in Computational Physics. 31 (4). 1049-1082. doi:10.4208/cicp.OA-2021-0087
Copy to clipboard
The citation has been copied to your clipboard