Consistent Generative Query Networks
Ananya Kumar, S. M. Ali Eslami, Danilo Jimenez Rezende et al. · arXiv (Cornell University) · 2018
Stochastic video prediction models take in a sequence of image frames, and generate a sequence of consecutive future image frames. These models typically generate future frames in an autoregressive fashion, which is slow and requires the input and output frames to be consecutive. We introduce a model that overcomes these drawbacks by generating a latent representation from an arbitrary set of frames that can then be used to simultaneously and efficiently sample temporally consistent frames at arbitrary time-points. For example, our model can "jump" and directly sample frames at the end of the video, without sampling intermediate frames. Synthetic video evaluations confirm substantial gains in speed and functionality without loss in fidelity. We also apply our framework to a 3D scene reconstruction dataset. Here, our model is conditioned on camera location and can sample consistent sets of images for what an occluded region of a 3D scene might look like, even if there are multiple possibilities for what that region might contain. Reconstructions and videos are available at https://bit.ly/2O4Pc4R.
Cited by 15 works. doi:10.48550/arxiv.1807.02033 Open-access PDF
Authors
- Ananya Kumar
- S. M. Ali Eslami
- Danilo Jimenez Rezende
- Marta Garnelo
- Fabio Viola
- Edward Lockhart
- Murray Shanahan
References
- GAN(Generative Adversarial Nets)
- Auto-Encoding Variational Bayes
- Siamese Neural Networks for One-shot Image Recognition
- DRAW: A Recurrent Neural Network For Image Generation
- Unsupervised Learning of Video Representations using LSTMs
- Visual Modeling with a Hand-Held Camera
- Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
- 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction
- Video Frame Synthesis Using Deep Voxel Flow
- Multi-view 3D Models from Single Images with a Convolutional Network