Pano2GS: Single-View Panoramic 3D Gaussian Splatting with Learned Geometric Priors

We address the problem of generating complete 3D scene representations from single panoramic images. While existing methods require multi-view inputs or expensive per-scene optimization, we propose Pano2GS, a feedforward architecture that directly regresses 3D Gaussian Splat parameters from a single equirectangular image. Our key insight is that spherical projections contain sufficient geometric cues when combined with learned priors from large-scale 3D data. We introduce a Distilled Prior Network (DPN) trained on 500K panorama-3DGS pairs, a spherical vision encoder with geodesic attention, and a cascaded refinement pipeline including gap filling, density upsampling, and volumetric completion modules. Experiments on Matterport3D, Replica, and our new Pano3D-500K benchmark demonstrate state-of-the-art reconstruction quality with 50× faster inference than optimization-based alternatives.

Given an equirectangular panorama I ∈ ℝ^(H×W×3), our method produces a set of 3D Gaussians G = {(μᵢ, Σᵢ, αᵢ, cᵢ)} representing the complete scene. The pipeline consists of: (1) a spherical vision encoder E_sph that extracts distortion-aware features, (2) a Distilled Prior Network that provides geometric priors from the latent space, (3) a Gaussian prediction head that regresses initial point clouds, and (4) cascaded refinement modules for gap filling, upsampling, and volumetric completion.

We train a variational autoencoder on 500K panorama-3DGS pairs to learn a compressed representation of indoor scene geometry. The encoder maps panoramas to a 512-dimensional latent code z, while the decoder reconstructs 3DGS parameters. At inference, the encoder provides geometric priors that guide the Gaussian prediction head, enabling plausible reconstruction of occluded regions based on learned scene statistics.

Initial coarse predictions are refined through learned upsampling. Our density-aware splitting network analyzes local geometry to determine optimal Gaussian subdivision strategies. Unlike naive interpolation, the network learns to add detail where it matters—edges, textures, and complex surfaces—while preserving smooth regions. We achieve 2× and 4× upsampling with <10mm and <18mm Chamfer-L1 error respectively.

We introduce Pano3D-500K, a large-scale dataset of 500,000 panorama-3DGS pairs for training and evaluation. The dataset combines: (1) 200K rendered pairs from synthetic environments (Replica, HM3D, Gibson), (2) 200K pairs from real captures with multi-view 3DGS reconstruction (Matterport3D, RealEstate10K), and (3) 100K augmented samples with domain randomization. All data is released under CC-BY-4.0.

Current limitations include: (1) reduced quality on outdoor scenes with distant geometry, (2) tendency to hallucinate plausible but incorrect content in heavily occluded regions, (3) fixed maximum scene scale. Future directions include video input for temporal consistency, text-conditioned generation for creative applications, and integration with VLMs for semantic scene understanding.

This work was supported by [Funding Agency] under Grant [Number]. Compute resources were provided by [Cloud Provider]. We thank the authors of 3D Gaussian Splatting, Matterport3D, and HuggingFace for their open-source contributions that enabled this research.