Semantic Structure From Motion

Categorie: 3D Scanning Publicat: 02 Septembrie 2015

Scris de Alex Accesări: 870

Semantic Structure From Motion (SSFM) Sid Yingze Bao, Mohit Bagra, Yu-Wei Chao, and Silvio Savarese Computer Vision Lab, University of Michigan at Ann Arbor
Traditional Structure from motion (SFM) aims at jointly recovering the structure of a scene as a collection of 3D points and estimating the camera poses from a number of input images. In this project, called Semantic Structure from Motion (SSFM) , we generalize this concept: not only do we want to recover 3D points, but also recognize and estimate the location of high level semantic scene components such as regions and objects in 3D. As a key ingredient for this joint inference problem, we seek to model various types of interactions between scene components. Such interactions help regularize our solution and obtain more accurate results than solving these problems in isolation. Experiments on public datasets demonstrate that: 1) our framework estimates camera poses more robustly than SFM algorithms that use points only; 2) our framework is capable of accurately estimating pose and location of objects, regions, and points in the 3D scene; 3) our framework recognizes objects and regions more accurately than state-of-the-art single image recognition methods. Check out our paper for details!
Researchers who are interested in methods for 3D reconstruction from multiple views, object detection and recognition, scene segmentation as well as in applications such as autonomous navigation, robotics, object manipulation and surveillance. Papers and citations
S. Yingze Bao, M. Bagra, Y. Chao, and S. Savarese, Semantic Structure from Motion with Points, Regions, and Objects, Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2012 download pdf and bibtex S. Yingze Bao and S. Savarese, Semantic Structure from Motion: a Novel Framework for Joint Object Recognition and 3D Reconstruction, book chapter in "Outdoor and Large-Scale Real-World Scene Analysis", Springer, in press download pdf S. Yingze Bao and S. Savarese, Semantic Structure from Motion, Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, 2011 download pdf (long version) and bibtex S. Yingze Bao, M. Bagra, S. Savarese, Semantic Structure From Motion with Object and Point Interactions, IEEE Workshop on Challenges and Opportunities in Robot Perception (in conjunction with ICCV-11) download pdf and bibtex. Winner of the best student paper award
Sid Yingze Bao is a 4th year PhD student in the Vision Lab at the University of Michigan, at Ann Arbor, EECS department Silvio Savarese is an assistant professor of Electrical and Computer Engineering at U-M and director of the Vision Lab.
Below are 3 YouTube videos that illustrate the ability of SSFM to recover the structure of a scene from multiple images and highlight the important semantic phenomena. For more results please refer to our papers. Video credits: Mohit Bagra
Soure code June 7 2011, version 0.1 See README.txt for installation. This code accompanies our CVPR 11 paper.
Data sets Kinect dataset (testing set) (version 0.2, date: Sep 29). See README.txt in. Email the author for obtaining training set. Object detector models are included. Ford Car Dataset (version 0.3, date: Oct 20). This dataset is a joint effort of Pandey et al. (for collecting images, Lidar points, calibration etc.) and us (for annotation of 2D and 3D objects). So please cite both papers if you appreciate the authors' effort.