? Reference: Large-scale 3D Modeling from Crowdsourced Data https://demuc.de/tutorials/cvpr2017/
"Large-scale image-based 3D modeling has been a major goal of computer vision, enabling a wide range of applications including virtual reality, image-based localization, and autonomous navigation. One of the most diverse data sources for modeling is Internet photo collections. In the last decade, the computer vision community has made tremendous progress in large-scale structure-from-motion and multi-view stereo from Internet datasets. However, utilizing this wealth of information for 3D modeling remains a challenging problem due to the ever-increasing amount of image data. In a short period of time, research in large-scale modeling has progressed from modeling using several thousand images, to modeling from city-scale datasets of several million, and recently to reconstructing an Internet-scale dataset comprising 100 million images. This tutorial will present the main underlying technologies enabling these innovations."
Installation of COLMAP
COLMAP is a general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline with a graphical and command-line interface. It offers a wide range of features for reconstruction of ordered and unordered image collections. The software is licensed under the new BSD license.
website: https://colmap.github.io/
Environment
- ubuntu 20.04
- gcc-7,g++-7
- CUDA 10.2
Install Ceres Solver
Dependencies
Dependencies from the default Ubuntu repositories:
sudo apt-get install \
git \
cmake \
build-essential \
libboost-program-options-dev \
libboost-filesystem-dev \
libboost-graph-dev \
libboost-system-dev \
libboost-test-dev \
libeigen3-dev \
libsuitesparse-dev \
libfreeimage-dev \
libmetis-dev \
libgoogle-glog-dev \
libgflags-dev \
libglew-dev \
qtbase5-dev \
libqt5opengl5-dev \
libcgal-dev
Ceres-Solver 2.0.0
Important: should install Ceres-Solver with 2.0 or 2.1. Not use the latest source, it is not compatible to COLMAP!!.
if installed with wrong verison, when make the code, the error message is like:
*/usr/local/include/ceres/product_manifold.h:260:10: error: ‘exclusive_scan’ is not a member of ‘std’ std::exclusive_scan(values.begin(), values.end(), result.begin(), 0); ^
~~~~~ src/CMakeFiles/colmap.dir/build.make:62: recipe for target ‘src/CMakeFiles/colmap.dir/base/camera.cc.o’ failed make[2]: ** [src/CMakeFiles/colmap.dir/base/camera.cc.o] Error 1
- Avaliable versions: https://github.com/ceres-solver/ceres-solver/tags
Ceres Solver 1 is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems. It is a mature, feature rich, and performant library that has been used in production at Google since 2010.
wget https://github.com/ceres-solver/ceres-solver/archive/refs/tags/2.0.0.tar.gz ceres-solver-2.0.0.tar.gz
tar zxf ceres-solver-2.0.0.tar.gz
mkdir build
cd build
cmake ..
make -j 8
sudo make install
Install COLMAP
git clone https://github.com/colmap/colmap.git
cd colmap
git checkout dev
mkdir build
cd build
cmake ..
make -j 8
sudo make install
Run COLMAP
colmap -h
colmap gui
The GUI looks like:
Sample use of COLMAP
Data preparison
Sample video can be downloaded from https://connectpolyu-my.sharepoint.com/:v:/g/personal/18048204r_connect_polyu_hk/EZUB0zD_f4lBuvcpJBtTcwsBcZgSjUK0WpwqAb9w7Hm3Fw?e=dYNd7U
Capture a short videos with a moving camera, and extract about 40 ~ 100 video frames.
Sample code for extract video frames via opencv:
import cv2
video_file = "FilmPool\\trimmed\\quidditch_pitch.mp4"
cap = cv2.VideoCapture(video_file)
fps = cap.get(cv2.CAP_PROP_FPS)
length = cap.get(cv2.CAP_PROP_FRAME_COUNT)
print("Video File %s ==> fps: %.2f, total: %d frames" % (video_file, fps, length))
video_start = int(0) # start from index 0
video_end = video_start+120 # read the first 50000 frames
cap.set(1, video_start)
for frame_index in range(video_start, video_end):
if not frame_index%3==0: continue
else: cap.set(1, frame_index)
ret, frame = cap.read()
print("image_shape: ", frame.shape)
# rsz_img = cv2.resize(frame, None, fx=0.5,fy=0.5)
cv2.imwrite("dataset\\quidditch_pitch\\images\\%03d.png"%frame_index, frame)
if not ret: break
cv2.putText(frame, "Frame:" + str(frame_index), (20, 100), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)
cv2.imshow("frame", frame)
cv2.waitKey(1)
Recover camera poses
Open COLMAP GUI through the following command in ubuntu
colmap gui
Click "File" and Create a "new Project". Here select the image folder and create a new database.
Feature Extraction
Click "Processing" ==> "Feature Extraction". Then extract with the default setting.
The log region will display the results.
Feature Matching
Click "Processing" ==> "Feature Matching". Then match the features with the default setting.
The log region will display the results.
Reconstruction
Click "Reconstruction" ==> "Start Reconstruction".
Command-line Interface via Python
Copied from "pose_utils.py" of LLFF(https://github.com/Fyusion/LLFF)[https://github.com/Fyusion/LLFF]
import os
import subprocess
def run_colmap(basedir, match_type):
logfile_name = os.path.join(basedir, 'colmap_output.txt')
logfile = open(logfile_name, 'w')
feature_extractor_args = [
'colmap', 'feature_extractor',
'--database_path', os.path.join(basedir, 'database.db'),
'--image_path', os.path.join(basedir, 'images'),
'--ImageReader.single_camera', '1',
# '--SiftExtraction.use_gpu', '0',
]
feat_output = ( subprocess.check_output(feature_extractor_args, universal_newlines=True) )
logfile.write(feat_output)
print('Features extracted')
exhaustive_matcher_args = [
'colmap', match_type,
'--database_path', os.path.join(basedir, 'database.db'),
]
match_output = ( subprocess.check_output(exhaustive_matcher_args, universal_newlines=True) )
logfile.write(match_output)
print('Features matched')
p = os.path.join(basedir, 'sparse')
if not os.path.exists(p):
os.makedirs(p)
mapper_args = [
'colmap', 'mapper',
'--database_path', os.path.join(basedir, 'database.db'),
'--image_path', os.path.join(basedir, 'images'),
'--output_path', os.path.join(basedir, 'sparse'), # --export_path changed to --output_path in colmap 3.6
'--Mapper.num_threads', '16',
'--Mapper.init_min_tri_angle', '4',
'--Mapper.multiple_models', '0',
'--Mapper.extract_colors', '0',
]
map_output = ( subprocess.check_output(mapper_args, universal_newlines=True) )
logfile.write(map_output)
logfile.close()
print('Sparse map created')
print( 'Finished running COLMAP, see {} for logs'.format(logfile_name) )
Troubleshooting
from https://github.com/Fyusion/LLFF#using-your-own-poses-without-running-colmap
COLMAP fails: If you see "Could not register, trying another image", you will probably have to try changing COLMAP optimization parameters or capturing more images of your scene. See (here)[https://github.com/Fyusion/LLFF/issues/8#issuecomment-498514411].