? Reference: Large-scale 3D Modeling from Crowdsourced Data https://demuc.de/tutorials/cvpr2017/
"Large-scale image-based 3D modeling has been a major goal of computer vision, enabling a wide range of applications including virtual reality, image-based localization, and autonomous navigation. One of the most diverse data sources for modeling is Internet photo collections. In the last decade, the computer vision community has made tremendous progress in large-scale structure-from-motion and multi-view stereo from Internet datasets. However, utilizing this wealth of information for 3D modeling remains a challenging problem due to the ever-increasing amount of image data. In a short period of time, research in large-scale modeling has progressed from modeling using several thousand images, to modeling from city-scale datasets of several million, and recently to reconstructing an Internet-scale dataset comprising 100 million images. This tutorial will present the main underlying technologies enabling these innovations."

Installation of COLMAP

COLMAP is a general-purpose Structure-from-Motion (SfM) and Multi-View Stereo (MVS) pipeline with a graphical and command-line interface. It offers a wide range of features for reconstruction of ordered and unordered image collections. The software is licensed under the new BSD license.
website: https://colmap.github.io/

Environment

ubuntu 20.04
gcc-7,g++-7
CUDA 10.2

Install Ceres Solver

Dependencies

Dependencies from the default Ubuntu repositories:

sudo apt-get install \
    git \
    cmake \
    build-essential \
    libboost-program-options-dev \
    libboost-filesystem-dev \
    libboost-graph-dev \
    libboost-system-dev \
    libboost-test-dev \
    libeigen3-dev \
    libsuitesparse-dev \
    libfreeimage-dev \
    libmetis-dev \
    libgoogle-glog-dev \
    libgflags-dev \
    libglew-dev \
    qtbase5-dev \
    libqt5opengl5-dev \
    libcgal-dev

Ceres-Solver 2.0.0

Important: should install Ceres-Solver with 2.0 or 2.1. Not use the latest source, it is not compatible to COLMAP!!.

if installed with wrong verison, when make the code, the error message is like:

*/usr/local/include/ceres/product_manifold.h:260:10: error: ‘exclusive_scan’ is not a member of ‘std’ std::exclusive_scan(values.begin(), values.end(), result.begin(), 0); ^~~~~~ src/CMakeFiles/colmap.dir/build.make:62: recipe for target ‘src/CMakeFiles/colmap.dir/base/camera.cc.o’ failed make[2]: ** [src/CMakeFiles/colmap.dir/base/camera.cc.o] Error 1

Avaliable versions: https://github.com/ceres-solver/ceres-solver/tags

Ceres Solver 1 is an open source C++ library for modeling and solving large, complicated optimization problems. It can be used to solve Non-linear Least Squares problems with bounds constraints and general unconstrained optimization problems. It is a mature, feature rich, and performant library that has been used in production at Google since 2010.

wget https://github.com/ceres-solver/ceres-solver/archive/refs/tags/2.0.0.tar.gz ceres-solver-2.0.0.tar.gz
tar zxf ceres-solver-2.0.0.tar.gz
mkdir build
cd build
cmake ..
make -j 8
sudo make install

Install COLMAP

git clone https://github.com/colmap/colmap.git
cd colmap
git checkout dev
mkdir build
cd build
cmake ..
make -j 8
sudo make install

Run COLMAP

colmap -h
colmap gui

The GUI looks like:

Sample use of COLMAP

Data preparison

Sample video can be downloaded from https://connectpolyu-my.sharepoint.com/:v:/g/personal/18048204r_connect_polyu_hk/EZUB0zD_f4lBuvcpJBtTcwsBcZgSjUK0WpwqAb9w7Hm3Fw?e=dYNd7U

Capture a short videos with a moving camera, and extract about 40 ~ 100 video frames.

Sample code for extract video frames via opencv:

import cv2

video_file = "FilmPool\\trimmed\\quidditch_pitch.mp4"
cap = cv2.VideoCapture(video_file)
fps = cap.get(cv2.CAP_PROP_FPS)
length = cap.get(cv2.CAP_PROP_FRAME_COUNT)
print("Video File %s ==> fps: %.2f, total: %d frames" % (video_file, fps, length))

video_start = int(0)  # start from index 0
video_end = video_start+120  # read the first 50000 frames
cap.set(1, video_start)

for frame_index in range(video_start, video_end):
    if not frame_index%3==0: continue
    else: cap.set(1, frame_index)
    ret, frame = cap.read()
    print("image_shape: ", frame.shape)
    # rsz_img = cv2.resize(frame, None, fx=0.5,fy=0.5)
    cv2.imwrite("dataset\\quidditch_pitch\\images\\%03d.png"%frame_index, frame)
    if not ret: break
    cv2.putText(frame, "Frame:" + str(frame_index), (20, 100), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2)
    cv2.imshow("frame", frame)
    cv2.waitKey(1)

Recover camera poses

Open COLMAP GUI through the following command in ubuntu

colmap gui

Click "File" and Create a "new Project". Here select the image folder and create a new database.

Feature Extraction

Click "Processing" ==> "Feature Extraction". Then extract with the default setting.

The log region will display the results.

Feature Matching

Click "Processing" ==> "Feature Matching". Then match the features with the default setting.
The log region will display the results.

Reconstruction

Click "Reconstruction" ==> "Start Reconstruction".

Command-line Interface via Python

Copied from "pose_utils.py" of LLFF(https://github.com/Fyusion/LLFF)[https://github.com/Fyusion/LLFF]

import os
import subprocess

def run_colmap(basedir, match_type):

    logfile_name = os.path.join(basedir, 'colmap_output.txt')
    logfile = open(logfile_name, 'w')

    feature_extractor_args = [
        'colmap', 'feature_extractor', 
            '--database_path', os.path.join(basedir, 'database.db'), 
            '--image_path', os.path.join(basedir, 'images'),
            '--ImageReader.single_camera', '1',
            # '--SiftExtraction.use_gpu', '0',
    ]
    feat_output = ( subprocess.check_output(feature_extractor_args, universal_newlines=True) )
    logfile.write(feat_output)
    print('Features extracted')

    exhaustive_matcher_args = [
        'colmap', match_type, 
            '--database_path', os.path.join(basedir, 'database.db'), 
    ]

    match_output = ( subprocess.check_output(exhaustive_matcher_args, universal_newlines=True) )
    logfile.write(match_output)
    print('Features matched')

    p = os.path.join(basedir, 'sparse')
    if not os.path.exists(p):
        os.makedirs(p)

    mapper_args = [
        'colmap', 'mapper',
            '--database_path', os.path.join(basedir, 'database.db'),
            '--image_path', os.path.join(basedir, 'images'),
            '--output_path', os.path.join(basedir, 'sparse'), # --export_path changed to --output_path in colmap 3.6
            '--Mapper.num_threads', '16',
            '--Mapper.init_min_tri_angle', '4',
            '--Mapper.multiple_models', '0',
            '--Mapper.extract_colors', '0',
    ]

    map_output = ( subprocess.check_output(mapper_args, universal_newlines=True) )
    logfile.write(map_output)
    logfile.close()
    print('Sparse map created')

    print( 'Finished running COLMAP, see {} for logs'.format(logfile_name) )

Troubleshooting

from https://github.com/Fyusion/LLFF#using-your-own-poses-without-running-colmap
COLMAP fails: If you see "Could not register, trying another image", you will probably have to try changing COLMAP optimization parameters or capturing more images of your scene. See (here)[https://github.com/Fyusion/LLFF/issues/8#issuecomment-498514411].