Taken SP25. Taught by Bharath Hariharan and Wei-Chiu Ma.
What if you want information about images? This can be done using machine learning. Specifically, classification of images is useful, and we need classification datasets to train models. MNIST was the first major dataset and consisted of handwritten numbers. Caltech 101 consisted of objects, but there were spurious correlations. ImageNet contains lots of labeled images.
So far, when talking about 3D structure from images, we have been talking about the depth of each pixel $D[\mathbf{p}] \in \mathbb{R}^+$. However, we can also look at the surface normal at any point because up close, any object approximates a plane. We can let this information be conveyed in another channel $N[\mathbf{p}] \in \mathbb{S}^2$. Sometimes, when things in images are too far, the depth channel can have less information, but we can use surface normal to fill in missing information to complete the depth map....
Structure From Motion If we have two cameras whose parameters we don’t know, can we recover camera location and orientation without a calibration object in order to recover the 3D world? Recall that $\mathbf{q}^TF\mathbf{p} = 0$. In order to find $F$, we can use an 8-point algorithm. Suppose $\mathbf{p} = \begin{bmatrix}x_1 \\ y_1 \\ 1\end{bmatrix}$ and $\mathbf{q} = \begin{bmatrix}x_2 \\ y_2 \\ 1\end{bmatrix}$ correspond to the same 3D point. Then, we have a linear system of equations with the entries of $F$ as coefficients....
Given two camera centers and a point in 3D space, we can define the plane containing these points. We call this the epipolar plane. The epipolar line for a camera is the intersection of the epipolar plane and the camera plane. We can also define the epipolar pencil, which is the family of planes formed by changing the point in 3D space. For the math, we assume the intrinsic parameters are the identity matrix, and we assume the world coordinate system is centered at the 1st camera pinhole with the $Z$ axis along the viewing direction....
Since we can set the world coordinates ourselves, we can let the world $Z$ coordinate be $0$, discarding the third column of the matrix $P$, resulting in a matrix we call $H$. $H$ has $9$ entries, but accounting for $\lambda$, there are only $8$ parameters. Thus, we need four points to determine $H$. Homography works for image stitching because it helps us map different camera angles onto the same coordinate plane....
It is hard to do 3D reconstruction with a single image because obviously we have lost information about depth. However, if we have two cameras, we can use parallax to triangulate. Assume we have two calibrated cameras and we have already run correspondence to find a pair of matching points: $$\begin{aligned} \begin{bmatrix} x_1 \\ y_1 \\ 1 \end{bmatrix} &\equiv P^{(1)}\begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix} \\ \begin{bmatrix} x_2 \\ y_2 \\ 2 \end{bmatrix} &\equiv P^{(2)}\begin{bmatrix} X \\ Y \\ Z \\ 1 \end{bmatrix} \end{aligned} $$...
Recap of Aligning Coordinate Systems Recall that $\mathbf{x}_c = R\mathbf{x}_w + \mathbf{t}$. We first find $R$: $$R = \begin{bmatrix}X_c & Y_c & Z_c\end{bmatrix}^T.$$ Once we find this, if we let $\mathbf{c}$ be the location of the camera, then $$R\mathbf{c} + \mathbf{t} = 0 \implies \mathbf{t} = -R\mathbf{c}.$$ Then, the extrinsic (world to camera) matrix is $$\begin{bmatrix} R & \mathbf{t} \\ \mathbf{0}^T & 1 \end{bmatrix}.$$ Overall, we have $$ \begin{bmatrix} u \\ v \\ 1\end{bmatrix} \equiv \begin{bmatrix} a_{11} & a_{12} & a_{13} \\ a_{21} & a_{22} & a_{23} \\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} f & 0 & 0 & 0 \\ 0 & f & 0 & 0 \\ 0 & 0 & 1 & 0 \end{bmatrix} \begin{bmatrix} R & \mathbf{t} \\ \mathbf{0}^T & 1 \end{bmatrix} \begin{bmatrix} U \\ V \\ W \\ 1\end{bmatrix}, $$...
How do we go from the film plane (projected image) to pixel array? Recall that for the latter, we place the origin in the top left. However, for the former, the origin is in the middle. Thus, to go from the former to the latter, we need to include an offset: $$u = f\frac{X}{Z} + o_x, \quad v = f\frac{Y}{Z} +o_y.$$ This assumes that the focal length $f$ is the same in each direction....
Recall our projection equations $$x = \frac{X}{Z}, \quad y = \frac{Y}{Z}.$$ However, because we have division, this is not linear. How can we represent it like $$\mathbf{x}_c = R\mathbf{x}_w + \mathbf{t}?$$ This brings us to homogeneous coordinates, whereby 2D points are actually 3D points and 3D points are actually 4D points. For example, we can write $$(x,y) = (x,y,1) = (kx, ky, k).$$ This allows us to write the transformation above as...
Say we have a line passing through a point $A$ with direction $D$. Then, we can write $$Q(\lambda) = A + \lambda D.$$ Then, we can have a parallel line $$R(\lambda) = B + \lambda D.$$ Then, as $\lambda \to \infty$, the projections of both lines go to $(D_X/D_Z, D_Y/D_Z)$. If $D_Z = 0$, it means the lines lie parallel to the $xy$-plane. Additionally, we can take the limit of planes....