Question

Light field transformations Derive the equations relating regular images to \(4 \mathrm{D}\) light field coordinates. 1\. Determine the mapping between the far plane \((u, v)\) coordinates and a virtual camera's \((x, y)\) coordinates. (a) Start by parameterizing a 3D point on the \(u v\) plane in terms of its \((u, v)\) coordinates. (b) Project the resulting 3D point to the camera pixels \((x, y, 1)\) using the usual \(3 \times 4\) camera matrix \(\boldsymbol{P}\) (2.63). (c) Derive the \(2 \mathrm{D}\) homography relating \((u, v)\) and \((x, y)\) coordinates. 2\. Write down a similar transformation for \((s, t)\) to \((x, y)\) coordinates. 3\. Prove that if the virtual camera is actually on the \((s, t)\) plane, the \((s, t)\) value depends only on the camera's optical center and is independent of \((x, y)\). 4\. Prove that an image taken by a regular orthographic or perspective camera, i.e., one that has a linear projective relationship between 3D points and \((x, y)\) pixels (2.63), samples the \((s, t, u, v)\) light field along a two- dimensional hyperplane.

Short answer

The mapping between the far plane \((u, v)\) coordinates and a virtual camera's \((x, y)\) coordinates and the similar transformation for \((s, t)\) to \((x, y)\) coordinates can be determined through parameterizating 3D points and using the camera matrix \(P\). The \(2 \mathrm{D}\) homography that relates these coordinates can be derived using the first 3 columns of P. If the camera is actually on the \((s, t)\) plane, then the \((s, t)\) values depend only on the camera's optical center. A regular perspective camera with a linear projective relationship between 3D and pixel coordinates samples the \((s, t, u, v)\) light field along a two-dimensional hyperplane.

Step-by-step solution

- Parameterizing a 3D point on the uv plane

A 3D point \((X, Y, Z)\) can be parameterized on the \(uv\) plane using its coordinates \((u, v)\) as \((X, Y, Z) = (u, v, 1)\).

- Project the 3D point to the camera pixels

The 3D point can be projected onto the camera pixels \((x, y, 1)\) using the usual \(3 \times 4\) camera matrix \(P\). The equation can be postulated as follows: \([x, y, 1]^T = P * [u, v, 1]^T\).

- Derive the 2D homography

A 2D homography is a linear transformation that maps points in one plane to another. For our camera, this mapping happens from \((u, v)\) coordinates to \((x, y)\) coordinates. It can be derived from the previous step by isolating (x, y) in the camera matrix as \(H = P[:, :3]\), which is the first 3 columns of P.

- Transforming (s, t) to (x, y) coordinates

Just like how we transformed \((u, v)\) to \((x, y)\), we also transform \((s, t)\) to \((x, y)\). This uses the same method as above and just substitutes \((u, v)\) for \((s, t)\).

- Proof of (s, t) value dependence

If the virtual camera is on the \((s, t)\) plane, the \((s, t)\) value is indeed only dependent on the camera's optical center. This is because the optical center defines the view direction, and essentially the point of view, from which the image is projected.

- Prove that an image taken by a regular camera samples the light field along a 2D hyperplane

From the above steps, we have established that the camera maps \((u, v, s, t)\) to \((x, y, s, t)\). A regular orthographic or perspective camera does indeed sample the 4D light field along a 2D hyperplane because, with a linear projective relationship between 3D points and pixel coordinates, it is, in effect projecting the 4D light field onto a 2D plane, hence sampling along a 2D hyperplane.

Key concepts

3D Point Parameterization

Understanding the 3D point parameterization is critical when dealing with computer vision and graphics, as it allows us to express the position of a point in three-dimensional space. This process involves assigning a set of coordinates to a specific point within a defined coordinate system. In the context of transforming light fields, the parameterization of a 3D point on the UV plane is given simply by its UV coordinates, implying that every point on the UV plane is at the same depth.

When we establish a point in 3D space using parameters, such as the UV coordinates, we express it as \( (X, Y, Z) = (u, v, 1)\). This says that each point on the UV plane can be represented as a 3D point with a specific Z-coordinate, often set to 1 for simplification, resembling an orthographic projection. This step forms the basis for further transformations, as it provides a link between 2D image data and 3D scene representations.

2D Homography

A 2D homography is a crucial concept in computer vision, particularly in the process of image warping, stabilization, and perspective corrections. It refers to a transformation that maps points on one plane to another plane, preserving the straight lines. This process can be especially useful in establishing correspondences between different views of the same scene.

In the light field transformation exercise, after parameterizing 3D points on the UV plane, we use the camera matrix to project these points onto a camera's pixel coordinates \( (x, y, 1)\). The resulting transformation matrix, known as the homography matrix \(H\), is crucial because it encapsulates all the information needed to map between the UV plane and the pixel coordinates. Understanding and computing 2D homography allows one to reproject images taken from one viewpoint to that of another, fundamentally altering the perceived perspective.

Camera Matrix

The camera matrix plays an instrumental role in the process of mapping 3D world coordinates to 2D image coordinates. It is essentially a 3x4 matrix that translates the 3D points in space to their corresponding 2D points on a camera sensor. This matrix is often denoted as \(P\) and encapsulates the intrinsic parameters (like focal length and skew) and extrinsic parameters (like rotation and translation) of the camera.

Using the

camera matrix

in the context of light field transformations, we transform the parameterized 3D points on the UV plane onto the camera's pixel coordinates. This operation is expressed mathematically by multiplying the camera matrix \(P\) with the vector representing the 3D point, yielding the pixel coordinates on the camera's sensor. The camera matrix is the bridge between the 3D points and the final 2D image that is captured, and includes the effects of the camera lens and sensor.

Projective Relationship

The term projective relationship pertains to how a three-dimensional structure is represented on a two-dimensional plane through projection. The essence of this concept lies in how points, lines, and planes in the 3D space are connected to their 2D counterparts. In the realm of photography and imaging, this relationship is fundamental because camera lenses are designed to project a 3D scene onto a 2D sensor.

The representation of a 3D point in 2D space depends on the projection model used by the camera. For example, perspective projection considers the depth, while orthographic projection ignores it, approximating the view of the scene as if all the objects are at a similar distance from the camera. These projective relationships need to be well-understood when interpreting the resulting images, developing computer vision algorithms, and creating visual effects that mimic the real-world behavior of light and cameras.

Orthographic Camera

An orthographic camera captures scenes without accounting for perspective. In essence, objects far away from the camera appear the same size as those close to it. This camera model simplifies many calculations in computer graphics, assuming that rays of light are parallel and do not converge.

In the context of light field transformations, when using an orthographic camera model, the projective relationship between 3D points and the 2D image plane is linear. This simplifies the process of mapping from the 4D light field coordinates to the 2D image plane, as perspective effects and depth-induced distortions are not present. As such, orthographic cameras are particularly useful for technical and architectural visualizations where accurate proportions are required.

Perspective Camera

Conversely, a perspective camera captures images by simulating the way human eyes perceive depth, which results in objects farther away from the camera appearing smaller—the phenomenon known as perspective. This camera model is more complex than its orthographic counterpart because it involves a converging point for light rays (the vanishing point).

During the transformation of light field coordinates to a 2D image using a perspective camera, the depth of each point in the 3D world is considered, resulting in a non-linear projective relationship. An image captured by a perspective camera embodies the dynamics of a 4D light field and scales the 3D world onto a 2D plane while preserving the illusion of depth. Hence, this model is highly relevant when aiming to create photorealistic renders and in the field of photography for an authentic representation of spatial relationships.