Question

View morphing and interpolation Implement automatic view morphing, i.e., compute two-frame structure from motion and then use these results to generate a smooth animation from one image to the next (Section 7.2.3). 1\. Decide how to represent your 3D scene, e.g., compute a Delaunay triangulation of the matched point and decide what to do with the triangles near the border. (Hint: try fitting a plane to the scene, e.g., behind most of the points.) 2\. Compute your in-between camera positions and orientations. 3\. Warp each triangle to its new location, preferably using the correct perspective projection (Szeliski and Shum 1997). 4\. (Optional) If you have a denser 3D model (e.g., from stereo), decide what to do at the "cracks". 5\. (Optional) For a non-rigid scene, e.g., two pictures of a face with different expressions, not all of your matched points will obey the epipolar geometry. Decide how to handle them to achieve the best effect.

Short answer

The exercise involves implementing view morphing - transitioning smoothly from one image to another using structure from motion. This involves creating a 3D scene representation (possibly using Delaunay triangulation), determining in-between camera positions and orientations, warping triangles to their new location (with correct perspective projection), along with optional steps of handling cracks in denser 3D models and managing non-rigid scenes.

Step-by-step solution

3D Scene Representation

Start by deciding how to represent a 3D scene. Compute the matched points and get a Delaunay triangulation of these matched points. To handle the triangles near the border, consider fitting a plane to the scene, preferably behind most of the points.

Compute Camera Positions and Orientations

Compute the in-between camera positions and orientations. Since the view will be morphing between two frames, it is essential to calculate the camera's in-between positions and orientations in the 3D scene.

Triangle Warping

Warp each triangle to its new location. Use the correct perspective projection (as per the one suggested by Szeliski and Shum in 1997) to achieve the right image perspective during view morphing.

Handling Denser 3D Model (Optional)

If a denser 3D model is available (for example, from stereo imaging), then decide on what to do at the 'cracks'. This refers to how to handle the places where there might be gaps in the 3D model.

Handling Non-Rigid Scenes (Optional)

In the case of a non-rigid scene, such as two pictures of a face with different expressions, not all matched points will obey the epipolar geometry. Decide how to handle these points to achieve the best view-morphing effect.

Key concepts

3D Scene Representation

A 3D scene representation is crucial for tasks involving view morphing. It involves creating a model of the real-world scene in a 3-dimensional space that computers can process. To effectively represent a scene, we must first identify

• Key reference points within the images
• Determine their corresponding 3D positions

An excellent way to establish these points is by using a technique called Delaunay Triangulation. This method helps break down the scene into smaller triangles, formed by connecting the matched points.
For fitting planes and handling the edges of these triangles near the scene's border, it’s helpful to imagine that the scene lies on a flat surface, a little behind the main points of interest.
This assumption helps smooth out the representation, ensuring the scene transitions smoothly during morphing.

Delaunay Triangulation

Delaunay Triangulation is a mathematical tool used to connect points in a plane to form a network of triangles. This network has the advantage of ensuring that no point is inside the circumcircle of any triangle, optimizing the connections

• Ensuring that the triangles are as equilateral (i.e., balanced) as possible
• Highlighting connectivity among scene points

In the context of view morphing, Delaunay Triangulation is utilized to create a mesh-like framework over the matched points in our images.
The triangulation facilitates easier calculations for morphing, as each triangle can be independently transformed using perspective projection techniques. By leveraging this triangulation, you not only identify the spatial relationships of the points but also create a robust starting point for morphing transitions.

Perspective Projection

Perspective projection is an essential concept that helps in converting 3D scenes into 2D images, mimicking how our eyes perceive the world. In view morphing, it is used for warping triangles accurately, ensuring the final visual is realistic.
This process involves understanding how the camera's position and orientation affect the view. For each triangle identified in Delaunay Triangulation:

• Apply the perspective transformation, shifting triangles to new positions
• Ensure the perspective matches how the camera would"see" at these stages

By using the method proposed by Szeliski and Shum, triangles maintain their proportionality, preventing distortions and creating a smoother morph. This part of the process is central to making transitions believable, as maintaining perspective helps preserve depth and orientation cues in both original and morphed views.

Epipolar Geometry

Epipolar geometry deals with the geometric relationship between two views of a scene. It involves the study of the intrinsic projective geometry between a pair of images

• Facilitating understanding of cameras' relations
• Helping in determining how points in one image correspond to points in another

For view morphing, understanding epipolar geometry is key, especially when handling nonrigid scenes where matched points might not align perfectly. This understanding enables us to accommodate differences like varying facial expressions across images.
By creatively managing these variations, we achieve more refined results in morphing, allowing for adjustments that ensure changes in the scene, like expression or viewpoint shifts, to appear natural. Epipolar constraints offer the mathematical backbone to adjust points and enforce consistency across morphs.