Abstract
In recent times, we have witnessed remarkable progress in the realm of 3D video technology, allowing us to capture the three-dimensional surface data of moving individuals without resorting to specific markers or costumes. This article introduces innovative 3D algorithms for human sensing based on 3D video. These algorithms facilitate the extraction of essential information, including body movement and viewing directions, devoid of any disruptions caused by the sensing system itself.
Introduction
The fast-paced evolution of 3D video technology has brought about a revolution in capturing objects in motion, including human subjects, eliminating the need for attaching specialized markers or donning specific outfits [3] [1] [2] [4]. This presents a remarkable advantage over other motion capture technologies, making 3D video highly suitable for the three-dimensional digital preservation of human motion, including intangible cultural assets. However, it is important to note that 3D video, in and of itself, merely provides unstructured three-dimensional surface data, akin to pixel streams in conventional 2D video. The objective of this article is to demonstrate the utilization of raw 3D video in sensing human activity.
3D Video
The term "3D video" or "free viewpoint video" encompasses two distinct approaches documented in the existing literature. The first approach, known as "model-based" methods, entails reconstructing the three-dimensional shape of the object and rendering it, much like computer-generated imagery (CG) [2] [4]. The second approach, referred to as "image-based" methods, involves interpolating a two-dimensional image at a virtual camera position directly from two-dimensional multi-viewpoint images. When it comes to 3D human sensing, model-based approaches prove more appropriate since image-based methods do not provide three-dimensional information. The estimation of the three-dimensional shape within model-based approaches remains a classic yet unresolved problem in computer vision.
Figure 1:
Workflow of 3D video capture. Multiple viewpoint silhouettes derived from the
object's images offer an initial estimation of the object's shape, termed as
the "visual hull." This estimation is subsequently refined by
optimizing photo-consistency among input images. The texture-mapped
three-dimensional surface enables the generation of virtual images from
arbitrary viewpoints.
Estimating the
original three-dimensional shape from its two-dimensional projections presents
an inherently challenging problem. However, recent years have witnessed the
proposal of numerous practical algorithms that address this issue by
integrating traditional stereo matching and shape-from-silhouette techniques,
resulting in the generation of a comprehensive three-dimensional shape termed
the "photo hull". Figure 1 provides an illustration of our 3D video
capture scheme. The top and second rows showcase an example of multi-viewpoint
input images and their corresponding object regions, respectively. The visual
hull of the object is then computed using multi-viewpoint silhouettes, as
depicted in the third row. Through photo-consistency optimization, we refine
the visual hull to obtain the optimal three-dimensional surface of the object,
as shown in the fourth row. Finally, textures are mapped onto the
three-dimensional surface, as evidenced in the bottom row, which presents a
sample rendering of the final three-dimensional surface estimated from the
multi-viewpoint images.
Estimating Kinematic Structure from 3D Video
This section introduces an algorithm designed to estimate the kinematic structure of an articulated object captured as 3D video. The input comprises a time-series of three-dimensional surfaces, which serves as the basis for building the kinematic structure. Let Mt represent the input three-dimensional surface at time t (Figure 2(a)). Initially, we construct the Reeb graph [6] of Mt, as depicted in Figure 2(b). The Reeb graph is computed based on the integral of geodesic distances on Mt, resulting in a graph structure that resembles the kinematic structure. Figure 2(a) showcases the surface segmentation based on the integral of geodesic distances.
However, it is
worth noting that the definition of the Reeb graph does not guarantee that all
graph edges remain within Mt; some edges may extend beyond it. To tackle this
issue, we modify the Reeb graph to ensure it remains enclosed within Mt. This
modified graph is known as the "pseudo Endoskeleton Reeb Graph"
(pERG), as illustrated in Figure 2(c). Initially, pERGs are constructed for
each frame, from which we select "seed" pERGs that exhibit no
degeneration in their body parts. Identifying seed pERGs relies on a simple
assumption, requiring them to possess five branches, given our focus on human
behavior.
Subsequently, we
perform pERG-to-pERG fitting, starting from seed frames and progressing to
their neighboring frames. The seed frame is deformed to match its neighbors,
and this process continues until the fitting error exceeds a specific
threshold. This process yields topologically isomorphic intervals for each seed
frame, as depicted at the top of Figure 3. Within each interval, node
clustering is employed to identify the articulated structure (Figure 4).
Finally, articulated structures estimated across all intervals are integrated
into a unified kinematic structure. Figure 2(d) and 5 present the final unified
kinematic structure estimated purely from the input 3D surface sequence.
Visibility of the Model Surface
Firstly, we introduce a visibility definition for the model M(p) using collision detection between body parts. Regions where collisions occur generally evade detection by any cameras, as exemplified in Figure 8. The color depicts the distance between a point and its nearest surface point of other body parts. Leveraging this distance and visibility, we define the reliability of M(p) as follows:
Visibility of the Observed Surface
Next, we introduce the concept of visibility for the observed surface, Mt. Since Mt is estimated from multi-viewpoint images, the vertices on Mt can be categorized based on the number of cameras that can observe them. If a vertex v can be observed by one or fewer cameras, it is deemed non-photo-consistent, and its position is interpolated based on neighboring vertices. Conversely, if two or more cameras can observe vertex v, it is considered photo-consistent, and its three-dimensional position is explicitly estimated using stereo matching. Consequently, the number of observable cameras for a vertex reflects the reliability of its three-dimensional position.
Conclusion
This article has introduced human activity sensing algorithms based on 3D video. The algorithms encompass three fundamental aspects: (1) global kinematic structure, (2) intricate motion estimation, and (3) detailed estimation of face and eye directions. Importantly, these algorithms facilitate non-contact sensing, eliminating the need for special markers or costumes on the subject. This represents a significant advantage of our 3D video-based sensing approach.
Hashtags/Keywords/Labels:
#3DHumanSensing
#3DVideo #KinematicStructure #MotionCapture #ComputerVision
References/Resources:
1. T. Kanade, P. Rander, and P. J. Narayanan. "Virtualized reality: Constructing virtual worlds from real scenes." IEEE Multimedia, 1997.
2. T. Matsuyama,
X. Wu, T. Takai, and S. Nobuhara. "Real-time 3D shape reconstruction,
dynamic 3D mesh deformation and high fidelity visualization for 3D video."
CVIU, 2004.
3. S. Moezzi,
L.-C. Tai, and P. Gerard. "Virtual view generation for 3D digital
video." IEEE Multimedia, 1997.
4. J. Starck and
A. Hilton. "Surface capture for performance-based animation." IEEE
Computer Graphics and Applications, 2007.
For more such Seminar articles click index
– Computer Science Seminar Articles list-2023.
[All images are taken
from Google Search or respective reference sites.]
…till
next post, bye-bye and take care.
