Interactive visualization

This tutorial introduces user interaction features of the visualizer window.

 27# examples/python/visualization/interactive_visualization.py
 28
 29import numpy as np
 30import copy
 31import open3d as o3d
 32
 33
 34def demo_crop_geometry():
 35    print("Demo for manual geometry cropping")
 36    print(
 37        "1) Press 'Y' twice to align geometry with negative direction of y-axis"
 38    )
 39    print("2) Press 'K' to lock screen and to switch to selection mode")
 40    print("3) Drag for rectangle selection,")
 41    print("   or use ctrl + left click for polygon selection")
 42    print("4) Press 'C' to get a selected geometry and to save it")
 43    print("5) Press 'F' to switch to freeview mode")
 44    pcd = o3d.io.read_point_cloud("../../test_data/ICP/cloud_bin_0.pcd")
 45    o3d.visualization.draw_geometries_with_editing([pcd])
 46
 47
 48def draw_registration_result(source, target, transformation):
 49    source_temp = copy.deepcopy(source)
 50    target_temp = copy.deepcopy(target)
 51    source_temp.paint_uniform_color([1, 0.706, 0])
 52    target_temp.paint_uniform_color([0, 0.651, 0.929])
 53    source_temp.transform(transformation)
 54    o3d.visualization.draw_geometries([source_temp, target_temp])
 55
 56
 57def pick_points(pcd):
 58    print("")
 59    print(
 60        "1) Please pick at least three correspondences using [shift + left click]"
 61    )
 62    print("   Press [shift + right click] to undo point picking")
 63    print("2) After picking points, press 'Q' to close the window")
 64    vis = o3d.visualization.VisualizerWithEditing()
 65    vis.create_window()
 66    vis.add_geometry(pcd)
 67    vis.run()  # user picks points
 68    vis.destroy_window()
 69    print("")
 70    return vis.get_picked_points()
 71
 72
 73def demo_manual_registration():
 74    print("Demo for manual ICP")
 75    source = o3d.io.read_point_cloud("../../test_data/ICP/cloud_bin_0.pcd")
 76    target = o3d.io.read_point_cloud("../../test_data/ICP/cloud_bin_2.pcd")
 77    print("Visualization of two point clouds before manual alignment")
 78    draw_registration_result(source, target, np.identity(4))
 79
 80    # pick points from two point clouds and builds correspondences
 81    picked_id_source = pick_points(source)
 82    picked_id_target = pick_points(target)
 83    assert (len(picked_id_source) >= 3 and len(picked_id_target) >= 3)
 84    assert (len(picked_id_source) == len(picked_id_target))
 85    corr = np.zeros((len(picked_id_source), 2))
 86    corr[:, 0] = picked_id_source
 87    corr[:, 1] = picked_id_target
 88
 89    # estimate rough transformation using correspondences
 90    print("Compute a rough transform using the correspondences given by user")
 91    p2p = o3d.pipelines.registration.TransformationEstimationPointToPoint()
 92    trans_init = p2p.compute_transformation(source, target,
 93                                            o3d.utility.Vector2iVector(corr))
 94
 95    # point-to-point ICP for refinement
 96    print("Perform point-to-point ICP refinement")
 97    threshold = 0.03  # 3cm distance threshold
 98    reg_p2p = o3d.pipelines.registration.registration_icp(
 99        source, target, threshold, trans_init,
100        o3d.pipelines.registration.TransformationEstimationPointToPoint())
101    draw_registration_result(source, target, reg_p2p.transformation)
102    print("")
103
104
105if __name__ == "__main__":
106    demo_crop_geometry()
107    demo_manual_registration()

This script executes two applications of user interaction: demo_crop_geometry and demo_manual_registration.

Crop geometry

34def demo_crop_geometry():
35    print("Demo for manual geometry cropping")
36    print(
37        "1) Press 'Y' twice to align geometry with negative direction of y-axis"
38    )
39    print("2) Press 'K' to lock screen and to switch to selection mode")
40    print("3) Drag for rectangle selection,")
41    print("   or use ctrl + left click for polygon selection")
42    print("4) Press 'C' to get a selected geometry and to save it")
43    print("5) Press 'F' to switch to freeview mode")
44    pcd = o3d.io.read_point_cloud("../../test_data/ICP/cloud_bin_0.pcd")
45    o3d.visualization.draw_geometries_with_editing([pcd])

This function simply reads a point cloud and calls draw_geometries_with_editing. This function provides vertex selection and cropping.

Note

Open3D has a VisualizerWithEditing class that inherits Visualizer class. It adds graphic user interaction features. Likewise examples in Customized visualization, VisualizerWithEditing() can be explicitly used instead of draw_geometries_with_editing([pcd]).

Once a geometry is displayed, press Y twice to align geometry with negative direction of y-axis. After adjusting viewing orientation, press K to lock screen and to switch to the selection mode.

../../_images/crop_lock.png

Tip

The practical step for selecting area is to align the geometry with arbitrary axis using orthographic projection model. This trick makes selection easier, because it avoids self-occlusion hassle due to perspective projection.

To select a region, use either mouse drag (rectangle selection) or ctrl + left mouse click (polygon selection). The below example shows a selected area using a polygon.

../../_images/crop_selection.png

Note that the selected area is dark shaded. To keep the selected area and discard the rest, press C. A dialog box appears, which can be used to save the cropped geometry. The cropping result is shown after saving.

../../_images/crop_save.png ../../_images/crop_result.png

To finish selection mode, press F to switch to freeview mode.

../../_images/crop_freeview.png

Manual registration

Select correspondences

The following script registers two point clouds using point-to-point ICP. It gets initial alignment via user interaction.

73def demo_manual_registration():
74    print("Demo for manual ICP")
75    source = o3d.io.read_point_cloud("../../test_data/ICP/cloud_bin_0.pcd")
76    target = o3d.io.read_point_cloud("../../test_data/ICP/cloud_bin_2.pcd")
77    print("Visualization of two point clouds before manual alignment")
78    draw_registration_result(source, target, np.identity(4))
79
80    # pick points from two point clouds and builds correspondences
81    picked_id_source = pick_points(source)
82    picked_id_target = pick_points(target)

The script reads two point clouds, and visualizes the point clouds before alignment.

../../_images/manual_icp_initial.png
57def pick_points(pcd):
58    print("")
59    print(
60        "1) Please pick at least three correspondences using [shift + left click]"
61    )
62    print("   Press [shift + right click] to undo point picking")
63    print("2) After picking points, press 'Q' to close the window")
64    vis = o3d.visualization.VisualizerWithEditing()
65    vis.create_window()
66    vis.add_geometry(pcd)
67    vis.run()  # user picks points
68    vis.destroy_window()
69    print("")
70    return vis.get_picked_points()

The function pick_points(pcd) makes an instance of VisualizerWithEditing. To mimic draw_geometries, it creates windows, adds the geometry, visualizes the geometry, and then terminates. A novel interface function from VisualizerWithEditing is get_picked_points() that returns the indices of user-picked vertices.

To pick a vertex, press shift + left click on a window. If a vertex is selected, the visualizer window overlays a sphere on a selected vertex. For example, after picking three vertices in the source point cloud, it shows:

../../_images/manual_icp_source.png

This will print:

Picked point #58481 (2.14, 1.56, 1.53) to add in queue.
Picked point #77321 (2.86, 1.92, 1.09) to add in queue.
Picked point #42639 (3.28, 1.53, 1.45) to add in queue.

Press Q to close the window. The next step is to pick the same correspondences in the target point cloud. The color of the sphere helps to identify the same correspondence.

../../_images/manual_icp_target.png

This will print:

Picked point #54028 (1.62, 1.81, 1.23) to add in queue.
Picked point #97115 (2.45, 2.19, 1.11) to add in queue.
Picked point #47467 (2.75, 1.71, 1.45) to add in queue.

Tip

To get a good registration result, try to pick more than three points that are well-distributed in the scene. Using a vertex in the corner region is a good way to easily pick the right correspondence.

Registration using user correspondences

 83    assert (len(picked_id_source) >= 3 and len(picked_id_target) >= 3)
 84    assert (len(picked_id_source) == len(picked_id_target))
 85    corr = np.zeros((len(picked_id_source), 2))
 86    corr[:, 0] = picked_id_source
 87    corr[:, 1] = picked_id_target
 88
 89    # estimate rough transformation using correspondences
 90    print("Compute a rough transform using the correspondences given by user")
 91    p2p = o3d.pipelines.registration.TransformationEstimationPointToPoint()
 92    trans_init = p2p.compute_transformation(source, target,
 93                                            o3d.utility.Vector2iVector(corr))
 94
 95    # point-to-point ICP for refinement
 96    print("Perform point-to-point ICP refinement")
 97    threshold = 0.03  # 3cm distance threshold
 98    reg_p2p = o3d.pipelines.registration.registration_icp(
 99        source, target, threshold, trans_init,
100        o3d.pipelines.registration.TransformationEstimationPointToPoint())
101    draw_registration_result(source, target, reg_p2p.transformation)
102    print("")

The later part of the demo computes an initial transformation based on the user-provided correspondences. This script builds pairs of correspondences using Vector2iVector(corr). It utilizes TransformationEstimationPointToPoint.compute_transformation to compute the initial transformation from the correspondences. The initial transformation is refined using registration_icp.

The registration result is as follows:

../../_images/manual_icp_alignment.png