Skip to content

react-viewer: enable 3D point cloud view (RSDEV-9239) - #15110

Merged
remibettan merged 9 commits into
realsenseai:developmentfrom
remibettan:react-viewer-3d
Jun 25, 2026
Merged

remibettan merged 9 commits into
realsenseai:developmentfrom
remibettan:react-viewer-3d

Conversation

@remibettan

@remibettan remibettan commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

Tracked by: RSDEV-9239

Summary

Phase 2 / 3D view of the React viewer (RSDEV-9239). The 3D View button was
permanently disabled with a "coming soon" tooltip; the underlying pipeline
existed but was not wired up and the backend never produced point cloud frames
in sensor mode. This PR makes 3D actually work end-to-end:

  • Enable the 3D View button (Header.tsx); the checkbox in PointCloudViewer
    is replaced by auto-activation when the user enters 3D and deactivation
    on leaving (store.setViewMode side-effect).
  • Backend: compute point cloud in sensor mode too (rs_manager._process_sensor_frame),
    not just pipeline mode. The is_pointcloud_enabled flag was a no-op for the
    per-sensor streaming path the DevicePanel uses.
  • Bandwidth: a 640×480 point cloud is ~5 MB / frame and was being shoved
    through Socket.IO at 30 fps, stalling the server.
    • decimate verts[::4]
    • throttle point_cloud emission to every 3rd metadata broadcast (10 fps)
    • raise socketio.AsyncServer max_http_buffer_size to 10 MB
  • PointCloudViewer rendering: flip Y/Z to convert RealSense (y-down, z-forward)
    to Three.js (y-up, z-toward-camera); replace the custom blue→red gradient
    with a histogram-equalized jet colormap matching rs.colorizer()'s
    default histogram_eq=1.0; move initial camera + OrbitControls target
    so the cloud is centered.
  • App.tsx: keep StreamViewer and PointCloudViewer both mounted across view
    toggle (visibility-swap rather than conditional render) so the WebRTC
    peer survives 3D → 2D switches — no more "Connecting…" flash.

Not in scope (future PR): color-to-depth texture alignment.

Test plan

  • Start streaming depth (sensor mode via DevicePanel) → click 3D View → point cloud renders within a frame
  • Cloud is oriented correctly (Y-up, scene in front of viewer) and colored across the full jet range, similar to the 2D depth thumbnail
  • Switch back to 2D → no "Connecting…" delay, WebRTC video resumes instantly
  • Server stays responsive (no stalls) for several minutes of continuous 3D streaming
  • Export PLY button still produces a valid .ply

Copilot AI review requested due to automatic review settings May 28, 2026 07:25
metadata["width"] = depth_frame.get_width()
metadata["height"] = depth_frame.get_height()

if self.is_pointcloud_enabled.get(device_id, False):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Point-cloud vertex processing duplicated in _process_sensor_frame and _collect_frames; extract into a shared helper to avoid duplicated conversion/filter/decimation logic.

Details

✨ AI Reasoning
​A non-trivial block that builds point-cloud metadata (reading points.get_vertices(), converting to numpy, filtering by z, decimating, assigning metadata['point_cloud']) now appears twice in the same file: once in the pipeline frame loop and once in the sensor-frame processor. This duplicates business logic that would need updating in two places if changed and should be consolidated.

🔧 How do I fix it?
Delete extra code. Extract repeated code sequences into reusable functions or methods. Use loops or data structures to eliminate repetitive patterns.

Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore: fixed - extracted point cloud vertex processing into a shared _build_point_cloud_metadata() helper called from both _collect_frames (pipeline) and _process_sensor_frame (sensor) in 6c977c0.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

fixed - extracted point cloud vertex processing into a shared _build_point_cloud_metadata() helper called from both _collect_frames (pipeline) and _process_sensor_frame (sensor) in 6c977c0.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR wires up end-to-end 3D point cloud viewing in the React-based REST API viewer, enabling the previously “coming soon” 3D mode and adding backend support to produce/transmit point cloud frames efficiently without stalling the server.

Changes:

  • Frontend: enables the 3D View toggle, keeps 2D/3D viewers mounted across switches, and updates point-cloud rendering (axis flip + histogram-equalized jet colormap).
  • Frontend store: toggling 2D↔3D now activates/deactivates point cloud generation via the REST API.
  • Backend: computes point clouds in sensor mode as well, reduces payload (decimation + throttled emission), and increases Socket.IO buffer size to accommodate frames.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
wrappers/rest-api/tools/react-viewer/tests/unit/components/Header.test.tsx Updates unit test to reflect 3D View being enabled.
wrappers/rest-api/tools/react-viewer/src/store/index.ts Adds view-mode side effects to enable/disable point cloud and clear vertices on exit.
wrappers/rest-api/tools/react-viewer/src/components/PointCloudViewer.tsx Removes manual enable checkbox; improves point cloud orientation and coloring.
wrappers/rest-api/tools/react-viewer/src/components/Header.tsx Enables 3D View button and hooks it up to setViewMode('3d').
wrappers/rest-api/tools/react-viewer/src/App.tsx Keeps StreamViewer and PointCloudViewer mounted, swapping visibility to preserve WebRTC.
wrappers/rest-api/app/services/socketio.py Raises Socket.IO max_http_buffer_size to allow point cloud payloads.
wrappers/rest-api/app/services/rs_manager.py Decimates point cloud vertices in pipeline mode; adds point cloud generation in sensor mode.
wrappers/rest-api/app/services/metadata_socket_server.py Throttles inclusion of point_cloud in broadcasts (every Nth update).

Comment on lines +1092 to +1096
const activeDevices = Object.values(get().deviceStates).filter(ds => ds.isActive)
if (mode === '3d') {
for (const ds of activeDevices) {
apiClient
.enablePointCloud(ds.device.device_id)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Partially addressed in 6c977c0: setViewMode is now async, Promise.all-awaits the enable/disable calls across active devices, and routes failures through set({ error }) so they surface in the existing toast UI. The request-id / cancellation guard for rapid toggles is left for a follow-up - in practice users do not flip 2D/3D fast enough to race the round-trip, and a proper guard is more naturally addressed in the upcoming GPU-unproject refactor that removes the enable/disable lifecycle entirely.

Comment on lines +95 to +99
metadata["point_cloud"]["vertices"] = base64.b64encode(
metadata["point_cloud"]["vertices"].tobytes()
).decode("utf-8")
else:
del metadata["point_cloud"]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c977c0. Outer metadata dict and point_cloud sub-dict are now shallow-copied before base64-encoding or deleting the point_cloud key, so the cached metadata in rs_manager queues is no longer mutated.

Comment on lines 1286 to 1290
v, t = points.get_vertices(), points.get_texture_coordinates()
verts = np.asanyarray(v).view(np.float32).reshape(-1, 3)
verts = verts[verts[:, 2] >= 0.03] # Filter out z < 0.03
verts = verts[::4] # decimate to keep payload under Socket.IO buffer cap
metadata["point_cloud"] = {"vertices": verts, "texture_coordinates": []}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 6c977c0. get_texture_coordinates() removed; the helper now calls get_vertices() only (texture_coordinates was already serialized as an empty array).

Comment on lines +7 to +8
# max_http_buffer_size raised to 10 MB to fit point cloud frames (default 1 MB is too small).
sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins='*', max_http_buffer_size=10_000_000)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Acknowledged - deferring. The REST API server is intended for trusted local/LAN use (not public exposure); cors_allowed_origins='*' already reflects that posture. Making the buffer size env-configurable is reasonable hygiene and we will roll it into the upcoming refactor (RSDEV ticket follow-up to #9239) that switches the 3D path off the Socket.IO point-cloud channel altogether, removing the need for a large buffer.

remibettan added a commit to remibettan/librealsense that referenced this pull request May 31, 2026
remibettan added a commit to remibettan/librealsense that referenced this pull request Jun 1, 2026
@remibettan
remibettan requested a review from AviaAv June 1, 2026 10:20
@Nir-Az

Nir-Az commented Jun 4, 2026

Copy link
Copy Markdown
Collaborator

@remibettan can you add a gif/image of this feature in the description?

remibettan added a commit to remibettan/librealsense that referenced this pull request Jun 14, 2026
remibettan added a commit to remibettan/librealsense that referenced this pull request Jun 18, 2026
"""The core loop that fetches and broadcasts metadata."""
print(f"[MetadataBroadcaster] Starting broadcast loop for {device_id}...")

# Per-loop counter so each device's throttle is independent — a shared

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MetadataSocketServer._broadcast_metadata_loop gained throttling, metadata copying and base64 encoding responsibilities, increasing its complexity; consider extracting throttling/serialization into helpers.

Details

✨ AI Reasoning
​1. The change introduced an independent per-broadcaster counter, point-cloud throttling logic, copying/encoding of binary buffers, and conditional deletion of point_cloud metadata inside the broadcast loop.
2. _broadcast_metadata_loop already coordinates status queries, per-stream metadata collection, error mapping, payload assembly and emission — adding binary encoding and throttle control mixes serialization/performance policy into the loop's control flow.
3. This increases the cognitive load for the loop and makes the function responsible for both collecting metadata and non-trivial decisions about payload composition and encoding frequency.
4. Separating the throttling and serialization (encoding) into helper functions would keep the loop easier to read and reduce maintenance burden, but the current change clearly increased the function's responsibilities.

🔧 How do I fix it?
Break down long functions into smaller helper functions. Aim for functions under 60 lines with fewer than 10 local variables.

Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AikidoSec ignore: ok

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Based on your feedback, we ignored this issue because of the following reason:

ok

@AviaAv AviaAv left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@remibettan
remibettan merged commit 947a40d into realsenseai:development Jun 25, 2026
29 of 30 checks passed
@remibettan
remibettan deleted the react-viewer-3d branch June 28, 2026 19:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants