Repository navigation
react-viewer: enable 3D point cloud view (RSDEV-9239) - #15110
Conversation
| metadata["width"] = depth_frame.get_width() | ||
| metadata["height"] = depth_frame.get_height() | ||
|
|
||
| if self.is_pointcloud_enabled.get(device_id, False): |
There was a problem hiding this comment.
Point-cloud vertex processing duplicated in _process_sensor_frame and _collect_frames; extract into a shared helper to avoid duplicated conversion/filter/decimation logic.
Details
✨ AI Reasoning
A non-trivial block that builds point-cloud metadata (reading points.get_vertices(), converting to numpy, filtering by z, decimating, assigning metadata['point_cloud']) now appears twice in the same file: once in the pipeline frame loop and once in the sensor-frame processor. This duplicates business logic that would need updating in two places if changed and should be consolidated.
🔧 How do I fix it?
Delete extra code. Extract repeated code sequences into reusable functions or methods. Use loops or data structures to eliminate repetitive patterns.
Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
There was a problem hiding this comment.
@AikidoSec ignore: fixed - extracted point cloud vertex processing into a shared _build_point_cloud_metadata() helper called from both _collect_frames (pipeline) and _process_sensor_frame (sensor) in 6c977c0.
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
fixed - extracted point cloud vertex processing into a shared _build_point_cloud_metadata() helper called from both _collect_frames (pipeline) and _process_sensor_frame (sensor) in 6c977c0.
There was a problem hiding this comment.
Pull request overview
This PR wires up end-to-end 3D point cloud viewing in the React-based REST API viewer, enabling the previously “coming soon” 3D mode and adding backend support to produce/transmit point cloud frames efficiently without stalling the server.
Changes:
- Frontend: enables the 3D View toggle, keeps 2D/3D viewers mounted across switches, and updates point-cloud rendering (axis flip + histogram-equalized jet colormap).
- Frontend store: toggling 2D↔3D now activates/deactivates point cloud generation via the REST API.
- Backend: computes point clouds in sensor mode as well, reduces payload (decimation + throttled emission), and increases Socket.IO buffer size to accommodate frames.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| wrappers/rest-api/tools/react-viewer/tests/unit/components/Header.test.tsx | Updates unit test to reflect 3D View being enabled. |
| wrappers/rest-api/tools/react-viewer/src/store/index.ts | Adds view-mode side effects to enable/disable point cloud and clear vertices on exit. |
| wrappers/rest-api/tools/react-viewer/src/components/PointCloudViewer.tsx | Removes manual enable checkbox; improves point cloud orientation and coloring. |
| wrappers/rest-api/tools/react-viewer/src/components/Header.tsx | Enables 3D View button and hooks it up to setViewMode('3d'). |
| wrappers/rest-api/tools/react-viewer/src/App.tsx | Keeps StreamViewer and PointCloudViewer mounted, swapping visibility to preserve WebRTC. |
| wrappers/rest-api/app/services/socketio.py | Raises Socket.IO max_http_buffer_size to allow point cloud payloads. |
| wrappers/rest-api/app/services/rs_manager.py | Decimates point cloud vertices in pipeline mode; adds point cloud generation in sensor mode. |
| wrappers/rest-api/app/services/metadata_socket_server.py | Throttles inclusion of point_cloud in broadcasts (every Nth update). |
| const activeDevices = Object.values(get().deviceStates).filter(ds => ds.isActive) | ||
| if (mode === '3d') { | ||
| for (const ds of activeDevices) { | ||
| apiClient | ||
| .enablePointCloud(ds.device.device_id) |
There was a problem hiding this comment.
Partially addressed in 6c977c0: setViewMode is now async, Promise.all-awaits the enable/disable calls across active devices, and routes failures through set({ error }) so they surface in the existing toast UI. The request-id / cancellation guard for rapid toggles is left for a follow-up - in practice users do not flip 2D/3D fast enough to race the round-trip, and a proper guard is more naturally addressed in the upcoming GPU-unproject refactor that removes the enable/disable lifecycle entirely.
| metadata["point_cloud"]["vertices"] = base64.b64encode( | ||
| metadata["point_cloud"]["vertices"].tobytes() | ||
| ).decode("utf-8") | ||
| else: | ||
| del metadata["point_cloud"] |
There was a problem hiding this comment.
Fixed in 6c977c0. Outer metadata dict and point_cloud sub-dict are now shallow-copied before base64-encoding or deleting the point_cloud key, so the cached metadata in rs_manager queues is no longer mutated.
| v, t = points.get_vertices(), points.get_texture_coordinates() | ||
| verts = np.asanyarray(v).view(np.float32).reshape(-1, 3) | ||
| verts = verts[verts[:, 2] >= 0.03] # Filter out z < 0.03 | ||
| verts = verts[::4] # decimate to keep payload under Socket.IO buffer cap | ||
| metadata["point_cloud"] = {"vertices": verts, "texture_coordinates": []} |
There was a problem hiding this comment.
Fixed in 6c977c0. get_texture_coordinates() removed; the helper now calls get_vertices() only (texture_coordinates was already serialized as an empty array).
| # max_http_buffer_size raised to 10 MB to fit point cloud frames (default 1 MB is too small). | ||
| sio = socketio.AsyncServer(async_mode="asgi", cors_allowed_origins='*', max_http_buffer_size=10_000_000) |
There was a problem hiding this comment.
Acknowledged - deferring. The REST API server is intended for trusted local/LAN use (not public exposure); cors_allowed_origins='*' already reflects that posture. Making the buffer size env-configurable is reasonable hygiene and we will roll it into the upcoming refactor (RSDEV ticket follow-up to #9239) that switches the 3D path off the Socket.IO point-cloud channel altogether, removing the need for a large buffer.
6c977c0 to
cb4985a
Compare
|
@remibettan can you add a gif/image of this feature in the description? |
527ed58 to
493f31e
Compare
493f31e to
6e355aa
Compare
| """The core loop that fetches and broadcasts metadata.""" | ||
| print(f"[MetadataBroadcaster] Starting broadcast loop for {device_id}...") | ||
|
|
||
| # Per-loop counter so each device's throttle is independent — a shared |
There was a problem hiding this comment.
MetadataSocketServer._broadcast_metadata_loop gained throttling, metadata copying and base64 encoding responsibilities, increasing its complexity; consider extracting throttling/serialization into helpers.
Details
✨ AI Reasoning
1. The change introduced an independent per-broadcaster counter, point-cloud throttling logic, copying/encoding of binary buffers, and conditional deletion of point_cloud metadata inside the broadcast loop.
2. _broadcast_metadata_loop already coordinates status queries, per-stream metadata collection, error mapping, payload assembly and emission — adding binary encoding and throttle control mixes serialization/performance policy into the loop's control flow.
3. This increases the cognitive load for the loop and makes the function responsible for both collecting metadata and non-trivial decisions about payload composition and encoding frequency.
4. Separating the throttling and serialization (encoding) into helper functions would keep the loop easier to read and reduce maintenance burden, but the current change clearly increased the function's responsibilities.
🔧 How do I fix it?
Break down long functions into smaller helper functions. Aim for functions under 60 lines with fewer than 10 local variables.
Reply @AikidoSec feedback: [FEEDBACK] to get better review comments in the future.
Reply @AikidoSec ignore: [REASON] to ignore this issue.
More info
There was a problem hiding this comment.
✅ Based on your feedback, we ignored this issue because of the following reason:
ok
…2D colors, head-on camera
23a1da6 to
cb71958
Compare
Tracked by: RSDEV-9239
Summary
Phase 2 / 3D view of the React viewer (RSDEV-9239). The 3D View button was
permanently disabled with a "coming soon" tooltip; the underlying pipeline
existed but was not wired up and the backend never produced point cloud frames
in sensor mode. This PR makes 3D actually work end-to-end:
is replaced by auto-activation when the user enters 3D and deactivation
on leaving (store.setViewMode side-effect).
not just pipeline mode. The is_pointcloud_enabled flag was a no-op for the
per-sensor streaming path the DevicePanel uses.
through Socket.IO at 30 fps, stalling the server.
verts[::4]to Three.js (y-up, z-toward-camera); replace the custom blue→red gradient
with a histogram-equalized jet colormap matching
rs.colorizer()'sdefault
histogram_eq=1.0; move initial camera + OrbitControls targetso the cloud is centered.
toggle (visibility-swap rather than conditional render) so the WebRTC
peer survives 3D → 2D switches — no more "Connecting…" flash.
Not in scope (future PR): color-to-depth texture alignment.
Test plan
.ply