Scene-level 3D object detection

Setup

1. Data. The ten CA-1M validation sequences listed in the Boxer appendix [1] (45662921, 45261179, 47115543, 45261143, 45261615, 42897545, 45261133, 42897552, 45663113, 42897521, with 629 frames and 856 target objects) and the ADT sequence Apartment_release_work_seq106_M1292 (1WM103600M1292_optitrack_release_work_seq106 in the Boxer appendix, with 92 frames and 135 static target objects). NymeriaPlus is not included yet.
2. Frames. We take one frame per second, the one closest to each whole second of the capture. Every method receives the same frames, and sequence IDs and timestamps are stored with the inputs.
3. Inputs. Track A gives GT camera poses and sensor depth (laser-registered poses with ARKit LiDAR depth on CA-1M, GT poses and depth on ADT), the RGB-D input of Boxer. Track B gives RGB only, with poses and depth from Pi3X, one shared geometry per sequence for all methods, and all boxes are mapped to the GT world by one similarity per sequence fitted on depth (Pi3X scale 0.90–1.10, 2–5 cm residual). In track B the baselines also receive the true gravity direction, as an IMU would provide, while Mundo estimates gravity itself.
4. Ground truth. World-level boxes of the objects visible in the evaluated frames. CA-1M follows the CA-1M loader of Boxer, so floor, wall and shelter boxes and boxes with a side over 3 m are ignored, and thin sides of GT and predictions are expanded to 5 cm. ADT follows the ADT loader of Boxer, so a static object is a target if, in at least one frame, its visibility is at least 0.3, its 2D box is at least 2% of the image and its centre is within 5 m of the camera. Every other observed object is ignored.
5. Metrics. Class-agnostic precision, recall and F1 at 3D IoU 0.25 and 0.5, with one-to-one Hungarian matching of oriented boxes. A prediction on an ignored object is neither a true nor a false positive. Dataset numbers pool all matches over the sequences.
6. Baselines. Boxer [1] runs the official OWLv2 detector with the LVIS+ prompts, BoxerNet and the offline fusion. CuTR [2] runs its own detector with the RGB-D and the RGB model, and its per-frame boxes are fused with the Boxer fusion. SpatialLM 1.1 [3] reads the coloured point cloud of the same frames. For Boxer and CuTR we report the official fusion, the score threshold that maximises F1 on the four dev sequences, and, as an upper bound for the baselines, the threshold that maximises F1 on the test sequences.
7. Mundo. The reconstruction stage of the release (0.7.0rc3) with one configuration for every sequence and both tracks, namely the request “give every distinct physical object its own box” and Boxer lifts kept from a single view. The single-view setting was chosen on four further CA-1M validation sequences that are not part of the test set. Proxy mode, the default for video generation, is reported in the ablation.
8. Protocol. Boxer has not released its frame lists or scoring code, so all numbers here are re-run under this unified protocol. The reproduction check at the end compares the baselines' mAP with the published values.

Results on CA-1M

Mundo leads in both tracks and at both IoU thresholds. With GT poses and depth (track A) it reaches F1 0.508 at IoU 0.25, against 0.461 for CuTR RGB-D, which is trained on CA-1M. From RGB alone (track B) the margin grows to 9.8 points. Boxer and CuTR are listed with the official fusion, with a threshold chosen on the dev split and with the test-oracle threshold, and bold marks the best F1 in each column. A paired bootstrap over the ten sequences confirms the margins. The 95% interval of the F1 difference excludes zero against every baseline operating point in both tracks and at both thresholds, with one exception, CuTR RGB-D at its test-oracle threshold at IoU 0.5 in track A (+0.013, interval -0.013 to +0.044). With the threshold chosen on the dev split the same comparison gives +0.029 (+0.001 to +0.055).

Results on ADT

ADT seq106 has 135 static target objects in one apartment. Mundo leads at IoU 0.25 in both tracks and is on par with Boxer at IoU 0.5 in track A (0.189 against 0.186).

ADT, six sequences

Boxer evaluates ADT on a single sequence. To give the ADT comparison statistical weight we add five sequences of the same apartment chosen by a fixed rule, the lowest-numbered single-user sequence of each activity (clean, decoration, meal, recognition, work) other than seq106. Together with seq106 they form ADT-6, with 562 frames and 914 static target objects, evaluated with the same configuration and the same thresholds. At IoU 0.25 Mundo leads in both tracks (0.393 and 0.298 against 0.355 and 0.279 for the strongest baseline). At IoU 0.5 it is on par with Boxer in track A (0.202 against 0.197) and behind the official Boxer fusion in track B (0.132 against 0.175). Pi3X is less accurate on these Aria sequences than on CA-1M (scale off by 5 to 17%, 3 to 9 cm residual after alignment), and the official four-view support keeps fewer and tighter boxes, which pays off at the strict threshold.

Where the gain comes from

Mundo starts from Boxer lifts on its own geometry and from SAM3 masks fused in 3D. The agent keeps, merges, corrects and adds boxes against the photographs. On CA-1M track B it raises precision from 0.354 to 0.571 while recall moves from 0.471 to 0.442, and F1 at IoU 0.25 from 0.404 to 0.498. With the official four-view support of Boxer most lifts are discarded at 1 Hz, and keeping single-view lifts adds 3.2 points of F1 at IoU 0.5. Proxy mode keeps only the main bodies, with high precision and few small objects.

Recall by object size

Recall at IoU 0.25 by GT volume on CA-1M, track B (small under 0.01 m³, large over 0.1 m³). Small objects such as switches, bottles and frames make up half of the CA-1M targets and remain the main source of misses for every method.

Scenes in 3D

Four sequences in track B (estimated geometry), namely the three CA-1M sequences with the largest F1 margin of Mundo over the strongest baseline, and the ADT sequence. Points show the reconstructed geometry of the evaluated frames, shaded by height. Baseline boxes are shown at their best-F1 thresholds. Drag to orbit and use +/− or ctrl+wheel to zoom.

59 frames at 1 Hz and 60 GT objects, track B (estimated geometry). F1 at IoU 0.25 is 0.74 for Mundo (mean of 3 runs, the run with the median F1 is shown) and 0.55 for the strongest baseline.

70 frames at 1 Hz and 83 GT objects, track B (estimated geometry). F1 at IoU 0.25 is 0.55 for Mundo (mean of 3 runs, the run with the median F1 is shown) and 0.43 for the strongest baseline.

115 frames at 1 Hz and 190 GT objects, track B (estimated geometry). F1 at IoU 0.25 is 0.42 for Mundo (mean of 3 runs, the run with the median F1 is shown) and 0.30 for the strongest baseline.

92 frames at 1 Hz and 135 GT objects, track B (estimated geometry). F1 at IoU 0.25 is 0.27 for Mundo (mean of 3 runs, the run with the median F1 is shown) and 0.22 for the strongest baseline.

Per-sequence results

F1 at IoU 0.25 for each sequence, with F1 at IoU 0.5 as the smaller grey number. Boxer and CuTR RGB-D are shown at their best-F1 thresholds, Mundo as the mean over runs.

Reproduction check

Scene-level class-agnostic mAP over IoU 0.05 to 0.50 for the baselines in track A, which matches the input of the RGB+D rows in Boxer Table 3 [1]. The orders of magnitude agree, with higher values on CA-1M and lower values on ADT. Our CuTR run uses its own detector instead of OWLv2 boxes, and CA-1M is its training domain. The scene-level GT filtering of the paper is not published, so exact agreement is not expected.