← Kushal Kusram projects Waymo Open Dataset Toolkit GitHub

A reproducible perception benchmark for the Waymo Open Dataset

Camera, LiDAR and camera + LiDAR detection under one protocol, an official-metric LiDAR evaluation path, error analysis, and a camera tracking-by-detection baseline with a failure replay. It is a perception experimentation toolkit, not an autonomy stack.

Left: camera detection and tracking (YOLOv8n 640x960 + SORT). Right: LiDAR bird's-eye view, ground truth (green) against CenterPoint predictions (orange). Different models in each panel (not fusion); low-budget baselines. Validation data, 6 seconds, shown for illustration.

Results

Camera detection on the FRONT camera, COCO mAP@[.5:.95] on 10 frozen validation segments (1,967 frames), one RTX 4080 SUPER. Dots are individual seeds, bars are means. Settings are defined in the protocol: S is the small common budget, L has a larger input and more data.

LiDAR models (different metric, setting S, one seed)

ModelBEV proxy APOfficial 3-D L1 AP (vehicle / pedestrian / cyclist)Latency
PointPillars0.0880.048 / 0.014 / 0405 ms
CenterPoint0.1020.048 / 0.013 / 0397 ms

LiDAR scores are a different metric from camera mAP and cannot be ranked against it. Under the official Waymo metric the two LiDAR models tie, so the proxy's ranking of CenterPoint ahead is not supported. See why the official AP is low.

Benchmark protocol (v1.3)

The protocol fixes how every model is trained and scored, so a difference can be attributed to the model and not to the data, budget or metric.

SettingTraining segmentsCamera inputScheduleRun for
Sfirst 20 of 798 (2.5%)512 x 512, squashed10 passes over 20 segmentsall camera models, fusion, LiDAR
S + resolutionfirst 20640 x 960, aspect kept10 passes over 20 segmentsYOLOv8n (ablation)
Lfirst 100 of 798 (12.5%)640 x 960, aspect kept3 passes over 100 segmentsYOLOv8n, RT-DETR-L

What limits performance

Next experiments worth running: fusion and the LiDAR models at the larger budget and with more seeds; scoring the official metric within 50 m and a height-weighted LiDAR loss; a shorter track confirmation delay and appearance features to re-link tracks after gaps; more cyclist data.

Camera tracking baseline

A SORT-style tracker (constant-velocity Kalman filter, Hungarian matching on IoU, class-aware) on YOLOv8n detections, scored with MOT metrics against Waymo's persistent camera object ids on the same 10 evaluation segments. Tracker settings were tuned on three held-out validation segments, never on the evaluation set.

Detector and trackerMOTAIDF1HOTA
YOLOv8n 512 x 512, default tracker0.1040.2860.277
YOLOv8n 512 x 512, tuned tracker0.1660.3070.287
YOLOv8n 640 x 960 (setting L, seed 42), tuned tracker0.0540.3270.314
Box oracle (ground-truth boxes as detections): ceiling of an IoU tracker here0.8190.6790.650

The better detector raises IDF1 and HOTA but adds false positives, so MOTA falls. The seed-42 detector used here is the best of three seeds, so its cyclist tracking should not be expected to repeat.

Where tracking fails

Every missed ground-truth box is attributed to a cause: no detection overlaps it (detector), or a detection above the score gate overlaps it but no track covers it (association).

Detector misses dominate, and they are small: the median missed box is 22 to 28 pixels on a side. Association misses are larger boxes (40 to 53 px), some of them track start-up lag. Most vehicle ID switches (421 of 653) happen after the object was lost for more than one frame, which points to appearance features or longer memory, not a better matching rule.

Image strip of an ID switch after a gap
ID switch after a gap: a vehicle is lost for several frames and returns under a new track id.
Image strip of an association miss at track start
Association miss from start-up lag: the bus is detected from the first frame (thin yellow), but a track appears two frames later (thick blue).
Image strip of a detector miss on a large bus
Detector miss: a large bus passing close on the left, cut by the image edge, has a ground-truth box but no detection and no track. White is ground truth, thick coloured boxes with ids are tracks, thin yellow boxes are raw detections; each strip shows seven consecutive frames.

Two further upgrades: appearance ReID and LiDAR 3-D gating

Both were tested on detections from all three YOLOv8n setting-L seeds, with settings tuned on 3 held-out segments and scored once on the 10 evaluation segments. ReID embeds each detection crop with a pretrained ResNet-18, blends appearance into the association cost, and re-links tracks lost for a few frames. LiDAR gating projects LiDAR points into each box for a 3-D position and rejects associations that are too far apart in the world frame.

Change in IDF1 against the baseline tracker, one dot per detector seed (42, 43, 44); diamond = mean

Why the official LiDAR AP is low

Official vehicle AP (0.048) is about a quarter of the BEV proxy (about 0.2). A local analysis of the saved CenterPoint detections found no pipeline bug; the boxes themselves are the problem.

Reproducibility

Checked from a fresh clone, a new Python 3.12 environment and an empty data cache.

StepResult
Install requirementsWorks in 38 s with a warm pip cache.
Smoke trainingCompletes in about 6 minutes, nearly all first-time data download.
Re-evaluate the 640 x 960 checkpointmAP 0.15669 against 0.15661 recorded (Apple GPU against CUDA).
Tracking with the tuned settingsMOTA 0.0537, IDF1 0.3265, 650 vehicle ID switches: identical to the recorded run.

Not reproduced: training from scratch, LiDAR training and the official evaluator, the fusion model. The model architecture can depend on the Ultralytics version (the reference runs used 8.4.198), so use the version of the original runs when retraining.

git clone https://github.com/KushalBKusram/WaymoOpenDatasetToolKit
cd WaymoOpenDatasetToolKit
python3.12 -m venv .venv && source .venv/bin/activate
pip install -r docs/requirements-tested.txt
gcloud auth application-default login     # read access to the public Waymo bucket
python evaluate.py --config configs/yolov8n_hires_100.yaml --run-dir runs/yolov8n_hires_100-v1.2-s42

Model weights

YOLOv8n (12.7 MB, mAP 0.157; the best of three seeds, so expect about 0.13 from a retrained model) and RT-DETR-L (131.8 MB, mAP 0.202; close to its 0.192 mean), both at 640 x 960, setting L, seed 42. Weights-only files that load with torch.load(..., weights_only=True).

SHA-256 (YOLOv8n): af3cd3bea8803b68d8432049a207281d67642cad10a20ba55e988943a1e3ad5f
SHA-256 (RT-DETR-L): ea1283784d327bc833f287c45d1aa92bbdfc43f4fe2bd38e380410085ee38451

import cv2, torch, yaml
from models import build_detector

cfg = yaml.safe_load(open("configs/yolov8n_hires_100.yaml"))        # or configs/rtdetr_hires_100.yaml
cfg["resume_weights"] = "yolov8n_hires_100-v1.2-s42.weights.pt"     # or rtdetr_l_hires_100-v1.2-s42.weights.pt
detector = build_detector(cfg)
model = detector.build_model(cfg, torch.device("cpu")).eval()
for det in detector.predict(model, cv2.imread("frame.jpg"), cfg):   # BGR image
    print(det["box"], det["label"], det["score"])

Non-commercial. These weights are released under the Waymo Dataset License Agreement for Non-Commercial Use, and any further use or modification is subject to it, including its restrictions on use in vehicle operation, production systems and primarily commercial purposes. A copy of the agreement is attached to the release.

License and attribution

This software and the model weights were made using the Waymo Open Dataset, provided by Waymo LLC under the Waymo Dataset License Agreement for Non-Commercial Use, available at waymo.com/open/terms, and your access and use of such work are governed by the terms and conditions therein. The dataset and the example frames shown here belong to Waymo.

@misc{waymo_open_dataset,
  title = {Waymo Open Dataset: An autonomous driving dataset},
  website = {\url{https://www.waymo.com/open}},
  year = {2019-2025}
}