LettuceMOTS

# Dataset Card for LettuceMOTS This is a [FiftyOne](https://github.com/voxel51/fiftyone) dataset with 12 samples.  ## Installation If you haven't already, install FiftyOne: ```bash pip install -U fiftyone ``` ## Usage ```python import fiftyone as fo from fiftyone.utils.huggingface import load_from_hub # Load the dataset # Note: other available arguments include 'max_samples', etc dataset = load_from_hub("Voxel51/LettuceMOTS") # Launch the App session = fo.launch_app(dataset) ``` ## Dataset Details ### Dataset Description LettuceMOTS is a lettuce **multi-object tracking and segmentation (MOTS)** dataset captured by a downward-facing RGB camera mounted on VegeBot, an agricultural robot developed by China Agricultural University, as it traveled through rows of a lettuce farm. The robot's motion includes straight, backward-and-forward, and turning-out-and-in trajectories, so the same lettuce plant frequently exits and later re-enters the camera's field of view. The dataset was built specifically to support **re-identification of these "re-occurred" plants** — recovering a plant's original track ID after a long absence — which is the key unsolved problem for robots that must spray each vegetable exactly once while moving back and forth along farm rows. - **Curated by:** Nan Hu, Daobilige Su, Shuo Wang, Xuechang Wang, Huiyu Zhong, Zimeng Wang, Yongliang Qiao, Yu Tan (China Agricultural University and collaborating institutions) - **Funded by:** [More Information Needed] (not stated in the accessible preprint/journal indexing; a related dataset from the same lab and robot, LettuceMOT, was funded by the National Natural Science Foundation of China, Grant No. 3217150435, and a China Agricultural University international cooperation seed fund — it is unconfirmed whether the same funding covers this dataset) - **Shared by:** Nan Hu et al., via [github.com/NanH5837/LettuceMOTS](https://github.com/NanH5837/LettuceMOTS) - **Language(s):** N/A (image/video dataset, no text) - **License:** The original authors did not explicitly state a data license (only "publicly released for the benefit of the community"); the accompanying code repository is licensed GPL-3.0. This FiftyOne re-release on Hugging Face Hub is tagged **CC-BY-4.0** as a reasonable default for open research re-distribution — please attribute the original paper (see Citation) if you use this dataset. ### Dataset Sources - **Repository:** [github.com/NanH5837/LettuceMOTS](https://github.com/NanH5837/LettuceMOTS) - **Paper:** Hu, N., Su, D., Wang, S., Wang, X., Zhong, H., Wang, Z., Qiao, Y., & Tan, Y. (2024). Segmentation and tracking of vegetable plants by exploiting vegetable shape feature for precision spray of agricultural robots. *Journal of Field Robotics*, 41(3), 570–586. https://doi.org/10.1002/rob.22279 (preprint: [arXiv:2306.13518](https://arxiv.org/abs/2306.13518)) - **Demo:** [github.com/NanH5837/LettuceMOTS](https://github.com/NanH5837/LettuceMOTS) includes a demo video and reference YOLOv5-seg + shape-feature tracking implementation. ## Uses ### Direct Use - Training and evaluating **instance segmentation** models for lettuce plants in field conditions. - Training and evaluating **multi-object tracking / MOTS** methods, in particular data-association strategies that must handle long-term occlusion and re-entry (re-identification), as opposed to short-term occlusion handling alone. - Benchmarking robotic precision-spray perception pipelines end to end (segment → track → decide whether a plant has already been sprayed). ### Out-of-Scope Use - General-purpose or multi-class object detection/segmentation — the dataset contains a single class (`lettuce`) grown in one farm, so it is not suited for evaluating broader vegetable/plant taxonomies. - Deployment or generalization claims outside similar conditions to those captured: a fixed downward-facing camera at ~1.4 m height, ~810×1080 cropped frames, natural outdoor lighting, and lettuce spaced ~0.15–0.25 m apart. Different crops, camera geometry, or planting density would require re-validation. - The dataset contains no people or personally identifiable information; it should not be repurposed for anything involving personal data. ## Dataset Structure Imported as a FiftyOne **video** dataset (`media_type="video"`) named `lettucemots` with **12 samples** — one video per raw image sequence — totaling **1,308 frames**. Each of the 12 sequences corresponds to one continuous robot pass through a farm row; the 12 sequences span 3 growth stages (seedling/rosette, cupping, heading) × 4 sequences per stage, with 3 sequences per stage used for training and 1 held out for test (9 train / 3 test overall). Each video was re-encoded from the source PNG frame stack to H.264 MP4 at the dataset's native **15 FPS** acquisition rate (frame content and count are otherwise unmodified from the original release). ### Sample fields | Field | FiftyOne type | Description | |-------|---------------|-------------| | `filepath` | `StringField` | Path to the per-sequence MP4 (re-encoded from the source PNG frame stack) | | `tags` | `ListField(StringField)` | `train` or `test`, mirroring `split` | | `sequence` | `StringField` | Sequence identifier, `"0000"`–`"0011"` (verbatim from source directory names) | | `split` | `StringField` | `train` or `test`, matching the dataset's original directory split | | `metadata` | `VideoMetadata` | Frame size (810×1080), frame rate (15 fps), total frame count, duration — computed via `dataset.compute_metadata()` | ### Frame fields | Field | FiftyOne type | Description | |-------|---------------|-------------| | `frame_number` | `FrameNumberField` | 1-indexed frame number within the sequence (source `frame_id` + 1) | | `ground_truth` | `Detections` | Per-frame instance segmentation + tracking labels (see below) | ### Label type and mapping Each `ground_truth` detection is an `fo.Detection` with: - `label`: always `"lettuce"` (the dataset has a single class) - `mask`: a boolean instance mask, cropped to the detection's own tight bounding box - `bounding_box`: `[x, y, w, h]` in relative `[0, 1]` coordinates, computed as the tight extent of the instance mask (the source format ships pixel masks but no separate box annotation) - `index`: an integer **track ID**, unique within its sequence and stable across frames — this is what lets the same physical plant be re-identified across a long absence from the camera's field of view, and is what FiftyOne's App uses to render tracking trajectories `fo.Detections`/masks (rather than `fo.Segmentation`) were chosen because the source annotations are already per-instance (KITTI-MOTS-style: one RLE mask per tracked object per frame), and per-instance masks are what's needed to carry the per-object `index` used for tracking — a single semantic `fo.Segmentation` mask per frame would have discarded the track identity. Source ground truth ships in two redundant, pixel-verified-identical forms (only the RLE form was used for the FiftyOne import; the mask-PNG form is not otherwise referenced by the FiftyOne dataset): 1. A per-sequence KITTI-MOTS-style TXT file (`frame_id object_id class_id img_height img_width RLE`), one line per instance per frame. 2. A per-frame 16-bit grayscale PNG where pixel value `0` = background and `1000 + track_id` = that instance's pixels. The source `object_id` values are offset by `1000` (e.g. `1000`, `1001`, ...); the FiftyOne `index` field stores `object_id - 1000` for a cleaner, dataset-consistent 0-based-ish per-sequence ID space. ### `dataset.info` Not currently populated (`{}`); `dataset.default_classes` is also unset. Since there is only one label value (`"lettuce"`) across the whole dataset, no class-id map was needed for import, but a future pass could set `dataset.default_classes = ["lettuce"]` for clarity. ### Parsing decisions - **Bounding boxes are derived, not sourced**: the release does not ship separate bbox annotations, only masks; boxes are the tight pixel extent of each decoded instance mask. - **Frame numbering**: source `frame_id` is 0-indexed and contiguous per sequence with zero gaps (verified for all 12 sequences); FiftyOne's `frame_number` is 1-indexed, so `frame_number = frame_id + 1`. - **Video re-encoding, not raw frame import**: frames were stitched into MP4s (rather than imported as a flat image collection) specifically to use FiftyOne's video/track-native `index` field and App timeline scrubbing, since the source data is fundamentally a set of image sequences with persistent per-object identity. `fiftyone.utils.utils.video.reencode_video`'s `in_opts` is silently overridden when `reencode=True`, so the true 15 FPS input rate had to be set via a direct `ffmpeg -framerate 15` call rather than that helper's defaults (which fall back to 25 FPS for image-sequence inputs). - **YOLOv5-seg polygon labels not imported**: the release also ships a secondary, redundant label set (`LettuceMOTSyolo/`, normalized polygon labels in YOLOv5-seg format) for convenience training with the reference implementation. These cover only the 9 training sequences, carry no track ID, and are fully derivable from the mask/RLE ground truth already imported — they were left out of the FiftyOne dataset as redundant, but remain available on disk (`LettuceMOTSyolo/`) if needed for exact reproduction of the paper's YOLOv5 training recipe. - **Pretrained weights** (`weights/0003.pt`, `0007.pt`, `0011.pt`, one YOLOv5m-seg checkpoint per held-out test sequence) were downloaded alongside the dataset but are not part of the FiftyOne dataset itself; they require the original repo's `models/` code on `sys.path` to unpickle (not vanilla `ultralytics`). ## Dataset Creation ### Curation Rationale Existing agricultural MOTS/MOT datasets (e.g. Apple MOTS) are collected with cameras that move in one direction, so an object that leaves the frame never reappears — models built on them simply drop or discard tracks that disappear. Real spray robots, however, must reverse to avoid obstacles, temporarily leave a row (e.g. to recharge), or re-enter a row, causing genuine plants to leave and later re-enter the same camera's view. Without re-identification, a robot will spray such a plant more than once. LettuceMOTS was purpose-built with backward/forward and turn-out/turn-in robot trajectories specifically to create these long-gap re-occurrence cases and let researchers benchmark re-identification, not just short-term tracking-through-occlusion. ### Source Data #### Data Collection and Processing Images were collected by VegeBot, a four-wheel-steer, four-wheel-drive agricultural robot, in a lettuce farm in Tongzhou District, Beijing, China, in September–October 2022. Lettuces were spaced ~0.15–0.25 m apart within a row and ~0.25–0.3 m between rows, with weed density up to ~10/m². Data was collected at 3 growth stages. A downward-facing RGB camera (1/2.8" Sony IMX317 CMOS sensor) was mounted ~1.4 m above the ground, capturing at 1920×1080 / 15 FPS under natural light with automatic exposure; frames were subsequently cropped to 810×1080 to remove irrelevant background. The robot was manually remote-controlled (rather than using its own vision-based autonomous navigation) to ensure consistent image quality, and moved at roughly 0.35–0.4 m/s. Annotation used the CVAT tool (its tracking/interpolation features reduce manual per-frame labeling effort), producing masks in a KITTI-MOTS-style RLE format with persistent per-plant track IDs. #### Who are the source data producers? The lettuce plants and farm scene were part of a working farm in Tongzhou District, Beijing; data collection was performed by the paper's authors' research group (China Agricultural University) using their VegeBot robot platform. No crowdsourcing was involved in data collection. ### Annotations #### Annotation process Frames were manually annotated in CVAT with per-instance segmentation masks and consistent per-object track IDs across each sequence, then exported to a KITTI-MOTS-style text format (`frame_id object_id category img_height img_width RLE`) — one line per object instance per frame. `category` is always `1` (vegetable/lettuce). A secondary polygon-format label set for direct YOLOv5-seg training was also derived from these masks by the authors, covering only the training sequences. #### Who are the annotators? The paper states CVAT was used for annotation but does not name or describe the individual annotators. #### Personal and Sensitive Information None identified. The dataset consists exclusively of overhead farm imagery (soil, crop rows, and lettuce plants); no people, faces, or personally identifiable information appear in the released frames. ## Citation **BibTeX:** ```bibtex @article{hu2024segmentation, title = {Segmentation and tracking of vegetable plants by exploiting vegetable shape feature for precision spray of agricultural robots}, author = {Hu, Nan and Su, Daobilige and Wang, Shuo and Wang, Xuechang and Zhong, Huiyu and Wang, Zimeng and Qiao, Yongliang and Tan, Yu}, journal = {Journal of Field Robotics}, volume = {41}, number = {3}, pages = {570--586}, year = {2024}, publisher = {Wiley}, doi = {10.1002/rob.22279} } ``` **APA:** Hu, N., Su, D., Wang, S., Wang, X., Zhong, H., Wang, Z., Qiao, Y., & Tan, Y. (2024). Segmentation and tracking of vegetable plants by exploiting vegetable shape feature for precision spray of agricultural robots. *Journal of Field Robotics*, 41(3), 570–586. https://doi.org/10.1002/rob.22279 ## More Information Verified during import: total frame count (1,308), total tracked object instances (314), and total mask annotations (17,562) all match the values reported in the source paper exactly, as does the re-occurrence behavior the dataset was designed to capture (e.g. one track in sequence `0004` appears in frames 1–5, disappears, and reoccurs under the same ID in frames 114–117). Do not confuse this dataset with the earlier, related **LettuceMOT** (no "S") dataset from the same lab and robot platform, which has bounding-box-only MOT annotations (no segmentation masks) and is hosted separately on Mega.nz.