MultiCamAssembly: Distributed Robotic Vision System Software Framework

<h1>MultiCamAssembly</h1>

<p>
MultiCamAssembly is a distributed software framework for multi-camera perception and coordination in automated crane-based timber construction.
It orchestrates multiple, distributed assembly robots (cambots) with embedded vision that detect geometric features in prefabricated timber elements and transmit observations to a central coordinator for multi-view pose estimation.
The fused pose estimate enables vision-guided crane motion during on-site assembly.
</p>

<h2>Structure</h2>

<ul>
<li><b>base/</b> — UDP server, device management, plotting tools, CLI helpers</li>
<li><b>cambots/</b> — camera utilities, YOLO integration, motor control, and programs</li>
<li><b>overlay_tracker/</b> — geometry, detections, tracking, rendering, and CLI</li>
<li><b>ssh/</b> — SSH config (e.g., lights-off profile)</li>
</ul>

<h2>Project Structure</h2>

<p>
For a full folder tree with short descriptions for every file, see <code>STRUCTURE.md</code>
</p>

<h2>Key Workflows</h2>

<h3>Main App Workflow</h3>

<ol>
<li>Power on remote robots and connect to the local network</li>
<li>Verify SSH config (e.g. <code>ssh/config_lightsoff</code>) matches the network</li>
<li>Run <code>base/programs/App.py</code></li>
<li>
Guided steps:
<ol>
<li>Device discovery and selection</li>
<li>Remote code updates</li>
<li>Component selection</li>
</ol>
</li>
</ol>

<h3>Model Training Workflow</h3>

<ol>
<li>Create labeled data using Label Studio</li>
<li>Update dataset paths in <code>yolo/yolo_train.yaml</code></li>
<li>Set hyperparameters and epochs in <code>yolo/yolo_train.py</code></li>
<li>Train model with <code>yolo/yolo_train.py</code></li>
<li>Copy trained weights to <code>cambots/yolo/models/</code></li>
<li>Generate TensorRT engines on remote devices</li>
<li>Validate inference on remote devices</li>
</ol>

<h3>Pushing and Pulling Workflow</h3>

<ol>
<li>Push code via <code>deploy.bat</code> (Windows) or <code>deploy.sh</code> (Linux)</li>
<li>Command: <code>./deploy.sh lightsoff-j01 [config_file]</code></li>
<li>Pushes <code>cambots/</code> without deleting remote-only files</li>
<li>Pull data via <code>fetch.bat</code> (Windows) or <code>fetch.sh</code> (Linux)</li>
<li>Command: <code>./fetch.sh lightsoff-j01 [-d]</code></li>
<li>Retrieves data from remote <code>cambots/save/</code> to local <code>save/</code></li>
</ol>

<h3>Camera Calibration Workflow</h3>

<ol>
<li>Required only if cameras change</li>
<li>Measure checkerboard cell size and update <code>square_size</code></li>
<li>Run <code>calibrate_camera.py</code> on remote devices</li>
<li>Capture 20 checkerboard images</li>
<li>Copy <code>camera_calibration_data.npz</code> to each remote device</li>
</ol>

<h3>Logging Workflow</h3>

<ol>
<li>Enable logging in <code>base/programs/App.py</code></li>
<li>UDP messages logged as JSONL files in <code>logs/</code></li>
<li>Used for post-processing and algorithm prototyping</li>
</ol>