Low-Latency Multimodal Perception for Robotic Assembly and Real-Time Inspection of Automotive Wiring Systems using Frame-Based and Event-Based Vision

A masters research project for deformable linear object (wire) detection using two complementary vision sensors:

• Basler frame camera (grayscale, high-resolution frames)
• iniVation DVXplorer event camera (asynchronous events, high dynamic range)

The system combines AprilTag-based ROI motion detection, homography calibration, and real-time wire tracking with spline fitting.

---

Features

• Dual-camera acquisition (Basler + DVX)
• AprilTag ROI & motion detection inside a defined region
• Homography calibration (DVX → Basler alignment using ORB features)
• Real-time overlay of event data on frame-based images
• Spline-based wire detection from masked regions
• Screenshot saving & ROI cropping with keyboard shortcuts
• Interactive UI with visualization windows

---

Requirements

• Python 3.8+
• Basler Pylon SDK
• DVXplorer SDK

---

Core dependencies:

• opencv-python
• numpy
• scipy
• pypylon (Basler)
• dv-processing (DVXplorer)
• pupil-apriltags

---

Keyboard Shortcuts (wire detection window)

• ESC → exit
• p → save overlay screenshot
• m → save cropped ROI
• q/w → adjust low threshold
• a/s → adjust high threshold
• r/e → decrease/increase ROI width
• f/t → decrease/increase ROI height
• i/j/k/l → move ROI rectangle (up/left/down/right)

---

Example Output

License

This project is licensed under the MIT License. See LICENSE for details.

---

Acknowledgements

• Basler Pylon SDK
• iniVation DVXplorer SDK
• pupil-apriltags -OpenCV community
• FAU-FAPS

---

Citation / Bibtex If you use this project in your research, please cite:

@projekt{chaudhry2025lowlatency,
  title = {Low-Latency Multimodal Perception for Robotic Assembly and Real-Time Inspection of Automotive Wiring Systems using Frame-Based and Event-Based Vision},
  author = {Chaudhry, Affyief},
  supervisor = {Hartmann, Annalena},
  institution = {FAU-FAPS},
  year = {2025},
  note = {Project Arbeit},
  url = {https://github.com/Affyief/Project-arbeit---Event-based-vision}
}