This project aims to create a low-cost autonomous drone capable of visually detecting, tracking, and following a moving target using onboard computer vision. Rather than relying on expensive sensors such as LiDAR or depth cameras, the system uses a Raspberry Pi, a standard camera, and the drone's existing GPS and flight controller telemetry to estimate the target's position and command the aircraft autonomously.

The project combines computer vision, embedded systems, flight control, and robotics into a modular companion-computer architecture. The Raspberry Pi performs all perception and navigation calculations, while the flight controller remains responsible for stabilization and low-level flight dynamics.

This system consists of three main components: the drone/flight controller, the Raspberry Pi, and the camera. It is designed such that all processing is done within the drone itself, this eliminates the need for transmitting video data to an offboard computer. The only issue with this is that I need to make the computer vision processing program extremely lightweight, so that it would be able to run on the Pi.

One of the first challenges was adapting the computer vision system to run on the Raspberry Pi's limited processing power. While modern desktop computers can comfortably run object detection on every frame of a video stream, doing so on a Raspberry Pi quickly overwhelms the CPU and results in an unusable frame rate.

To overcome this, I redesigned the tracker as a hybrid system that combines deep learning with traditional computer vision techniques. Instead of running the YOLO object detector on every frame, the detector is executed only periodically. Between these detections, the target is tracked using the Lucas-Kanade optical flow algorithm, which estimates the movement of distinctive feature points within the object. Optical flow requires only a fraction of the computational power of a full neural network inference while still providing smooth, frame-to-frame tracking.

The system was further optimized by limiting object detection to a Region of Interest (ROI) rather than processing the entire camera image. Before a target is acquired, the ROI is centered around the manually controlled targeting box. Once a target is locked, the ROI follows the tracked object, allowing YOLO to search only in the area where the target is expected to appear. This significantly reduces the number of pixels processed during each detection cycle, improving both speed and reliability.

To maintain robustness, YOLO periodically re-detects the object within the ROI and corrects any drift introduced by optical flow. If the object is temporarily lost, the tracker attempts to reacquire it using overlapping detections before returning to manual targeting mode.

These optimizations dramatically reduced the computational load while preserving accurate tracking, making real-time operation feasible on the Raspberry Pi without requiring dedicated AI hardware.

Before any optimizations were made to the program, I tested the original YOLO tracker on the Raspberry Pi to get a performance baseline. The results confirmed that running object detection on every frame was too demanding for the Pi. The frame rate dropped to only a few frames per second, causing noticeable lag between the camera feed and the tracker. This made it nearly impossible to follow moving targets and showed that a different approach would be necessary for real-time operation.

After implementing the hybrid tracking system (detections and optical flow), I repeated the tests on the Raspberry Pi using both prerecorded videos and a live webcam feed. The combination of skipped YOLO detections, optical flow tracking, and region-of-interest detection produced a significant improvement in performance. Tracking remained smooth enough to follow moving objects while reducing the processor load compared to the original implementation.

These tests also proved the tracking logic itself worked. Once a target was selected, optical flow maintained the target's position between detection frames, while periodic YOLO inferences corrected any accumulated drift. The tracker was able to reacquire targets after brief occlusions or rapid motion, demonstrating that the hybrid approach maintained reliability despite performing far fewer neural network inferences.

Testing on the Raspberry Pi confirmed that the optimizations successfully transformed the tracker from an impractically slow proof of concept into a system capable of real-time performance on inexpensive embedded hardware, providing a solid foundation for integrating the vision system with the drone's navigation and control software.

With the vision system operating reliably on the Raspberry Pi, the next step was enabling communication between it and the drone's flight controller. Rather than directly controlling the motors or emulating radio control inputs, I chose to use MAVLink, a lightweight communication protocol widely used in autonomous aerial systems.

The only issue was, there is no way to tell the drone to "move left" or "go up" through MAVLink in iNav. After some research, though, I discovered the GCS Nav flight mode. It tells the drone to go into waypoint mission mode and creates a variable waypoint that changes it's coordinates based on telemetry through MAVLink. I can then use this to constantly feed destination coordinates to the flight controller. 

I made a prototype script to estimate the coordinates of a target using the drone's heading and GPS position, also estimating the distance to the target using the size and type of detection on the screen.

The Raspberry Pi was then connected to the flight controller through a dedicated UART configured for MAVLink telemetry. This provided the Raspi with a continuous stream of flight data, including GPS coordinates, altitude, heading, and other telemetry required for autonomous navigation. A dedicated script was used to receive and process incoming MAVLink messages independently of the vision system, ensuring that telemetry updates did not interfere with image processing performance.

I am currently still testing and tweaking the software and getting all of the different parts to play nicely with eachother. The next step is a flight test in a controlled environment where IO can try out the GCS Nav mode and see if it interfaces well with the Raspi's commands. So far, though, every part of this project seems to be working well independently, its just a matter of putting everything together and going out for a flight!