1National University of Science and Technology POLITEHNICA Bucharest, Romania 2Institute of Mathematics "Simion Stoilow" of the Romanian Academy, Romania
Indoor drone navigation is challenging due to GPS denial and sensor drift. We introduce a vision-language-guided framework where a frozen model generates pseudo-labels to train a lightweight YOLOv8 student for markerless drone detection. Coupled with a depth-free IBVS controller operating solely on 2D image features, our system enables robust, real-time indoor UAV tracking with minimal manual supervision. Validated in classroom flights, it performs reliably across varied trajectories and supports scalable deployment for surveillance, inspection, and automation.
Key Contributions
Vision-language teacher-student framework: We train a YOLOv8 student detector using pseudo-labels generated by a frozen vision-language model (GroundingSAM), enabling scalable, self-supervised annotation for drone detection.
Depth-free visual servoing controller: We design a reduced 4-DoF IBVS controller that operates exclusively on 2D image features, avoiding the instability and hardware requirements of depth estimation.
Curated indoor dataset with high-fidelity pseudo-labels: We contribute a high-resolution indoor aerial dataset annotated automatically with over 98% accuracy, preserving temporal structure from video and supporting domain-specific detection tasks.
Overview Video
Complete system demonstration showcasing our vision-language-guided UAV tracking framework with real-time object following, depth-free visual servoing control, and indoor autonomous navigation.
Detection Results & Samples
Our vision-language-guided UAV tracking framework demonstrates robust performance across diverse indoor scenarios. We collected 14,400 high-resolution frames (3840 × 2160) using automated pseudo-labeling with 98.6% accuracy.
Multi-Target Following: UAVs tracking various targets including ground vehicles and aerial drones across indoor classrooms, constrained spaces, and diverse conditions.
Detection Accuracy: Success cases (left, right) and failure case (center) showing 98.6% automated labeling accuracy with only 1.4% requiring manual correction.
Dataset Comparison
Our indoor drone dataset maintains comparable size to datasets used for YOLOv8 fine-tuning, with the key distinction being the temporal information preserved in our video-based dataset.
Paper
Dataset
No. of Videos
No. of Images
Ours
Ours (only drone footage)
4
14,400
Real-Time Flying OD with YOLOv8
Dataset 1
No videos
15,064
Real-Time Flying OD with YOLOv8
Dataset 2
No videos
11,998
Dogfight
NPS-Drones
14
70,250
Dogfight
FL-Drones
50
38,948
Table 1: Dataset comparison showing our indoor drone dataset maintains comparable size to datasets used for YOLOv8 fine-tuning. While smaller than Dogfight datasets, our high-quality annotations address potential labeling inconsistencies present in larger datasets.
GradCAM Attention Analysis
To validate detection reliability, we employed GradCAM to visualize YOLOv8's attention patterns across different scenarios. This analysis reveals important insights about the model's behavior in various environments.
Mixed Human-Drone Scenarios: The model shows strong bias toward human subjects, potentially compromising drone detection in human-populated environments.
Drone-only Scenarios: Distributed attention across multiple UAV targets with reduced environmental interference, indicating robust multi-target tracking capabilities.
Implications for Deployment: The observed attention bias suggests potential performance degradation in human-populated environments. This validates our dataset design focusing on indoor drone-to-drone scenarios where human interference is minimized. The distributed attention pattern in drone-centric scenarios indicates robust multi-target tracking capabilities essential for autonomous operation.
YOLOv8 Performance Evaluation
We assess model performance across precision, recall, mAP50-95, and inference latency to highlight trade-offs between accuracy and real-time feasibility. YOLOv8-nano trained for 25 epochs achieves the best balance: 0.97 precision, 0.97 recall, 0.79 mAP50-95, and a low 4.7 ms inference time.
Model
Epochs
Precision ↑
Recall ↑
mAP50-95 ↑
Inference Time ↓ (ms)
YOLOv8-nano
5
0.95
0.92
0.67
4.7
10
0.97
0.97
0.72
4.7
25
0.97
0.97
0.79
4.7
YOLOv8-small
5
0.93
0.93
0.62
6.9
10
0.95
0.94
0.72
7.0
25
0.98
0.97
0.78
7.0
YOLOv8-large
5
0.86
0.83
0.56
29.6
10
0.94
0.93
0.69
32.4
25
0.97
0.95
0.77
31.3
Table 2: Comparative evaluation of YOLOv8 architectures across training epochs, demonstrating the precision-efficiency trade-off critical for onboard deployment. Bold values indicate selected configuration optimizing for real-time constraints.
Control Algorithm Comparison
We compare our depth-free IBVS controller against traditional PID control for indoor UAV tracking. IBVS achieves superior centering performance (89-92% vs 68-71%) while operating solely on 2D visual features.
Algorithm Architectures
PID Controller: Traditional position-based control approach
IBVS Controller: Vision-based servoing control architecture
Spatial Distribution Analysis
IBVS Line Trajectory - Detection Heatmap
IBVS Circular Trajectory - Detection Heatmap
PID Line Trajectory - Detection Heatmap
Controller Performance Comparison
IBVS achieves superior tracking with 89% (straight-line) and 92% (circular) centering scores versus PID's 68% and 71%. IBVS demonstrates higher control authority but increased command variability.
Scenario
Controller
Range vx/vy/ωz Cmd (m/s)
Centering Score
Smoothness
Straight Line
PID
[0, 0]/[-10, 30]/[-4, 4]
0.68
High
IBVS
[-4, 4]/[-10, 40]/[0, 25]
0.89
Medium
Circular
PID
[0, 0]/[-10, 30]/[-5, 15]
0.71
Low
IBVS
[-10, 20]/[-20, 30]/[0, 9]
0.92
High
Table 3: Controller performance comparison. Centering Score represents the fraction of time the target remained within the central 30% of the image frame. Command Smoothness is qualitatively assessed from standard deviation analysis.
The depth-decoupled IBVS approach shows clear advantages in tracking accuracy and adaptability to complex trajectories. IBVS exhibits particularly improved performance during high-curvature segments where PID shows erratic behavior.
Conclusion
We presented a self-supervised framework for autonomous indoor UAV tracking that combines scalable dataset generation with depth-free visual servoing control. Leveraging a frozen vision-language model (GroundingSAM), we achieved 98.6% labeling accuracy across a high-resolution indoor drone dataset, significantly reducing manual annotation effort.
Our evaluation of YOLOv8 variants highlights the trade-off between accuracy and inference latency. YOLOv8-nano strikes the best balance for real-time deployment (0.97 precision/recall at 4.7 ms), while larger models offer marginal gains at a cost to onboard performance. The proposed depth-decoupled IBVS controller enables stable target following using only 2D visual features, improving centering performance (89–92% vs. 68–71%) compared to PID baseline, especially in complex trajectories.
Despite some limitations including fixed flight altitude assumptions and detection reliability degradation under occlusion or low contrast, our approach demonstrates a reproducible and scalable pipeline for markerless indoor UAV tracking with potential applications in warehouse automation, surveillance, and multi-agent indoor robotics.
For questions and collaboration inquiries, please contact the authors through the GitHub repository or academic channels.
Citation
@article{nae2025learning,
title={Learning Object Following for Small UAVs using Vision-Language and Efficient Visual Flight Control},
author={Nae, Sebastian-Ion and Leordeanu, Marius},
journal={2nd Best Romanian AI Thesis Awards (BRAIT 2025)},
year={2025}
}
