Detecting and Tracking
Flying Objects
A startup needed to see drones in the sky reliably, tell them apart from the birds and planes that fool most detectors, and know where each one was heading. I built a real-time system that does all three, and beats the published state of the art at drone detection on every object size, on the same public benchmark. On one brutally hard clip the best published model missed the drone in all 300 frames; mine found it in 262.
A startup working in the drone space needed to detect drones. They were still exploring where their technology goes, so the detector had to serve experiments they had not fully defined yet. Track a drone in flight. Spot one at a distance. Tell a drone apart from everything else in the sky. The one thing that was certain is that whatever they built next would need to trust what the camera was telling it, and a detector you cannot trust is worse than none at all.
Detecting a drone sounds like a solved problem. An object detector draws a box around a thing and names it, and there are good published models that do exactly this for drones. So the reasonable first move was to take the best one and reproduce it. If the state of the art already works, you stand on it.
Here is the first thing that went wrong. I picked the newest detection backbone as the foundation, because newer usually means better. But the published drone models were built on an older one, and most of the clever tricks in those papers, the special attention modules, the loss tuned for tiny objects, existed to patch weaknesses in that older base. The new backbone had already absorbed them. So when I ported those innovations onto it, they did nothing, or made it worse. Even the paper's custom small-object loss was redundant here; the stock one did as well or better. I was adding a second lock to a door that already had one.
The reason those tricks failed is not that they were bad. It is that they were answers to a question the newer model no longer asked. One change did earn its place: a high-resolution detection head that gives the model a finer view of small, distant objects, which is the whole game when a drone is far away. With the base already doing most of the work the old papers did by hand, the harder problems turned out to lie elsewhere.
One of them was the data. The four-class model, the one that separates drone from bird, airplane, and helicopter, started out hallucinating. It fired confident boxes at empty blue sky, at nothing at all. The instinct is to blame the model, but the fault was in what it had been fed. The public datasets behind it were subtly corrupted: synthetic-looking birds crudely cropped into skies with hard, unnatural borders, so the model had learned that a sharp edge against blue meant "object." And there were almost no examples of genuinely tiny, distant targets, the exact case that matters most when a drone is far away.
Once the cause was clear the fix followed: work out which datasets were teaching the model the wrong thing, drop them, and build up the coverage of small distant objects that was missing. The diagnosis was the hard part. The correction was mechanical once you knew what to correct for.
So what came of it? Start with the detector built to do one thing, find drones. On the same public footage, it beats the published state of the art across the whole spectrum of object sizes, from tiny distant specks to large close ones. The margin is widest exactly where it is hardest, on the smallest, most distant drones, which is the case that matters most in the field. That was the goal, and it was met.
Then I built a second model, and this is where it gets interesting. This one does not just find drones, it separates four kinds of flying object, drone from bird from airplane from helicopter. You would expect it to be a worse drone detector than the specialist, because it is dividing its attention four ways instead of pouring it all into one target. It is not. Here is the fact that surprised me. There is a brutally hard clip in the benchmark, a drone so small and distant it is barely there. The published baseline found it in zero of its 300 frames. My own single-class specialist, the one tuned only for drones, found it in just one. The four-class model found it in 262 of 300, at high confidence, with almost no false alarms. Teaching the model to tell four things apart had made it better at seeing the hardest drone than the model that only ever looked for drones.
Why would that happen? Because to reliably call something a drone, it helps to know what a drone is not. A model that has learned the shape of a bird and the shape of a helicopter has a sharper idea of the thing in front of it, and stops guessing. That is the deeper payoff of the four classes, and it shows up as a plain practical benefit too. A drone-only detector has one word, so everything that flies is a candidate drone. Show it a helicopter and it will often say drone, confidently and wrongly. A four-class model can say for certain whether it is a drone, a helicopter, a bird, or nothing at all.
There is a third piece, the forecasting. The system does not just see where each object is, it predicts where it is going, rolling every track forward over the next twenty frames and drawing that as an uncertainty cloud on the video. The cloud is a genuine bound, not decoration: even on the worst clip tested, the object's real position landed inside it more than 82 percent of the time, and on the cleaner cases well above that. So the system tells you not only what is in the sky right now, but where it will be a moment from now, which is exactly what you need if you intend to act on it.
That is the whole of it. The published state of the art was the bar, and the final system cleared it, while telling four kinds of flying object apart and forecasting where each one is headed. A detector the startup can actually trust to build on.