UA-1 DECISIVE: Development and Execution of Comprehensive and Integrated Systematic Intelligent Vehicle Evaluation

The images in the diagram show an operator holding a controller flying a drone with example bar charts of performance data. — A diagram showing each research area of the DECISIVE project: field readiness, trust, interface, situation awareness, communications, obstacle avoidance, navigation, autonomy, and mapping.

This page features products that are the result of the UA-1 DECISIVE: Development and Execution of Comprehensive and Integrated Systematic Intelligent Vehicle Evaluations project by the University of Massachusetts Lowell. This project is sponsored by the Department of the Army, U.S. Army Combat Capabilities Development Command Soldier Center (W911QY-18-2-0006 and W911QY-20-2-0005).

DECISIVE Test Methods Handbook

This handbook outlines all test methods developed under the DECISIVE project for evaluating small unmanned aerial systems (sUAS) performance in subterranean and constrained indoor environments, spanning communications, field readiness, interface, obstacle avoidance, navigation, mapping, autonomy, trust, and situation awareness. Specifications for 20 test methods are included:

Test Methods

Non-Line-of-Sight (NLOS) Communications
Non-Line-of-Sight (NLOS) Video Latency
Interference Reaction
Runtime Endurance
Takeoff and Land/Perch
Room Clearing
Indoor Noise Level
Logistics Characterization
Operator Control Unit (OCU) Characterization
Obstacle Avoidance and Collision Resilience
Position and Traversal Accuracy
Navigation Through Apertures
Navigation Through Confined Spaces
Indoor Mapping Resolution
Indoor Mapping Accuracy
Non-Contextual Autonomy Ranking
Contextual Autonomy Ranking
Characterizing Factors of Trust
Interface-Afforded Attention Allocation
Situation Awareness (SA) Survey Comparison

The DECISIVE Test Methods Handbook (v1.1) is available on arXiv:

Adam Norton, Reza Ahmadzadeh, Kshitij Jerath, Paul Robinette, Jay Weitzen, Thanuka Wickramarathne, Holly Yanco, Minseop Choi, Ryan Donald, Brendan Donoghue, Christian Dumas, Peter Gavriel, Alden Giedraitis, Brendan Hertel, Jack Houle, Nathan Letteri, Edwin Meriaux, Zahra Rezaei Khavas, Rakshith Singh, Gregg Willcox, and Naye Yoni. "DECISIVE Test Methods Handbook: Test Methods for Evaluating sUAS in Subterranean and Constrained Indoor Environments, Version 1.1." arXiv preprint arXiv:2211.01801, November 2022.

DECISIVE Benchmarking Data Report

This report includes the results of performance benchmarking of a set of 8 small unmanned aerial systems (sUAS) using the test methods specified in the DECISIVE test methods handbook (see above). The following sUAS platforms were evaluated: Cleo Robotics Dronut X1P (P = prototype), FLIR Black Hornet PRS, Flyability Elios 2 GOV, Lumenier Nighthawk V3, Parrot ANAFI USA GOV, Skydio X2D, Teal Golden Eagle, and Vantage Robotics Vesper. The data contained in the report is the result of conducting over 230 tests in-house at the NERVE Center and off-site at Fort Devens and Muscatatuck Urban Training Center, and analyses of best-in-class systems per test method category.

The DECISIVE Benchmarking Data Report (results from Phase I) is available on arXiv:

Adam Norton, Reza Ahmadzadeh, Kshitij Jerath, Paul Robinette, Jay Weitzen, Thanuka Wickramarathne, Holly Yanco, Minseop Choi, Ryan Donald, Brendan Donoghue, Christian Dumas, Peter Gavriel, Alden Giedraitis, Brendan Hertel, Jack Houle, Nathan Letteri, Edwin Meriaux, Zahra Rezaei Khavas, Rakshith Singh, Gregg Willcox, and Naye Yoni. "DECISIVE Benchmarking Data Report: sUAS Performance Results from Phase I." arXiv preprint arXiv:2301.07853, January 2023.

Publications

Zahra Rezaei Khavas, Edwin Meriaux, Amin Majdi, and Paul Robinette. "Validating Drone Trust Testing in Navigation Deviation Cases in Simulation." arXiv preprint arXiv:2310.04794, October 2023.
Edwin Meriaux, Jay Weitzen, and Adam Norton. "Simulation of the Effect of Correlated Packet Loss for sUAS Platforms Operating in Non-Line-of-Sight Indoor Environments." Drones, Volume 7, Issue 7, July 2023.
Zahra Rezaei Khavas, Amin Majdi, S. Reza Ahmadzadeh, and Paul Robinette. "Human Trust After Drone Failure: Study of the Effects of Drone Type and Failure Type on Human-Drone Trust." In Proceedings of the 20th International Conference on Ubiquitous Robots (UR) 2023, pp. 685-692. IEEE, June 2023.
Edwin Meriaux, Jay Weitzen, Naye Yoni, and Adam Norton. "Test Methodologies for Evaluating the Effectiveness of sUAS Communication Links for Operation in Indoor and Subterranean Environments." ASTM Journal of Testing and Evaluation, Volume 51, Issue 6, June 2023.
Minseop Choi, and Thanuka L. Wickramarathne. "On the Use of Real-Time Multi-Object Detection and Classification for Automated Assessment of Operator Visual Attention Allocation." In Proceedings of the 26th International Conference on Information Fusion (FUSION) 2023, pp. 1-7. IEEE, June 2023.
Ryan Donald, Peter Gavriel, Adam Norton, and S. Reza Ahmadzadeh. "Contextual Autonomy Evaluation of Unmanned Aerial Vehicles in Subterranean Environments." In Proceedings of the 9th International Conference on Automation, Robotics and Applications (ICARA) 2023, February 2023.
Edwin Meriaux, and Kshitij Jerath. "Evaluation of Navigation and Trajectory-following Capabilities of Small Unmanned Aerial Systems." In Proceedings of the IEEE International Symposium on Technologies for Homeland Security (HST), IEEE, November 2022.
Minseop Choi, John Houle, and Thanuka L. Wickramarathne. "On the Development of Quantitative Operator Situational Awareness Assessment Methods for Small-Scale Unmanned Aircraft Systems." In Proceedings of the 25th International Conference on Information Fusion (FUSION) 2022, pp. 1-8. IEEE, July 2022.
Brendan Hertel, Ryan Donald, Christian Dumas, and S. Reza Ahmadzadeh. "Methods for Combining and Representing Non-Contextual Autonomy Scores for Unmanned Aerial Systems." In Proceedings of the 8th International Conference on Automation, Robotics and Applications (ICARA) 2022, pp. 135-139. IEEE, February 2022.
Adam Norton, Peter Gavriel, Brendan Donoghue, and Holly Yanco. "Test Methods to Evaluate Mapping Capabilities of Small Unmanned Aerial Systems in Constrained Indoor and Subterranean Environments." In Proceedings of the IEEE International Symposium on Technologies for Homeland Security (HST) 2021, IEEE, November 2021.
Minseop Choi, John Houle, Nathan Letteri, and Thanuka L. Wickramarathne. "On The Use of Small-Scale Unmanned Autonomous Systems for Decision-Support in Subterranean Environments: The Case of Operator Situational Awareness Assessment." In Proceedings of the IEEE International Symposium on Technologies for Homeland Security (HST) 2021, IEEE, November 2021.

Advancing and Evaluating UAS for Defense

The capstone event for the DECISIVE project, "Advancing and Evaluating UAS for Defense," was held on October 12, 2022. The event feature research briefs on the UA-1 and UA-2 projects funded by the UMass Lowell (UML)/DEVCOM HEROES program. All presentations given at the event are available for download via a publicly accessible Google Drive folder. Below is a list of the presentations given: