Table of Contents
In the public safety drone space, I’ve watched people get excited about or host a drone performance test based on the National Institute of Standards and Technology (NIST), and I think that is misplaced energy.
NIST and others promote the course, leading to a testing threshold that can lead to some competency awards.
NIST and others say that drone pilot can measure their proficiency through the test. I’ve even observed pilots using the trial and competing on who can complete the course the fastest.
But Here is My Primary Issue
The comfort or reliance on the NIST test gives a false impression that this is a pilot competency test. To be accurate, this is a test of how the pilot can maneuver the drone up-down-right-left.
It is NOT a Test of Pilot Competency
Being a competent pilot involves much more than coordination. Unfortunately, being a qualified pilot is probably only 20 percent coordination.
It is the missing majority of skills a competent drone pilots needs. But, unfortunately, the test presented today never addresses those more critical factors and can’t.
So What is Missing?
A competent pilot must possess good Aeronautical Decision Making and Risk Management skills and factor in environmental, situational, and flight compliance issues.
Speed alone is never a primary factor in being a good pilot. Many Certificated Flight Instructors teach their students the “slow is smooth, smooth is fast” approach, which means taking purposeful action results in a better outcome than taking fast action.
In a flight emergency, a good pilot takes purposeful action, not the fastest action.
Simply focusing on speed makes you a more dangerous pilot, not a better one.
NIST says, “Emergency response organizations can quantitatively establish reliability and confidence during training before drones and remote pilots are deployed in real-life situations,” said NIST mechanical engineer Kamel Saidi.”
But I feel that is a significantly misguided and naive view of competency skills.
The FAA says, “Safe operations in today’s National Airspace System (NAS) require integration of aeronautical knowledge, risk management, and flight proficiency standards. To accomplish these goals, the FAA drew upon the expertise of organizations and individuals across the aviation and training community to develop the Airman Certification Standards (ACS). The ACS integrates the elements of knowledge, risk management, and skill listed in 14 CFR part 61 for each airman certificate or rating. It thus forms a more comprehensive standard for what an applicant must know, consider, and do for the safe conduct and successful completion of each Task to be tested on both the qualifying FAA knowledge test and the oral and flight portions of the practical test. [No mention of speed or coordination.]
During the ground and flight portion of the practical test, the FAA expects evaluators to assess the applicant’s mastery of the topic in accordance with the level of learning most appropriate for the specified Task. The oral questioning will continue throughout the entire practical test. For some topics, the evaluator will ask the applicant to describe or explain. For other items, the evaluator will assess the applicant’s understanding by providing a scenario that requires the applicant to appropriately apply and/or correlate knowledge, experience, and information to the circumstances of the given scenario. The flight portion of the practical test requires the applicant to demonstrate knowledge, risk management, flight proficiency, and operational skill in accordance with the ACS.”
And Here is Why Fast is Dangeous
I’ve observed myself and others attempt to fly fast, and that only appears to create tunnel vision on the aircraft and not factor in surrounding factors.
For example, at a public safety incident scene, someone walks under the drone’s flight path. You lose sight of the drone; the mission changes midflight, an unusual software message appears, etc.
Almost none of the critical risk factors of an uncertified UAS aircraft have anything to do with flying the fastest on a bucket test course.
From my point of view, as a Chief Pilot for the department and an experienced aviator, I would take any pilot that scored high on the NIST course challenge back to square one and retrain them.
I agree that a pilot should put the drone in a position to accomplish the mission, but it has to be within ALL the mission parameters and not just align yourself for a bucket.
Here is How the NIST Course Should be Utilized
The NIST course could be a tool to use during a practical test that involves situation-based issues that test pilot thought process and operational competency and not just coordination.
It would be much more effective to put the pilot in an unknown situation and observe their decision-making and flight preparation. I would urge some scenarios to be marginal or explicit no-go situations to measure when a pilot weighs risks and comes to the logical conclusion that the flight is not safe to conduct.
I can think of many situations where a pilot that says they are unable to continue the flight or conduct the flight is substantially more competent than one who thinks speed and coordination make a great pilot.
Forcing or encouraging pilots to compete on coordination test scores or speed, in my opinion, creates less safe pilots. It also gives public safety agencies a false sense of competency and more exposure to operational liability.
I don’t want a pilot to roll up to a scene and launch the drone the fastest. Instead, I want a pilot that will weigh all risk factors, decide how to accomplish the mission, and execute that plan with intent.
Last Point to Ponder
If the ability for a pilot to execute flight operations the fastest were an essential factor, it would be part of the Federal Aviation Regulations. I encourage you to read through Part 107 and see if you can find any reference to flying the fastest.
Finally, if speed and coordination were the most crucial skill, the paramedic that ran the fastest and jumped over the most obstacles into the incident would be the highest-paid. That’s ridiculous.
Thank you Steve. I was a Navy officer on a guided missile frigate with two tours to Vietnam. As a trained and qualified Officer of the Deck (i.e., commanding the ship when the captain is not on the bridge) the most important skill was situational awareness. I had to be aware of what was happening on the bridge, monitoring the radar, keeping an eye out for traffic, keeping navigational rules of the sea in mind, etc. Doing all this and training the Junior Officer of the Deck standing watch with me. Flying my Mini 2, I find I need that skill watching the wind, looking out for obstacles (especially birds), monitoring the screen on my smart controller, etc. I try to fly smoothly and steadily so I can maintain that situational awareness. Thanks for your thoughtful post and keep up the good work.
Thank you for your comment and insight.
Flying for time alone is an error and not the only portion of the NIST course. I have flown the NIST course biannually since 2018 and there is a definite crawl, walk, run process and a slow is smooth, smooth is fast dynamic to the NIST. If it is not how you experienced then NIST is being used as a stand alone solution, which is not. NIST is one portion of what should be a comprehensive training in everything you describe in your article/post. The alternative is to follow your framework with no hands-on, practical exercise? NIST sUAS lane should be a portion of a comprehensive training plan and any Chief Pilot that uses the standard of only the lowest time is failing to train their pilots in CRM, Risk Management and mitigation, and ADM. The benefit of the NIST lane is that the RPIC must know the control station system. NIST is only as good as the proctor and the total checkride and training system.
I completely agree. Thanks for sharing your feedback.
Steve: Thank you for writing about the NIST/ASTM tests that are being developed. I would like to take this opportunity to clarify a few points if I may.
1. With regards to speed: The primary metric in the tests that NIST is helping develop is always completion and never time. Time (or task rate) is usually a secondary metric. However, it is not uncommon for pilots who are going through the tests amongst their peers to have friendly competitions to see who can achieve the fastest time. This is neither how the tests are intended to be conducted nor what NIST advocates. Nevertheless, certain organizations may choose to require their pilots to finish a test within a certain reasonable time limit (which is similar to a time limit imposed on a pilot’s knowledge exam). Furthermore, for practical reasons the tests are also designed to take no longer than the endurance of a typical drone battery (in the hands of an average pilot).
2. With regards to pilot competency: The tests that NIST is helping develop do not claim to measure a drone pilot’s competency in everything required to effectively use a drone on a certain mission. These tests are simply ways of measuring a pilot’s proficiency at executing very specific skills. Some of these skills are very basic skills such as moving up, down, left, and right, while others require coordination between all of those controls in addition to camera pan, tilt, and zoom. Other skills yet focus on a pilot’s ability to use some of the autonomous features that may be available to them or to effectively use the user interface. None of the tests that NIST has helped develop are meant to test certain other aspects of being a remote pilot such as aeronautical decision making.
3. With regards to the basic nature of the tests: The tests that NIST is helping develop may be conducted under varying conditions and within different scenarios, and users are encouraged to practice the tests under more operationally-significant conditions (e.g., in the rain, with gloves on, while other activities and distractions are present, etc.). There is nothing to preclude the tests from being used that way, as long as the specific conditions under which the tests are conducted are recorded. The basic tests that NIST is helping develop are meant to evaluate basic drone capabilities or pilot proficiency. For pilots, one can think of these tests much like a motor-vehicle driving test that is conducted on a closed test course. While a driver’s ability to deal with traffic, pedestrians, other passengers, phones, etc., may not be challenged on a driving test conducted on a closed course, their ability to handle a vehicle and conduct basic maneuvers is (e.g., parallel parking, right and left turns, starts and stops, etc.). This is one of the reasons why some departments of motor vehicles require a newly-issued driver’s license holder to be on a probationary period of 1 year or more. They understand that a 15 minute knowledge test and a 20 minute driving test do not prove that the driver is fully competent and able to deal with every type of situation, and that more “on-the-job” training is required. The tests that NIST is helping develop are just the beginning. There are more tests being developed for more advanced skills as well as to test basic skills in more operationally-significant scenarios.
4. ASTM Sub-Committee E54.09 on Response Robots for Homeland Security Applications (where these tests are being developed) welcomes input and participation in the process of developing these tests in order to help make them better. Anyone interested in participating may contact NIST or ASTM (athttps://robottestmethods.nist.gov or https://www.astm.org/COMMITTEE/E54.htm, respectively).
I completely agree with your statement, “The tests that NIST is helping develop do not claim to measure a drone pilot’s competency in everything required to effectively use a drone on a certain mission. These tests are simply ways of measuring a pilot’s proficiency at executing very specific skills. None of the tests that NIST has helped develop are meant to test certain other aspects of being a remote pilot such as aeronautical decision making.”
You have designed a very specific skills test. It is NOT a test for public safety pilot competency, as some claim, and there is a misperception that it somehow is the measurement pilots should pass before being released on mission flights.
In fact, the course is probably about 10% of the skills needed to be a safe, competent, and effective public safety or first responder pilot. Hopefully, you can help to correct that misperception among people using the course.
I’m a big fan of practice like you fly and creating an atmosphere where a public safety pilot could perceive speed is the goal or the desired outcome, creates, in my opinion, an atmosphere of incomplete training.
Now, if you combined and integrated the NIST course with scenario-based training that required the pilot to have situational awareness, demonstrate risk management analysis, regulation compliance, and Aeronautical Decision Making, then you have a more realistic training scenario.
Steve