Rod's Letter to the FAA and ACS Committee Members

Rod's Letter to the FAA and ACS Committee Members

Yes, really! The Private Pilot ACS adds 363 Risk Management items to the PPL practical exam testing requirements. Students are responsible for knowing the answers to each and every one of these items, all of which require highly subjective answers (according to the FAA's own Risk Management Handbook). Furthermore, the flight instructor is responsible for teaching these highly subjective answers to his/her students.

The ACS requires the DPE to ask at least one risk management question from each task. How many tasks are there in the ACS? I count 51 tasks in the PPL ACS. That's 51 additional questions the DPE is required to ask. 

If we assume at least a 1.5 minute-answer for each subjective risk management question and answer and the time required to ask the question, you're looking at about an hour and a half extra time for the PPL checkride. And I'm being conservative with the times here. Try answering the following ACS risk management question in less than 1.5 minutes:

"Tell me about the risks associated with the range and limitations of [the airplane's] stall warning indicators (e.g.: aircraft buffet, stall horn, etc.). How would you identify, assess and mitigate these risks?"

The fact is that you won't find an answer to this question even if you look in the three reference sources listed under the slow flight task in the ACS. The answer to this question is mostly subjective. Yet, the ACS says the following in its preamble: "The ACS does not change the tolerances for any skill Task, and it is important for applicants, instructors, and evaluators to understand that the addition of knowledge and risk management elements is not intended to lengthen or expand the scope of the practical test."

Really? A PPL oral exam shouldn't take more than two hours to complete. A recent post on my Facebook site identifies a student who just suffered through a five hour oral exam for his PPL practical exam. Go figure.

 THE FOLLOWING IS A LETTER I SENT TO THE FAA REGARDING THE ACS

ACS Committee Members  
Federal Aviation Administration 
Regarding the ACS  
June 6th, 2016   

Greetings ACS Committee Members:

My primary goal in aviation is to ensure that the average person has a fair chance of earning a private pilot certificate at a reasonable costs and minimal regulatory burden. The ACS does not support either of these goals (see my 2013 response to the ACS—a response that has probably not been read by many ACS--if any--committee members).

After our meeting at Sun n’ Fun (I met with five ACS committee members) and my follow-up questions to you (i.e., the FAA person in charge of the ACS), additional research on my part has reinforced my belief that the ACS will not have any positive influence on the aviation accident rate. Instead, it will merely increase the educational burden on anyone who desires to obtain a private pilot certificate. To make matters worse, much of the ACS is based on fantasy-based thinking, incorrect assumptions, and a lack of evidence that would leave an aviation defense attorney giggling with pleasure. Unfortunately, the ACS is being forced on an unwitting general aviation community by fiat.

Let me begin by offering one example to make my point about how ill-conceived some aspects of the ACS are. The ACS committee promotes the use of scenario-based questions on the knowledge exam. Unfortunately, unless these questions are carefully constructed and vetted, they can be highly subjective and misleading. For example, the following is a knowledge test question that was, until last week, on the private pilot knowledge exam. This question was also used in the 2015 sample knowledge test questions. Clearly, someone created the following question, had it reviewed by those supporting the ACS philosophy, proudly presented it as an example of a scenario-based question for the FAA’s sample test questions, and used it as an actual question on the knowledge exam (as identified by two students on my Facebook site).

Here is the question the ACS members apparently believe will make aviation safer:

 QUESTION #28
The destination airport has one runway, 8-26, and the wind is calm. The normal approach in calm wind is a left-hand pattern to runway 08. There is no other traffic at the airport. A thunderstorm about 6 miles west is beginning its mature stage, and rain is starting to reach the ground. The pilot decides to 

1. fly the pattern to runway 8, since the storm is too far away to affect the wind at the airport. 
    2. fly the normal pattern to runway 8, since the storm is west and moving north and any unexpected wind will be from the east or southeast toward the storm. 

    3. fly an approach to runway 26, since any unexpected wind due to the storm will be westerly.

       (Source: PLT271 / PA.I.H.K4 Aeronautical decision-making as affected by hazardous attitudes.)

      This is a terribly misleading scenario-based question. Just in case it is not evident, the FAA has a minimum distance policy from thunderstorm activity (to say nothing of airline and military recommendations for gust-front avoidance while landing). Is the ACS committee saying that this recommendation no longer applies during landings? If you’d like to see additional discussion of this question (and other reasons why it’s such a misleading question), please take a look at my Facebook site and scroll down. Look at what your fellow pilots are saying about this question.

      Fortunately, a friend who happens to be an FAA inspector expressed my concerns about this question to someone at AFS-630. The question was promptly removed from the exam. What concerns me most is the quality, integrity, and accuracy of the other newly created scenario-based questions on the knowledge exam. While I’m not against scenario-based questions, I am against questions that are not well-constructed and properly vetted.

      Of course I am always willing to entertain any challenges to my arguments on this issue. To date, I have not seen a cogent, logical argument on how the ACS will reduce aviation accidents at the private pilot level. The best anyone can do so far is to make emotional claims and offer emotional pleas about how the ACS will reduce aviation accidents. No facts, just emotional declarations. That’s it. 

      What follows is a rather long argument reflecting my new concerns with the private pilot ACS. Clearly, this won’t have any effect on ACS implementation. My intent here is to go on record regarding my opposition to the ACS.

      Rod’s Challenge and the End Result
      It’s clear to me that my original attempt to dissuade the FAA and the ACS committee from changing the Practical Testing Standards (the PTS being a simple, clear, and practical “testing” standard) was an endeavor that never had the chance to succeed. The ACS was a fait accompli long before the FAA requested public comments on it. Given that a change to the PTS is not a regulatory issue, the FAA could implement whatever changes it felt were necessary in creating the ACS. It’s clear to me that the ACS committee decided that it knew what was best for the general aviation (GA) community and that it could implement the ACS without the need to publicly defend that product. Of course, the committee will point to the ACS Q&As as a defense of the ACS. Unfortunately, the ACS Q&As are essentially a dismissal of dissenting points rather than a logical rebuttal against them.

      One of the many things that are disturbing about the ACS is that it is scheduled for implementation on June 15, 2016, with no trial period offered to the general aviation public. It seems clear to me that the ACS committee felt that the ACS didn’t need feedback on its use by instructors in the general aviation community prior to its implementation. I would like to know why the ACS committee felt its decision-making ability was so superior and full-proof that it elected to deprive the general aviation community of an opportunity for an ACS “trial period” before implementation? (Keep in mind that the original 2013 request for comments couldn’t possibly solicit informed comments on use of the ACS, as the product was still in development and not in use at that time.)

      This is especially peculiar as compared to the implementation of the PTS. The FAA introduced the PTS on September 1, 1984, and made it a voluntary testing option for pilot applicants for a period of one year. The private pilot PTS became mandatory on September 1, 1985. The commercial PTS also debuted on September 1, 1985, and was likewise a voluntary option for one year. The flight instructor PTS followed on September 1, 1986, with the same voluntary one-year option. There is no practical reason whatsoever that the FAA could not or should not have offered the GA community the same option with the ACS (irrespective of whether the relevant knowledge test had been properly adapted at this point). At least the GA community would have had a chance to become familiar with the product before it was implemented by fiat from FAA headquarters.

      The Macro Perspective
      From a macro perspective, it’s important to remember that the FAA gave us FITS (FAA Industry Training Standards) in the early 2000s. FITS is now dismissed as ineffective by your branch of the FAA—the same one that’s giving us the ACS (at our SNF meeting, you specifically stated that you no longer want anything to do with FITS. On the other hand, I specifically recall that you were a “very” big fan of the program several years ago). Even the DOT appeared to have difficulty in attempting to find something worthwhile to say about FITS in its 2004 FITS assessment (and this was before the accident record had a chance to reflect that FITS offered no discernable safety benefit to GA). FITS injected higher-order thinking skills into the private pilot curriculum (yes, for TAAs or Technically Advanced Airplanes, but also for non-TAAs, too—as stated in the FITS literature). Therefore, the ACS is essentially a FITS-Lite spinoff, making the value of FITS a relevant topic for discussion.*

      (*NOTE: In your recent Q&A updates [the Q&As are now 28 pages long! Really?] you stated that “There is no connection between the ACS and FAA/Industry Training Standards (FITS) program.” Well, that is simply incorrect and terribly misleading. The ACS contains two very important components of FITS: risk management and scenario-based training—applied as SBT-based knowledge test questions. Are you not aware that the FAA’s FITS Q&As suggest that FITS was a tool intended to supplement the PTS to function similar to the way that the ACS now claims to do? Take a look at your own FITS Q&As from 2006: reference point #7. The fact is that the FAA can’t escape applying its own FITS philosophy to the ACS.)

      The practitioners of FITS clearly intended their project to inject HOT topics such as “risk management” into the private pilot training curriculum. This is exactly what the ACS claims to do, too. Look at the original request for comments on the ACS. The document stated: “The goal of the airman certification process is to ensure the applicant is ready to safely manage the risks of flight as pilot-in-command, consistent with the privileges of the certificate or rating exercised.”

      So how did the FITS philosophy help general aviation at the private pilot level? Well, not  much, if at all. The fact is that is the FAA has no proof—not one single fact—supporting the assertion that risk management introduced at the private pilot level, will reduce aviation accidents. What we do have is a 10 year experiment with FITS that shows it had no positive effect whatsoever on the aviation accident rate at the private pilot level. No effect! Ten years of solid statistical data are compelling and difficult to dismiss! 

       
      This type of higher-order thinking that the FAA seeks to implement at the private pilot level is what I call fantasy-based flight training (FBFT). It’s based on the feeling that, if something looks like a good idea and enough people have good feelings about it, then it has to be a good idea. Emotions, however, are not often the best guide to public policy.

      The fact is that private pilot applicants can be taught to think like ATP-rated pilots as long as they are willing to spend years in training and acquire 1,500 hours of flight time in the process. That, of course, means fewer private pilots. This might be a difficult concept to accept, but you can’t teach pilots to think “practically” and “meaningfully” in HOT ways until you first teach them to fly. Student pilots need to acquire basic flying experience first before higher-order thinking skills will have a meaningful effect on their behavior. Attempting to develop these HOT skills early in a pilot’s training essentially sabotages the learning process. This is a basic principle of elementary psychology and it should be self-evident to any instructor.

      Unfortunately, it appears that the ACS committee is not paying attention to psychology or history here. There have been many other well-intentioned educational groups that have imposed their untested “feel good” ideas on students to the detriment of those same students. Allow me to illustrate this point with an example.

      History Repeats Itself
      The ACS is supported with intentions that are strikingly similar in structure to those used in support of the Whole Language Reading concept (a popular idea originating in the progressive movement of the early 1930s). Its proponents—very intelligent, capable, and well-meaning educators—believed that, if young students were simply exposed to words and the context in which they were used, they’d learn to interpret, pronounce, read, and understand those words correctly. In the minds of these educators, learning to sound out phonemes relating to individual letters (the basics of reading known as phonics) and combining these sounds to pronounce words was unnecessary. Higher-order training was what excited all the educators (HOT? sound familiar?).

      This is, of course, is fantasy-based thinking, but it’s the same thinking that sustained FITS and now sustains the ACS (read my blog piece on the subject). In other words, if you feel deeply that it will work, then it will work—no proof necessary. Care to take a guess about what happened to children when they were deprived of phonics training? They didn’t learn to read well, if they learned to read at all. Even the uber-progressive educators of California had to abandon their Whole Language Reading silliness in 1995. You can’t sacrifice the basics (i.e., phonics) and hope higher-order reading skills will automatically develop. They won’t.

      Therefore, I’m asking the ACS committee to show me its evidence supporting the belief that insertion of risk management training at the private pilot level produces safer pilots. Please show me the proof. Remember, I can show you 10 years of empirical evidence supporting the conclusion that risk management training, situational awareness training, scenario-based training, etc., have no discernable effect on the fatal accident rate in aviation at the private pilot level. The only evidence anyone has offered so far is to point to an experiment conducted under highly controlled conditions in a flight-laboratory, using a small sample of students from an aviation college.

      Just in case the ACS committee thinks I’m against psychological training for private pilots, consider this. I am and have always been a big fan of the original ADM (Aeronautical Decision Making) training created by Dr. Al Diehl, Dr. Janeen Kochan, Transport Canada, etc. in the early 1980s. ADM was simple, practical, and did not interfere with the acquisition of basic, fundamental flying skills. ADM applied sound psychological principles to flight training. It was very effective and easily applied by flight instructors. Risk management, as applied to primary flight training, however, is fantasy-based as I’ll show below in Item #6.

      FAA and Loss of Control Accidents
      Isn’t it ironic that the FAA is now promoting loss of control (LOC) accident prevention along with improvements in basic flying skills? For example, over the past few years, the FAA has explicitly stated that it wants all pilots (GA and airline) to have a more thorough understanding of such basic concepts as angle of attack. It also wants all pilots to improve their stick and rudder skills. Yet, the branch of the FAA promoting the ACS seems to believe that it is a lack of higher-order thinking that’s primarily responsible for aviation accidents at the private pilot level.

      Once again, look at the results that FITS produced (or didn’t produce). Until recently, Cirrus, a company responsible for driving the early development of the FITS syllabus, had one of the worst safety records. When I asked the FAA manager (now retired) who was responsible for creating and promoting FITS to explain this, he replied, “Well, they [instructors] are just not doing it [FITS] right.” Really? Isn’t this the same rationale that those “fellow travelers” used in the United States in the 1950s to explain the failure of communism under Stalin? FITS could be of value during commercial pilot training and for the higher ratings, but not at the private pilot level, where a pilot applicant has yet to master precise airspeed control, much less stick and rudder coordination. I can only imagine the number of students forced to abandon their aviation ambitions because of the extra training time and training cost inspired by FITS.

      Airplanes Are Not Falling Out of the Sky
      The fact is that small airplanes are not falling out of the sky. However, when one does crash, it’s not primarily because the pilot didn’t assess the risks properly. The evidence indicates that the pilot lost control because of a skill-based error, not a decision-based error. That’s why it’s called a loss of control accident. Without realizing it, the ACS 
      committee’s objective is interfering with teaching the basics by attempting to simultaneously teach higher-order thinking. Inevitably, this results in deemphasizing the acquisition of basic flying skills. (Remember: whole language reading vs. phonics.)

      For instance, one major aviation university learned this when they developed their FITS-based private-instrument flight training program. In their Lessons Learned review, the abstract says, “Our findings indicate the need for inclusion of several maneuver-based lessons that facilitate the physical skills training required for some tasks (e.g., landing), early in the FITS syllabus.” Really? Let me translate this for you in my own words: We discovered that, by using a FITS-based curriculum and focusing on higher-order thinking skills, our students didn’t learn how to fly. So, we had to stop with the FITS nonsense and actually teach them how to land an airplane.

      This university was attempting to combine private and instrument training with the hope of building a professional pilot right from the start of training. Really? When you look at the school’s training syllabus, it has a student pilot flying an instrument approach on Lesson #6. In all my years in education, I’ve never seen a more naïve attempt to skip the basics of flight training in hopes of accelerating pilot development. Yet, this combined training program was sanctioned by the same FAA that produced FITS—but that shouldn’t come as a surprise to anyone.

      To date, I know of no flight school that has been successful using this combined curriculum in an ab initio flight training program. If you want someone to go from zero time to instrument rated pilot, here’s the secret: Teach them how to fly first, then teach them how to fly on instruments. Why that’s a secret to some individuals in the FAA is difficult to understand. Perhaps the response of one instructor from that university says it all. When asked about the combined private-instrument training, the instructor said: “My student could fly an ILS approach but he couldn’t land the airplane.”

      It’s a License to Learn
      Ultimately, we call a private pilot certificate a license to learn, because it’s a license that allows a pilot to “learn how to operate safely” in the aviation system. It’s not a license that ensures an airline pilot’s mindset. Trying to teach a 40 or more hour private pilot applicant (who has just gained enough experience to control an airplane safely) about “higher-order thinking” topics is doable if you want to extend private pilot training times and costs to a degree comparable to that required of a commercial pilot. Herein lies the problem.

      As I stated earlier, my primary objective in aviation is to ensure that a private pilot certificate is both affordable and free of excessive government (or industry/stakeholder) mingling for the average person who wants to earn one. Despite the ACS committee’s noble intentions and its protestations to the contrary, common sense suggests that the ACS will extend training times and costs.

      Here are a few more specific comments that speak directly to the lack of factual information supporting ACS development.

      1. Rod never spoke about the “intentions” of the ACS committee members

        Allow me to make one thing very clear. Not once did I challenge the qualifications of the ACS committee, nor did I ever question the motives of its members.

        In fact, on more than one occasion, I’ve written, “These activities are well intentioned, but I’ve long since realized that having good intentions is a highly overrated virtue.” My challenge doesn’t involve the will of the committee, only its deed, and it doesn’t attempt to divine the committee’s motives, either.

        It’s clear to me that every single committee member is a highly-respected professional and has nothing but good intentions toward aviation. My concern is with the end result of those good intentions—the deed—which I believe will ultimately hurt, rather than help, general aviation at the private pilot level.

        That said, I will use the ACS with my students simply because I am left with no choice. Do, however, keep one thing in mind. The FAA and the ACS committee have taken a once simple and easy-to-understand document (i.e., PTS) and doubled its verbiage by adding higher-order thinking skills and knowledge exam requirements. The result? Most pages of the ACS now look as textually dense as the backside of a car rental agreement. That, in itself, should have given the ACS committee pause during its development. My guess is that more than a few prospective aviation students will look at this document, roll their eyes to the back of their heads, and then walk out of flight school, never to return. 

        2. There is no compelling evidence whatsoever supporting the need for a radical change to the PTS.

          None! Where’s the evidence that says pilots are crashing airplanes, being harmed, or even disenfranchised because of the present structure of the PTS? There is no factual evidence supporting this claim. There is, however, strong evidence that the ACS won’t have any positive effects on the private pilot accident rate.

          For instance, how is aviation made safer by testing private pilot applicants on their “risk management” skills? Before anyone elects to answer this question, he or she should read the FAA’s landmark 2005 HFACS study (I’ve highlighted the essential parts). This comprehensive study clearly indicates that skill-based errors, not higher-order thinking errors, are responsible for upwards of 80% of all accidents. Only 29% of our accidents are HOT-based, and of these, 19% were initiated (preceded) by a skill-based error. The ACS is an attempt to reduce accidents at the private pilot level by applying the wrong solution to the problem. 

          3. Where is the evidence that the current private pilot knowledge exam isn’t serving the aviation community properly?

            One of the common protestations made by ACS committee members is that the knowledge exam contains ancient, antiquated, and outdated test questions. Really? The questions on LORAN (an often-cited example supporting outdated tests) were removed the same month the president shut down the LORAN system in 2010. That was over six years ago. ADF/NDB questions? They were also removed from the private pilot knowledge exam several years ago.

            But who says that ADF skills are irrelevant? We use ground track and bearing to station when navigating with GPS. There are still NDB approaches in the national database, too. And Garmin’s G-whatever has both an RMI needle and a bearing pointer in its HSI display. So, who says that ADF navigation skills are irrelevant? They aren’t.

            The problem was that some of these older questions remained on the FAA’s web-published sample test, thereby leading others to believe they were still on the actual test. Hmmm, most likely the sample test questions would have been removed earlier had the FAA not reduced the staff at AFS-630 from about 20 in the 1990s to about seven today. Yet, ACS committee members still promote the ACS as a tool to remedy irrelevant, confusing, and incorrect knowledge test questions. This seems to be more propaganda than fact to me.

            I have posed an ongoing challenge to anyone who can show me five outdated or irrelevant questions on the current private pilot knowledge exam (a challenge I’ve offered for over a year). So far, no takers. My guess is that the only bad questions on the exam are the scenario-based ones that are being currently added without careful consideration and vetting. 

            4. Does the ACS committee understand what it is that the FAA knowledge exams actually attempts to measure?

              Where’s the evidence that says the current private pilot knowledge exam doesn’t “truly measure what you [the FAA] are trying to test”? I’m not at all convinced that everyone knows what the FAA is actually testing for on these FAA knowledge exams, much less what it is attempting to measure.

              Let me explain. The 60-question FAA private pilot knowledge exam is not—I repeat—is not a comprehensive knowledge exam. This exam was NEVER intended to be a comprehensive assessment of an applicant’s knowledge in the same way that the SAT, LSAT, and MCAT tests are—these tests are assessment-based exams. FAA knowledge exams can’t possibly assess someone’s level of understanding (assessment of knowledge) in an area as vast as general aviation with 60 questions. By comparison, the SAT (appx. 170 questions), LSAT (appx. 130 questions), and MCAT (appx. 200 questions) exams are used to assess someone’s level of understanding over a broad spectrum in one or more areas of knowledge.

              According to the FAA’s own testing philosophy, the tests used in general aviation are personnel-based “certification” exams, which typically use fewer test questions. Personnel-based exams determine whether a person meets the minimum standards to be a pilot—minimum standards that involve studying a broad area of general aviation knowledge. These exams operate on the assumption that, if a student knows he or she will be asked a question in a certain area of aviation knowledge, then that student will study all the relevant material in that broad area of knowledge. In a sense, this type of testing simply “ensures” that a student will study enough to be a competent applicant because he doesn’t know what questions he’ll be asked to answer on the exam. To date, personnel-based testing has proven to be a practical and viable concept for aviation.

              Clearly, some individuals are under the impression that a short 60 question private pilot exam is a comprehensive exam. These individuals believe that the FAA’s knowledge exam evaluates the typical “comprehensive” ground training program that all pilots should have experienced during training. This is an unfounded assumption, because each FAA knowledge exam question should target specific categories of knowledge that the FAA wishes the applicant to study.

              Of course, many instructors claim that the questions on the private knowledge exam are all so well-known that students merely memorize the answers to these questions and make haste through the test. As a practical matter, however, most students do study diligently for the exam and don’t attempt to memorize the answers to questions. This is explained by the fact that the national average score on the PPL exam is 82%. It's pretty hard to make a case that the test is “too” easy (or that the answers to the questions are too well-known) based on that score. Clearly, the current FAA test questions are doing the job they were intended to do.

              Ultimately, there is absolutely no evidence that there was ever anything wrong with the PPL knowledge exam. There is, however, a lot of evidence and history to suggest that this exam is doing exactly what it was intended to do. 

              5. What about the “two guys in a room” who write these questions and have no current flight experience?

                On a number of occasions, I’ve heard ACS committee members refer to the “two guys in a room” who create knowledge exam questions. This reference does not accurately reflect the good work done by those at AFS-630. Fortunately, this comment doesn’t comport with anything I’ve learned about AFS-630 over the years. Let me offer my understanding about how AFS-630 actually creates knowledge test questions.

                In the late 1990s, the FAA had 20+ individuals working in its knowledge test development branch at AFS-630. Today, due to budget cuts, that number is seven (to the best of my knowledge). To create test questions, each inspector reviews NTSB reports, FAA enforcement activities, AOPA Flight Safety Institute findings, incidents and accidents, and directions from AFS-800, AFS-300, and AFS-200. Yes, the AFS-630 inspector might even notice an item in the local newspaper that reflects an area of aviation knowledge deficiency. So, what? There are many sources for this information.

                Once a new test question is created (or changed), it must be reviewed, evaluated, and approved by other responsible inspectors. Once corrections/modifications are made and these inspectors are satisfied with the new/modified question, it is sent to an editor and education specialist for another review (spelling, grammar, etc.). Once completed and finalized, the test question is placed on an actual test to determine the public’s response. During this period, the testing applicant won’t know if this is a new or older test question. This provides the FAA with an unbiased means of evaluating that question.

                Once this testing cycle is complete, the FAA’s assigned certification manager reviews the responses to that question. Then, test statistics are developed by the FAA’s in-house statistician to determine if the question’s validity and reliability fall within allowed parameters. In addition, the FAA’s statistician develops statistical values from each test question to see if the question is too easy or too difficult and then makes recommendations based on this evaluation.

                As an aside, the idea that test questions are old or outdated is quite misleading. AFS-630 assigns a certification manager to review every—EVERY—test question in his assigned question bank every two to four years (it’s now every four years, as I understand it). Each question is reviewed to ensure that the reference for the question is current and valid, the testing results are within guidelines, and the topic is still valid in accordance with FAR 61 and 91 (or whatever FAR is applicable).

                AFS-630 has always had a policy not to test something if there wasn’t a reference for that question in the public domain. Additionally, the certification inspector can change variables from older questions or develop new questions to highlight any apparent lack of knowledge as indicated by aviation accidents, incidents, or violations. That same inspector is also mandated to develop new questions from new reference materials, too.

                So, to say that there are “two guys in a room” who create test questions is misleading. 

                6. How does the ACS committee know the ACS won’t do more harm than good to general aviation?

                  The fact is, it doesn’t. During our meeting at SNF, when I asked if anyone knew how the original PTS was tested, no one knew. That surprised me. After all, if the ACS committee is going to make a major change in a traditional and effective training document (PTS), one would at least think that thorough testing would be done to prove its value. Here’s what actually happened in the development of our present PTS:

                  • When the original PTS was created in the mid-80s by the Examiner Standards Section (AFS-634) of the FAA, a plan of action was introduced and a 10-member examiner team was created.
                  • A PTS format was provided to all DPEs in the U.S.—1,845 DPEs—and 75% of all ASIs (aviation safety inspectors).
                  • Testing, examination, development, and feedback were accomplished over a two-year period, with DPEs and ASIs providing their input; this included recommendations, opinions, insight, corrections, modifications, and so on.
                  • Only those who were intimately involved in hands-on pilot certification were allowed to participate in PTS development.
                  • All the DPEs and ASIs were told that they would have to live with the PTS, so they should design it with usability in mind.
                  • When the PTS was implemented in 1985, pilots were given a year to use it or its precursor, the Flight Test Guide. Apparently, the FAA didn’t feel the need to offer pilots a choice here.

                  So how does this compare with the FAA’s method of testing the ACS? Hmm, not so well, I’m afraid. Here are the facts regarding Phase II testing of the ACS (i.e., the testing of private pilot applicants):

                  • Private pilot testing of the ACS was accomplished over a period of six months.
                  • Fifty-four students were involved in the testing.
                  • 20% were college-age students.
                  • 80% were of unknown demography (info not released by the ACS committee).
                  • Eight flight-training providers were involved.
                  • All testing was done in the Orlando FSDO area.
                  • Eight DPEs were involved in the testing.
                  • No reference is made to the number of students who started this program and dropped out. That information is just as important (if not more so) than the number completing the training.

                  It seems quite clear to me that testing of the ACS was relatively limited and offered only to a select group of participants. There doesn’t appear to be any indication that the CFIs or students involved were chosen at random to participate in this evaluation. I can’t imagine that the ACS committee would pick an instructor to train students who didn’t already have an established track record of training successes. But there’s no way to actually determine this, as none of this information is available for public inspection. Clearly, the FAA didn’t consider the ACS in need of extensive testing, at least not to the degree that the PTS was tested.

                  But that’s not the most peculiar aspect of the Spartan-like ACS testing. When I asked the individual in charge of ACS testing how he knew whether or not the ACS would increase training times and costs, he told me that none of the participants/representatives reported the need for a curriculum modification, nor was there any indication of increased training time, unsatisfactory lessons, or overall training cost.

                  That, however, doesn’t provide any proof that these things didn’t happen. If you don’t directly ask questions about training times and costs, you might not learn about these things. Given the small sample of test applicants, the limited number of DPEs involved, and the soft-questionnaire method of data collection, there is insufficient evidence here on which to place any confidence in the efficacy or value of the ACS.

                  Finally, what is the ACS committee’s plan if, for some reason, the ACS turns out to increase training times and costs? I doubt anyone has ever considered such a thing. 

                  7. The ACS committee says that the ACS’s risk management requirement won’t increase the subjectivity of the practical test while the FAA suggests that it will.

                    The ACS expects private pilots to identify, assess and mitigate risks on their practical exams. Please tell me what type of advice the FAA offers on assessing the probability and acceptability of risk in each task-specific area of the ACS (to “assess” a risk means to identify the probability as well as the acceptability of that risk).

                    The fact is that the FAA offers no advice whatsoever to help private pilot applicants calculate the potential of an assumed risk, much less whether that risk is acceptable or unacceptable. Yet, the ACS committee wants everyone to believe that the ACS won’t make the private pilot checkride more difficult, more ambiguous, much less more subjective. It’s clear to me that the branch of the FAA promoting the ACS isn’t clear about the fundamentals of its own risk management philosophy. Let me explain.

                    There are no FAA publications—none whatsoever!—that meaningfully inform a pilot about whether or not a specific risk is “worth taking.” This is entirely a subjective evaluation made by the pilot. You’ll never hear the FAA suggest that it’s OK for a VFR-only pilot to fly VFR “over the top” as long as there is less than a 5% chance of having a fatal accident. This would be silly advice to offer if such a thing could even be calculated for any specific individual. Yet, isn’t it strange that the FAA expects private pilots to assess a risk without providing any meaningful guidance on how to do so?

                    The reason the FAA fails to offer this advice is because it knows it can’t—and even says so! The FAA’s own Risk Management Handbook says that its risk management philosophy is “not a formal methodology of risk assessment.” Translation? It doesn’t actually allow a pilot to assess the quantitative value of a risk, much less decide whether or not taking that risk is acceptable or unacceptable (the latter being the single most important aspect of any “practical” risk management strategy). That’s why the FAA says that its risk management strategy merely “prompts a pilot to look at the simple realities of what he or she is about to do.” Nothing more.

                    The fact is that it is practical “flight experience” that allows a pilot to estimate the probability of being affected by a potential risk and whether or not to take that risk. This is why an unrestricted ATP certificate requires 1,500 hours of flight experience. Experience, however, is not what a private pilot applicant has. Therefore, it’s unreasonable to ask this applicant to make risk “assessments” associated with flying. What that applicant can do is identify and avoid hazards, something that they can easily be trained to do. That’s all that can be reasonably expected of someone with only 40 or so hours of flight training.

                    So what will private pilot applicants do on their checkrides if they can’t actually assess a risk? They’ll merely offer a sufficient number of vague, general, feel-good statements about the risk in question, smile a lot, then hope the designated examiner buys it. If this isn’t the height of subjectivity, I don’t know what is. The risk management requirement of the ACS is simply another example of FAA fantasy-based flight training. 

                    8. Exactly who is prepared to use the ACS?

                      The general aviation community is clearly unprepared for the ACS. Let’s remember that pilots have been in training long before the “implementation date” of the ACS was formally announced. Are we expecting these pilots to modify their training to accommodate the ACS? After all, many pilots will be responsible for unfounded and unsourced risk management information on the practical exam—information for which they have had no formal training. Why wasn’t a trial period offered where both the ACS or PTS could be used?

                      DPEs are also unprepared to implement the ACS. DPEs are required to formulate a written plan of action for each pilot applicant. What formal training have DPEs had on the ACS that allows them to formulate such a plan? The word on the street is that most DPEs have had very little formal training on the ACS.

                      Flight instructors are also unprepared to implement the ACS. As best I can tell, the courses offered by industry groups for CFIs on the ACS have only recently become available. In our April, 2016, Sun n’ Fun ACS meeting with an ACS committee member stated that all risk management questions in the ACS will have their answers sourced to some FAA document. Really? In the ACS’s Maneuvering During Slow Flight task, the following risk management question is asked, “The applicant demonstrates the ability to identify, assess and mitigate risks, encompassing [the] Range and limitations of stall warning indicators (e.g.: aircraft buffet, stall horn etc.).” Can you show me where this “risk management information” is covered in any of the references listed in the ACS for that task? I don’t see any coverage of this information. 

                      9. Very few individuals voluntarily join a committee so they can do the smallest amount possible in fulfilling the ambitions of that committee.

                        It’s only natural for a “voluntarily assembled” committee to strive for big changes, even when these changes are unwarranted. Believe me, I can certainly understand the enthusiasm to do something big. It’s in our nature to do so, especially when we feel we are making a positive change in aviation safely. It’s my belief, however, that the ACS committee could have had a positive impact on aviation by making small but effective changes in the PTS and creating a separate Knowledge Test Standard. Had it done this at the private pilot level, I would have been its biggest fan and supporter. 

                        My Recommendation
                        As I see it, there are simply too many unanswered questions regarding the ACS’s validity and reliability for it to be foisted on the general aviation community at this time. Sorry, but 54 students (all of which have an unrevealed “kept secret” demography and/or background) over six months with eight DPEs is not an adequate test.

                        Therefore, I believe that implementation of the ACS on June 15th, 2016 is a completely unacceptable date for the general aviation community. Given these concerns, I am suggesting that the implementation of the ACS be temporarily suspended based on the following recommendations.

                        For the next two years, utilize the DPE recurrent training cycles to introduce the ACS and seek feedback from each DPE as was done with the implementation of the PTS in the 1980s. DPEs will be required to review the ACS during their mandatory on-line pre-course requirements and again during their one-day FSDO residence requirement.

                        During this same period, require all CFI revalidation programs (web-based or in-person programs) to train CFIs on the use of the ACS. This would require a minimum of two years before the ACS would be implemented and would ensure that all revalidation course training cycles are covered for each CFI.

                        At the completion of this two year period, offer a one-year introductory time period for the ACS where flight instructors can use either the PTS or the ACS with their students. This seems like the only reasonable plan of action given the haste with which the ACS is being implemented. 

                        Finally…
                        Had the ACS committee not interfered with the private pilot PTS and only made the appropriate knowledge test modifications, you wouldn’t be hearing from me. I’m not overly concerned about how pilots applying for a higher rating will cope with the ACS. These individuals have sufficient experience to make informed decisions about the ACS and how it might affect them. Private pilot applicants, however, are not capable of making informed decisions about the ways in which the ACS might affect them.

                        That said, I’m always willing to entertain any cogent, logical challenge to these ideas. I’m a sucker for a logical argument and will roll over immediately when confronted by one. To date, I have yet to see even one article that logically supports the claims made by the ACS committee.

                        Sincerely,
                        Rod Machado 

                         

                        Leave a comment

                        Please note, comments need to be approved before they are published.