<p>I just came across an article entitled “Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors.” It was published in the most recent issue of the Journal of Political Economy. The authors (Scott Carrell of UC Davis and James West of the U.S. Air Force Academy) begin by noting that one can measure a professor’s performance by using student evaluations, students’ performance in the current class, or performance in follow-on classes. The authors have data from calculus classes at the AFA. Students are assigned to faculty members randomly, a common syllabus is used, a common exam is given, and the exams are graded jointly (e.g., Professor A grades question 1 for all sudents, Professor B grades question 2, etc.) All of these features are intended to keep “all else equal.” </p>
<p>Here is the punchline: </p>
<p>
</p>
<p>They also draw conclusions about student evaluations: </p>
<p>Interesting study, but I don’t like the setup. Forcing professors to stick to a common syllabus might fix one variable at the expense of not allowing professors to organize the material as they see fit, which will invariably affect their teaching performance. My all-time-favorite math professor did a less-than-stellar job when the department mandated him to follow a textbook he disliked.</p>
<p>I wonder how learning outcomes compare when professors have free reign over a course vs when the course structure and learning goals are predescribed.</p>
<p>It can matter. However, in some schools where the prof stands in front of the lecture hall with his canned lessons and leaves the students’ questions and discussions to recitations run by TAs, it isn’t going to matter much. I do look to see if there are too many adjunct profs on the faculty for the subjects of interest for my kids.</p>
<ol>
<li><p>I don’t see how they could draw any conclusion that goes beyond the teaching of a single subject – calculus. (Maybe not the same considerations when it comes to the teaching of history or philosophy?)</p></li>
<li><p>All the data comes from students at the US Air Force Academy, who are students who have been pre-selected for characteristics relative to aptitude and learning style. (Not so many artsy types at that particular school? How would these findings compare to, say, learning patterns in French classes at Bard?)</p></li>
<li><p>While I am sure their data is sound, I suspect there may be some strong bias in the conclusions they draw from their data. (Essentially: Student evaluations should be disregarded in tenure decisions because students don’t know what’s good for them.) Another equally valid conclusion might be that students who are taught calculus by teachers who fail to give them much encouragement and guidance are forced to rely on alternative strategies to learn: they either need to figure it out on their own, form study groups with other students, or find alternate resources like on-line math tutoring sites in order to pass the class. Such independent learning skills come in handy down the line when they are taught by equally inept and uncaring instructors in more advanced math courses.</p></li>
</ol>
<p>Anyone who’s had the good fortune to encounter a really good Professor knows that it is important. That doesn’t mean you can’t get a good education without such encounters …</p>
<p>@calmom: You are right that the study is limited in scope, so one should be careful in drawing broad conclusion, but parts do ring true to me. An internal study at my institution showed a positive correlation between grades and teaching evaluations. Some departments tended to have higher grades and higher evaluations while others (e.g., mine) had lower grades and lower evaluations. Those departments giving out high grades argued that they simply attracted better students. Eventually, someone debunked that myth by looking at students’ GPAs within the major and outside the major. Imagine that in Department A, majors have an overall 3.2 inside the major but 3.4 outside. In Department B, majors have an overall 3.6 inside the major but 3.1 outside. This was the general pattern: departments with low grades tended to have students who performed above the average outside the major, suggesting that they were actually the stronger students. Again, we did not do a full multivariate analysis with all the bells and whistles, but the results are suggestive. </p>
<p>It makes perfect sense to me that untenured faculty members and adjunct faculty members would care more about teaching evaluations. If, as at AFA, you do not have the freedom to raise grades unilaterally, the next best alternative is to teach to the test. If jane and John are otherwise indentical, but Jane has tenure and John does not, I would expect that John would care more about his teaching evaluations, which would make him more myopic. </p>
<p>The cautions about using teaching evaluations for tenure decisions also ring true. The evaluation of teaching is usually the weakest part of tenure dossiers. (I say this having sat through over 200 cases, across all departments.) Even when there are reports of classroom visits by senior faculty, they are often of little probative value.</p>
<p>I don’t think that untenured professors necessarily teach to the test on purpose. It takes a lot of experience teaching courses at various levels to see how the insights (or lack thereof) from one course affect learning later on. Calculus is a good example. If I had to teach a calculus class right now, I am not sure what aspect of the subject I would focus on. Strategies to evaluate tricky integrals? How to phrase “real-world” problems in the language of calculus? The underlying mathematical theory? Numerical methods?</p>
<p>It might take a little while to find a good balance. It might also be problematic that courses are taught by professors who were the at the top of the pack in their undergraduate years. Maybe they were able to extract deeper insights from computational examples themselves, when most students would need those relationships to be pointed out explicitly.</p>
<p>I have noticed that older math professors tend to focus more on insights than skills in their classes than younger faculty. But that seems to be a gradual change, not marked by a tenure decision.</p>
<p>I don’t think you can measure professor quality just by how experienced they are and how many qualifications they have. Some people are excellent academics with a long list of credentials, but they’re just not good teachers. Some people are excellent, natural teachers even if they’ve only been in academia a few years. So while the premise of the study is interesting, I’m not sure that the set up really works?</p>
<p>The quote that Coase gave is a little unclear. In plain language, the students of the experienced professors got worse grades in the calculus class, but better grades in the follow-on classes like physics that require a student to know calculus. In other words, the inexperienced professors were teaching how to take calculus tests, and the experienced professors were teaching calculus. </p>
<p>SmithieandProud, I’m not sure what your objection is. This was not an artificial situation. These were real professors and real students in real classes at the Air Force Academy. In this real situation, the experienced professors turned out to be better at teaching calculus than the inexperienced professors.</p>
<p>I once taught a theory-based semester of Calculus I as an adjunct. I had 32 students. In the room next to mine (at the same time) was a tenure-track faculty member with another 32 students. At least 1/3 of his students dropped the course before the drop date (I don’t know how many failed). I lost 2 to dropping, and had 2 who failed. </p>
<p>Clearly, there was some major difference between how we approached the material.</p>
<p>On the reviews, some of the students complained that I did too much theory, from which I knew that I had done enough theory (one of my concerns). Others said I did too many physics examples. Basically, freshmen are often not good reviewers, since they don’t know what to expect or have otherwise unrealistic expectations of what the class should be like.</p>
<p>Certainly the quality of the prof’s ability to speak English matters.</p>
<p>A common question to student tour guides at all college vistis with DD was: Do you ever have professors you cannot undrestand? The answers were alarming.</p>
<p>I was stunned when I learned that my DS was taking a class with one prof, but was taking exams written by another prof, i.e., departmental exams. What a fiasco. My DS’s class did terrible on the departmental exams while the students of the prof writing the exams fared much better. Overall, I think my DS was cheated out of what could have been a real learning experience. </p>
<p>Each prof has their own approach to teaching. Each prof knows what they have emphasized in class and, thus, can use the exams to reinforce the material. </p>
<p>Methinks departmental exams cover up who is the lousy teacher and who is the talented teacher. Not everyone can teach well. It is a talent. How can a university determine a prof’s teaching ability if that prof isn’t allowed to determine course material and testing style?</p>
<p>By the way, I taught college for 20+ years and managed to maintain a bell curve all 20+ of those years. Always had good evaluations - even from students in my 180-student classes. True story: I once had a student who failed my class, but gave me a glowing evaluation. She/He took full blame for failing the course and named several aspects of my teaching style that she/he appreciated. </p>
<p>By maintaining control over the material and testing, I was able to evaluate my own teaching skill (and, yes, occasionally I wrote a throw-away question) and tailor each class to the students enrolled in that class (and, yes, each class has its own “vibe” - some classes gelled better than others). What happened to academic freedom?</p>
<p>Departmental exams show who is not teaching the materials as outlined by the curriculum. Or at least not teaching them well. The answer is not for the profs to be making up their own exams. If you are taking certain courses, there are certain things you absolutely need to know well in order to build a solid foundation for future courses. Premed students can really be in trouble when taking MCATS if the professor did not cover what is presumed was covered. </p>
<p>I know that at some schools, the % of kids who fail the exams is examined and the whoever is teaching a section with bad stats, is warned that the material covered by him/her is not adequate for the students to pass. This is especially relevant if the students who are doing poorly on the exam are doing well on other aspects of the course. </p>
<p>Nearly all of my oldest son’s final exams which were responsible for 60% of the grade, were departmental. This allowed the department to pass those who met department standards, not someone who could pass a test his teacher happened to use covering what that teacher covered.</p>
<p>If you are teaching a seminar course or other type of course that does not have essential information that needs to be covered, it is not an issue. You can pick what you cover in the course and what you want to test. But for some subjects there is not that luxury.</p>
<p>In fields where knowledge is cumulative, I don’t want my professors to have the academic freedom to not teach a crucial part of the material and then not test it. It does me no good to get an A in calculus, but then be sitting in physics class not being able to solve the improper integral in front of me because my calculus prof didn’t get to improper integrals.</p>
<p>This isn’t about academic freedom. The point of having people with advanced academic degrees teach undergraduate courses is that there is supposed to be general agreement within a field about what beginning students should know. If there isn’t general agreement, then there’s a problem. In a well-run department, the people in charge will track what is being taught in order to protect their undergraduates.</p>
<p>Looking at how well students do in follow-on courses is extremely important. In my opinion, it is the best available way to measure professor quality. Students may love or hate a course, but in the final analysis the main purpose in taking one is to master some body of material. </p>
<p>Student evaluations are generally worthless for the reasons quoted, especially in introductory courses where students may be surprised and upset that expectations are quite different than they were in high school and one cannot get a high B just by handing in all the assignments.</p>
<p>Properly designed open-ended student evaluations may be helpful for smoothing out glitches in how a course is run and letting one know what students like/hate about it, but the students do not have enough experience to comment on class content or the way in which their mastery of the material is measured. Using success in higher level classes as a measure also helps separate the classes that make students work hard with a bunch of busywork from those in which the professor does his job of isolating and emphasizing the most important areas of study from a mass of material.</p>
<p>Colmom2 makes excellent points about the hallmarks of a properly run class. Having someone else write the exam does the students a huge disservice. People serious about actual teaching know what they’ve taught, have a good feel for when the shutters of boredom or ignorance slam shut during lectures, and take the time to tailor tests to their classes. The fact that many college courses test using nothing but multiple choice exams boggles the mind. </p>
<p>The service academics are good places for this kind of study because the student bodies are controlled, the curricula are controlled (hence the departmental exams), and classes are filled by random assignment. Reduces the variation created by things other than the variables of interest.</p>
<p>Having someone else write the exam particularly to departmental and academic standards is an excellent way to make sure the class i covering the right information. The professor can supplement all s/he wants and give his/her own exams, quizzes and papers for the rest of the class standards. At my son’s school, 60% of the grade was based on the departmental exam. That gives 40% the teacher can play with. These ratios can be adjusted. But there should be some safeguards that the course is covering the material the student is going to need.</p>
<p>All I know is my DS wanted to learn, say, Chemistry, but the department was so busy using it to weed out the pre-meds they forgot that not everyone is on that track. They ended up curving the heck out of the class because the vast majority of students were set to fail the course. They actually designed the course to flunk students. How is that teaching the material? The point of college is to go there to learn, not to become a statistic. </p>
<p>He had one prof (in another area) who refused to follow the departmental line and use the recommended departmental book, and he wrote his own exams. My DS learned so much in that class. His friends who were stuck with the “departmental” profs hated the course and were not as proficient in the subject as my DS and the other students in his class.</p>
<p>There is no benefit to teaching to the test - if the prof even knows what’s going to be on the exam. His math prof (a new prof at the college) followed the departmental syllabus, and his students ended up with terrible grades because the prof that wrote the exam stepped outside the syllabus. Needless to say, the exam prof’s students did great since he taught to his test while the other students of other profs following the syllabus ended up with lower grades.</p>
<p>There is a benefit to teaching to the test. Not 100% and not to the exact test but making sure the kids are able to do well on the test. That is the purpose of AP exams as well, to make sure the student knows certain information. For those who are going on to graduate school in these disciplines, or taking the MCATs or getting jobs that assume the student knows certain material that was in the courses taken, it is essential that those topics are covered. A departmental exam covering those things show who is teaching them and who is not. It’s wonderful to teach all the extras and instill that love of learning, but the essentials also need to be covered.</p>