Frequently Asked Questions:
Why did we change the course evaluation form?
For many years faculty have been asking for a form that would provide them with better feedback to help them monitor and improve their teaching. The previous form was very short and very general, and did not give instructors information that could be used to improve teaching practice. In our pilot tests students believed that the new form was more relevant to their courses, addressed issues they wanted to comment on, and would give instructors better feedback on their teaching.
This instrument seems long - will students fill it out?
Compared to several other universities, our old instrument was very short. The most widely used course evaluation instrument developed by the Educational Testing Service (ETS) is about the same length as the new FCE, with 45 base questions and an additional 10 supplementary questions that instructors can add. Several other institutions also have course evaluations of comparable length. In our pilot studies students reported that they would be just as likely to complete the new form, and that even though the new instrument was longer, it was more informative for them and for instructors. Carnegie Mellon instructors in our pilot studies were also very pleased with the quality of feedback they received from the new instrument and overwhelmingly preferred it over the old one.
What will happen to the response rate if the evaluation is online?
Based on two large-scale pilot studies and two summer sessions deployments, the online response rate across courses was 46.5%, compared to a 52% response rate with the old instrument distributed in-class. As with the old instrument, individual courses had very different responses rates, with some courses receiving as low as a 16% response rate and others receiving a 100% response rate. Data from other universities suggest that when moving to an online system the response rates increase as freshmen move through the ranks over the four years and online FCEs become a cultural norm.
Several studies have shown that the best way to encourage your students to respond is to share with them why their feedback is important to you and what you do with the information. Most students don't believe faculty even read the evaluations and report that they would be more likely to complete FCEs if they believed they were read and used for improving teaching and education on campus.
The most valuable source of information to me is the comments. How will online administration affect the quantity and quality of comments?
Studies have found that a greater percentage of students write comments and write longer comments when completing a faculty evaluation online compared to in-class with a paper-and-pencil. No differences have been found between online and in-class in the percentage of comments that are classified as positive, negative or mixed.
Have any differences been found in the ratings students give when responding
on-line vs in the classroom?
Our own studies and those done at other institutions have found no systematic differences in FCE ratings or the characteristics of students who complete FCEs in-class vs. online. In one of our studies we randomly assigned students from each of two large courses into an online vs. in-class administration and found no differences in the ratings or the characteristics of the students in terms of anticipated grades, GPA, or other potentially relevant variables.
Will students' honestly answer questions about their effort and participation?
In our pilot studies and in interviews with students about the instrument, many students were quite forthcoming about their attendance and effort, with a high percentage of them explaining their lapses and taking responsibility for their learning in the subsequent comment section. Of course this measure won't be 100% accurate, and taking attendance or using performance measures for participation would be more accurate. But for instructors with large classes, where taking attendance taking is too time-consuming, these subjective reports give some indication of student effort. One of the main reasons for including this section in the FCE was to give students an opportunity to reflect on their behavior and think about their responsibility for their own learning. Students in our interview study reported that answering these questions forced them to think about their behavior and how it related to their performance and stated that this kind of reflection would make them more likely to change their behavior in the future.
Will I be able to add or remove questions to make the evaluation more
relevant to my course?
Our plans for future system development include the ability to customize the instrument to better meet your course and goals. A core set of questions will continue to be standard for everyone but for some sections, such as Student Effort & Participation, Learning Outcomes, and The Course, it will be possible to add or remove items.
Some of these items are irrelevant to my course. How will they affect
my ratings?
"Not Relevant" responses do not contribute to the ratings. If all or a significant proportion of students answer an item with "Not Relevant," the rest of the responses for that item will not be computed and will not contribute to the section average rating.
Won't some students decline to respond because they believe they are less anonymous online than on paper-and-pencil?
The university takes the issue of student privacy and anonymity very
seriously, and all reasonable precautions to protect it are being taken.
To view the complete privacy statement, go to http://www.cmu.edu/fce/privacy/index.html.
The system uses Andrew IDs to identify the courses students are enrolled
in and to ensure that they have the opportunity to evaluate their courses.
Once they begin to fill out an evaluation, their Andrew ID and all personally-identifying
information are discarded. No personally-identifying information appears
anywhere on the evaluation.
When can I see my ratings?
As soon as your grades are submitted, and the deadline for grade submission has occurred, you will have access to your results. You will have the ability to view the data in several different ways, such as viewing only the summary statistics for each section, viewing response distributions and averages for each question, viewing only comments, or paging through individual evaluations. You will also be able to look at your ratings in comparison to those of your department, school, and the university. The online tool also provides some basic statistical tools that will allow you to filter your data to look at subgroups of interest (such as majors vs non-majors, or sophomores vs. seniors), or groups that answered specific questions differently (such as only those who anticipate receiving A's, or those who rated the course as "much too difficulty".) In addition, you will be able to view relationships between different items, such as the relationship between anticipated grade and rating of course, or between learning and rating of instructor.
Who can see my ratings?
Nothing has changed in the policy of who gets to see the data. As always, you the instructor, your department head, and your dean can view all the information. Students will only have access to the ratings for the questions; they do not have access to the comments.
How long will the data remain online?
The ratings from the questions will remain online in the same way as
the old FCE but comments will be purged after one full year (i.e.,
F04 comments will be purged in F05). If you want to maintain access
to your comments beyond that period you must download the data and maintain
it yourself.
I teach a course that is cross-listed with other course numbers. How will I see my Results?
When course information is entered into the Student Information System, cross-listing data is often incomplete, incorrect, or missing. We are working on a way to make it usable but for the present each course number will be represented as a separate course. However, you
can easily combine all the data from different course numbers into a single
report. When you enter the system to view your results, all the course numbers will be listed as separate courses. Once you select one course to view, you can then add all the other course numbers to it to create one aggregate data set. Or, if you choose, you can look at the data for each cross-listed course number separately, side-by-side, to see if there are systematic differences between students taking the course under different course numbers.
My courses include several TA-led recitations - will they be evaluated as well?
No, recitations will not be evaluated in this system. The new FCE instrument was developed to focus on a course in its entirety -- i.e., student effort, learning outcomes, instructor behaviors, and class resources and activities -- those things that faculty members designing the course have responsibility for and control over. As such, most of the questions are NOT appropriate for the roles and responsibilities of teaching assistants.
However, the Eberly Center does have both early course and end-of-course evaluation instruments designed to address the specific roles and responsibilities of teaching assistants in different instructional settings, such as discussion recitations, problem-solving recitations, labs, studios, grading, and/or project management. Many departments already use these instruments or some variation of them. These instruments are available for downloading at http://www.cmu.edu/teaching/assessment/courseeval_forms.html.
I co-teach a class with two other instructors. How will students evaluate me?
The current evaluation form consists of 4 main sections: Student Participation & Effort, Learning Outcomes, Instructor Behaviors, and The Course. When students access the evaluation form for your course, it will consist of the standard form but will include multiple Instructor Behavior sections, each one clearly identified with the instructor’s name. When you receive your feedback for the course, the Instructor Behavior section will only include the data that was completed for you, you will not see your co-instructors results. All of you will see the data for the other sections of the evaluation.
Will my course be evaluated if I only have one student officially enrolled?
No, the system will not evaluate one person courses for multiple reasons. First, it would compromise the confidentiality and anonymity standards that we promise our students. In addition, the data from a single student course is highly suspect because of the lack of anonymity. For feedback from single student courses other methods would be more appropriate, such as an outgoing interview with the student, an open-format questionnaire, or questions that target the specific aspects of the course that you want constructive feedback on.
Do we have any data on response rates from earlier deployments?
We have just completed the analysis of the deployment for F04 Mini-1 and the response rate was 70.7%, which virtually matches the response rate for the same set of mini courses from the last 4 years – 71%. The table below shows the previous response rate vs. the new instrument rate overall and by college.
| FCE Response Rates |
| |
2000-2004 |
F04 Mini-1 |
| Overall |
71% |
70.7% |
| SCS |
71% |
87.5%+ |
| CIT |
72% |
81.4%+ |
| MCS |
65% |
73.5%+ |
| H&SS |
87% |
67.7%– |
| CFA |
82% |
61.3%– |
|
