I am writing in response to your letter of December 8, 1999 in which you empaneled an ad hoc committee (Professors Harry Allen, Larry Anderson, Hu McCulloch, and Alan Randall and Dean Alan Goodridge, with myself as chair) to study the use of the SEI in the promotion and tenure process. Our recommendations pertaining to your charge are as follows:

1. We recommend that the SEI be re-named ("Student Feedback Survey" is one option) and that all references to the SEI as assessing the quality of instruction should be deleted from materials generated by OAA and the Office of the University Registrar. The primary materials currently being distributed by OAA concerning the SEI refer to "assess[ing] teaching quality," and "an overall assessment of the quality of the course and the instructor's performance" (both from SEI Handbook) and to "performance assessment," and "judging quality" (both from Provost's letter to all faculty 9/21/99). As the instrument's author, Richard Gunther, has repeatedly pointed out, the SEI is NOT an evaluation of the quality of instruction. Furthermore, as he has also repeatedly pointed out, it is a student satisfaction survey, no more and no less. This is not to say that the committee undervalues student feedback in pedagogical development; obviously it is an important tool in honing those skills. But student satisfaction and quality of instruction, despite some overlap, are not synonymous.

2. Question number 10, the "overall score" should be deleted from the SEI questionnaire. Current practice promotes a simplistic comparison between the "overall score" and the university mean on the part of departments and colleges. The considerable potential for misuse of this question as a kind of totalizing summary indicator of teaching performance by promotion and tenure committees has already been realized in a number of cases brought before CAFR. This question leads departments and colleges to ignore the rest of the information garnered by the SEI, a fact which, taken together with the generally questionable validity and value of the "overall score" (a very noisy indicator), makes it more misleading than informative, as it encourages irresponsible usage. Furthermore, the deletion of number 10 should encourage deliberative bodies to read the totality of the SEI in a more precise and less monolithic way. Since the SEI quantifies what is essentially unquantifiable human experience (a problem that does not exist with, say, narrative evaluations), its reliability is questionable, and the use of the "overall score" compounds the problem.

3. As stated in the SEI Handbook, small differences in SEI scores are meaningless and should be treated as such. We agree with the statement there that "Student evaluations of teaching are broad-gauge measures: results should not be regarded as being highly precise....Do not attempt to make fine distinctions between scores of various instructors. Differences of a few tenths of a point should not be the basis for personnel decisions" (10) and with "Evaluation of Instruction" (10/7/99 on OAA's web site) that "The focus should be on patterns of responses and on general comparisons rather than on trivial differences in mean values." Given CAFR's experience, this policy calls for further emphasis by OAA.

4. The report issued to the instructor (and sometimes the chair) should be based only on data from comparable courses: For regular faculty each of the nine SEI scores should be compared to a single comparison group based on the following characteristics: --classes taught only by regular faculty (no other instructional staff should be included in the comparisons for regular faculty) --class-size: 5-15, 16-35, 36-60, 61+ --course level: 100-499, 500-699, and 700-900 The recommendation above is grounded in the fact that current practice does not take sufficient account of the wide variety of teaching circumstances in which university instructors find themselves. In a large university that serves so many missions, these circumstances vary greatly; a situation that calls for different kinds of evaluation or, at the very least, more subtle and differentiated modes of evaluating the data in question. As but one example, most clinical faculty who have very different teaching situations than do regular faculty find the SEI meaningless for assessing their teaching. Furthermore, the system is often compromised by instances like the one adduced in Professor McCulloch's 1/14/00 letter to Nancy Rudd: Consider two instructors, one whose average SEI score is 3.50, and a second whose average SEI score is 4.38. The first is in the 8th percentile University-wide, and is well below the 3.7 (12th percentile) considered "adequate" by former Vice Provost Bob Arnold. The second is above the university average of 4.27, and is at the 52nd percentile University-wide. Yet if the first instructor is teaching a 400-level required lecture of 500 students and grades with a 2.5 average, while the second is teaching an elective course of 5 students at the same level and grades with a 3.5 average, they are both at exactly the 25th percentile in terms of fully adjusted scores. It is hardly fair to fire the first, who is bearing the brunt of OSU's teaching mission, and who is at least approaching the 2.0 GPA theUniversity Bulletin says is the average grade in each course, while retaining and even rewarding the second, who has a soft assignment and panders to students with high grades. Neither is outstanding, but both SEI results should be treated the same. 3 The SEI Handbook acknowledges that certain factors do influence mean scores: "the level of the course (with 700-, 800-, and 900-level courses producing significantly higher scores than undergraduate level courses); the average grade given by the instructor; the reason why the student enrolled in the course (with elective courses more positively evaluated than required courses); the size of the class (with small sections producing substantially higher average scores than classes in which over 60 students are enrolled." In light of this acknowledgement and of the fact that faculty careers are at stake in this matter, the university has an obligation to ensure that these reports provide fair comparisons. It is true that "specialized reports that compare instructors' scores with groups selected according to certain characteristics that may impact ratings may be requested from the Office of the University Registrar" (SEI Handbook 4). But it is the view of the committee that more extensive comparisons such as those we have enumerated above should be not merely options but essential features of all reports if the results of these surveys are going to have any credibility.

top of page

5. The data in the report should be rendered in an accurate and nuanced way and the report should include such material as might be helpful to personnel committees:

A. Averages for each comparison group should be reported for the university as a whole, and, cell size permitting, for the course's college and course offering unit.

B. In addition to the average SEI scores for the comparison group, the report should also provide the average grade given in the course alongside the average grade for all sections in the comparison group, university-wide, within the course's college, and for the course offering unit.

C. For each question the averages of the instructor's scores and the percentile within the comparison group should be reported. The use of histograms should be eliminated as they call undue attention to a single question, number 10, thereby promoting a distorted reading of the survey.

D. At present, SEI comparison averages are calculated on the basis of section averages rather than on the averages of student responses. The latter procedure gives a somewhat lower average than the former, because small sections, which have higher average scores, have higher weights. The committee is of the view that so long as the recommendation in #4 is implemented, the difference will not be important. Thus the committee has no objection to using the forrner procedure if #4 is adopted.

E. We recommend that the Office of the University Registrar run analyses to determine the best method of incorporating "reason for taking the course". These "reasons" should be modified to three options: a) elective, b) major/minor requirement, and c) BER/GEC or major prerequisite. These analyses should compare current practice with 1) the use of proportional averages in each cell in the matrix and 2) the effect of the averages in each case without taking into account the "reason for taking the course." Such analyses will then make it possible to ascertain which if any of these methods is appropriate for evaluating individual courses.

F. The SEI Handbook should state clearly and precisely what statistical methods are being used to compute the variances within comparison groups.

G. For ease of administration, FYI's should be incorporated onto the same questionnaire form as the SEI. The SEI report form should, however, be separate from the FYI report so as to stress their different functions. Measures such as these can, in the view of the committee, substantially improve both the accuracy and the usefulness of the SEI. Given the wide-spread conviction among the university's faculty that the SEI is often inappropriately used to assess the quality of instruction rather than student opinion, such measures seem long overdue. Since the SEI plays a prominent role in the careers of the faculty, the university should, we think, insist on responsible and meaningful reporting of the data.

6. Issues centering on the integrity of the administration process for the SEI were also discussed. Some members of the committee suggest that OAA develop a procedure according to which instructors, Tas., and students be excluded from the administration process so as to preclude the possibility of tampering with the form.

7. We recommend that OAA persist in its efforts to assure that meaningful and informative annual reviews take place for all faculty. In accordance with Faculty Rule 3335-47-03, section (C) (2), annual reviews "should encompass the faculty member's performance in teaching, in scholarship, and in service; as well as evidence of continuing development. The involvement of the tenure initiating faculty in annual reviews is strongly encouraged." Such reviews can be invaluable in helping faculty to improve their pedagogical skills, but, unfortunately, they are still the exception rather than the rule. We also note that Faculty Rule 3335-3-35, section (C) (14) assigns to the department chair the responsibility for "promot[ing] the improvement of instruction by providing for the evaluation of each course when offered, including written evaluation by students of the courses and instructors, and periodic course review by the faculty." We would like to suggest that OAA remind chairs of their responsibilities in this regard. The active discharging of this responsibility by all of the university's chairs would, we think, go a long way towards improving the quality of the university's instruction.

8. We further suggest that, after it has forwarded its recommendations to central administration, Steering ask Faculty Council to continue monitoring this issue.

The committee hopes that you will find this report helpful.

Respectfully submitted,

Marilyn Johns Blackwell,
Chair Senate Steering Committee's ad hoc Committee on the SEI

top of page