Policy briefing – making sense of the Teaching Excellence Framework (TEF) results
By Professor Paul Ashwin, Centre for Global Higher Education
In English higher education, the results of the second round of a new Teaching Excellence Framework (TEF) will be announced tomorrow. In a Centre for Global Higher Education policy briefing published today (http://www.researchcghe.org/publications/making-sense-of-the-teaching-excellence-framework-tef-results/ ), Professor Paul Ashwin outlines the purpose of the TEF, how it works, and discusses the extent to which the results provide valuable information about the quality of undergraduate degree programmes.
Why was the TEF introduced?
In 2012, tuition fees for UK and EU students in England were increased to a maximum of £9,000. All English universities now charge this maximum but the government is concerned that these flat fees mask large differences in the quality of degree programmes. The government has introduced the TEF with the intended purpose of providing students with better information about the quality of degree programmes so that they can make more informed choices about where to study. The government’s intention is also to raise the profile of teaching and ensure that it is better recognised and rewarded by universities. Currently, to raise fees in line with inflation, institutions need to make a TEF submission. In the future, increases in fees may be tied to TEF outcomes.
How does the TEF work?
Institutions that opted into the TEF this year were examined on three sets of metrics: students’ views of teaching, assessment and academic support from the National Student Survey (NSS); student dropout rates; and rates of employment. It is notable that none of these metrics directly measure the quality of teaching, although the NSS does give an insight into students’ perceptions of their teaching. Instead, they focus on examining the assumed effects of teaching.
Each submitting institution’s performance on these metrics was benchmarked against the demographic characteristics of its students, and its performance was flagged when it was statistically significantly better or worse than its benchmark. Assessors made an initial assessment of an institution’s performance based on its number of positive and negative flags and then examined contextual information and a 15-page institutional submission outlining the institution’s case for the excellence of its teaching. Based on this, institutions have been awarded a Gold, Silver, or Bronze TEF award.
Does the level of TEF award provide valuable information about the quality of a university’s teaching?
The TEF will provide students with better information about the quality of degree programmes than is currently offered by commercial higher education rankings. This is because the outcomes of higher education are shaped by the demographic characteristics of students, which have nothing to do with the quality of teaching in universities. The TEF attempts to control for these differences in student intake while university rankings do not.
While the TEF metrics do not directly measure the quality of teaching, there is a logic to them. The quality of a degree can reasonably be expected to be related to student perceptions of teaching, support, and assessment, and to the proportion of students staying on their degree programmes and gaining employment or a place on a postgraduate course. While some have criticised the use of NSS results because teaching evaluations can discriminate against female and minority ethnic lecturers, this is based on a misunderstanding of the NSS. The NSS is focused on teaching across a whole degree programme and so does not differentiate between individual lecturers.
Does a Gold TEF award mean that prospective students know they are applying to an excellent degree programme?
A Gold TEF award is based on an institutional level assessment. The same university can offer programmes that differ significantly in quality, which means that the TEF award does not tell prospective students about the quality of individual degree programmes. This means that it is highly likely that there are excellent degree programmes in universities with Bronze awards and less good degree programmes in universities with Gold awards. In addition, any student who uses the TEF to inform their choice of university will not graduate until at least four years after the metrics were taken and, as the TEF award is for three years, it could be as many as eight years for students on four-year degree programmes. By this time, it is entirely possible the quality of teaching at that university will have fallen.
TEF judgements are based on assessment criteria that examine ‘teaching quality’, ‘learning environment’ and ‘student outcomes’. For example, the assessment criteria for ‘teaching quality’ focus on the extent to which an institution: encourages student engagement, values teaching, offers programmes that involve rigour and stretch, and offers effective feedback on student work. It is unclear how these criteria were selected and why others, such as teaching expertise, were excluded. This raises questions of how the criteria form a coherent whole indicating something important about the excellence of teaching. These questions undermine the claim that the TEF offers a valid measure of high quality teaching.
Will the TEF lead to improvements in the quality of teaching in universities?
If the TEF is to lead to improvements in the quality of teaching in universities, then improvements in performance on the metrics used must only be possible through improvements in the quality of teaching that students experience. The three sets of metrics used this year are reasonable although, as discussed, there are weaknesses around the focus on the institutional level, the dated evidence that informs the metrics, and the lack of a coherent view of excellent teaching that informs the TEF.
In the future, the government wants to increase the number of metrics that are used and there are strong indications that this will include a metric related to the amount of contact hours on a programme. However, there is no evidence that contact hours are a valid measure of teaching quality. Conversely, factors that are known to be necessary elements of high quality teaching, such as the expertise of those who teach, do not appear to be under consideration. If the TEF is based on measures that are unrelated to the quality of teaching, then it will end up measuring institutional game playing rather than excellent teaching. If this happens then the TEF will not lead to improvements in the quality of teaching in universities.
Paul Ashwin is professor of higher education, Department of Educational Research, Lancaster University, and a researcher at the ESRC/HEFCE-funded Centre for Global Higher Education.