Understanding the impact of OFFA’s approach to evidence and effective practice
This project concluded in March 2018 and we have published the impact evaluation of the work of the evidence and effective practice function at OFFA. It demonstrates that:
- our approach to evidence and effective practice has influenced behaviour in universities and colleges and driven change in access and participation
- OFFA’s ‘support and challenge’ approach has enabled us to work effectively with colleagues at every level across the sector and secure much-needed improvements in access and participation work
- the research we have produced and commissioned has been highly valued by the sector, viewed as both credible and robust.
We have also published a summary of this research for those working in higher education policy, and one for staff in higher education institutions or third sector organisations working on access and participation.
Why did we do this?
Our 2015-2020 Strategic Plan called for universities and colleges to take an increasingly evidence-led approach to improving performance on access and participation across the whole student lifecycle, right through to the transition to work or further study.
Alongside this challenge to institutions, we have also been developing a more evidence-led approach to our own policy and practice. This impact review forms part of our commitment to more intelligent regulation, and was intended to:
- demonstrate the key role of evaluation and evidence in effective policy decisions
- provide evidence to inform discussions about the shape of evidence policies in the new Office for Students (OfS), which has responsibility for access regulation from April 2018.
What did we do?
We issued an invitation to tender in early September 2017, and at the end of the same month, commissioned Nursaw Associates to carry out an impact review of how far the activities of our Evidence and Effective Practice function have successfully met the objectives of our Evidence Strategy.
Since then Nursaw Associates has:
- analysed 30 institutions’ access agreements (from 2013-14 to 2018-19), to understand changes in behaviour following key publications and announcements from the Evidence and Effective Practice team
- surveyed institutions and external partners (where appropriate) to understand their perceptions, with 130 respondents
- interviewed a sample of 68, including practitioners and management at 10 institutions, three third sector organisations, researchers and student union officials, to understand their attitudes, how the Evidence and Effective Practice function may or may not influence them, and any attitudinal and behavioural change
- produced a final detailed report (the impact evaluation).
What did we find?
Support and challenge
In the minds of people working at institutions, there is significant interdependence between OFFA’s evidence and effective practice work and its regulatory function, with 81 per cent of survey respondents considering that OFFA’s role should be to both regulate higher education and support the sector in its efforts to widen participation.
The value of research to institutions
Survey and interview responses highlighted that OFFA’s research and evaluation work is valued by institutions in terms of:
- benchmarking against other institutions
- highlighting where they can improve their approach
- enabling change in the institution
- identifying best practice.
Widening participation staff felt that OFFA enabled them to influence leaders to understand the importance of their work, and would like OFFA to be more visible in the practitioner community, for example via targeted events and direct communications. For example, over 57 per cent of institutions from the access agreement analysis use the financial support toolkit developed through our research to understand the impact of financial support.
Eighty-eight per cent of survey respondents described having a close working relationship with OFFA, which was seen as positive and important for effective regulation. Interviewees from further education colleges (FECs) reported feeling more distant from OFFA, explaining that they do not consider a close relationship with OFFA to be necessary, but would like opportunities to share and discuss best practice with other FECs.
A sea change
Interviewees reported that OFFA has had a positive impact on the sector, with one describing OFFA creating a ‘sea change’. This is borne out by access agreement analysis which demonstrates a significant increase in evaluative activity within institutions: only 10 per cent of access agreements in 2013-14 mentioned an evaluation framework, but this had risen to 57 per cent in 2018-19.
However, interviewees widely acknowledged that evidence and evaluation are not yet embedded into the work of the sector and require further support and intervention from the regulator. Many did not believe that the evaluative work would continue at the same level without the commitment and support from OFFA.
Consultation and collaboration
Individuals and institutions described being part of working groups or consultative forums with OFFA as ‘a very positive experience’. Interviewees wished to be consulted more in terms of the subjects for future topic briefings or areas of research.
This review has provided valuable learning for OFFA as the current fair access regulator, which we hope will be developed by the Office for Students (OfS) as the incoming regulator, so that:
- an evidence and effective practice function remains an integral part of the OfS
- all engagement with institutions supports and encourages evidence and effective practice
- the OfS communicates and engages with students through student unions or the NUS
- the OfS uses its understanding of the sector alongside its work with government to identify areas for research and evaluative studies, and continue to be commission such work.
In order to work with higher education providers in the most effective ways, we also encourage the OfS to:
- consider all its communications with the sector to ensure that it is both challenging and supportive
- engage with groups of institutions to work together and share best practice, research and evidence
- ensure that research and evaluation tools are accessible for different types of institution and different staff within an institution (for example, those who do not have a research background)
- ensure it widely publicises the conferences and events it contributes to and organises.
As a result of this research, we also recommend that higher education providers:
- celebrate the work they do and work together to keep sharing excellent practice
- continue to engage with widening participation policy makers
- continue to invest in their widening participation staff
- be part of future research
- be their own champions for the importance of widening participation in their own institutions and across the sector.
Want more information?
Contact Richard Shiner, Head of Evidence and Effective Practice (0117 931 7467, firstname.lastname@example.org).