ThinkForward Evaluation: cost benefit analysis

The cost benefit analysis explored the financial implications of outcomes achieved by young people supported by ThinkForward. Data from ThinkForward London schools was included and we used the New Economy Manchester  model for financial implications of outcomes achieved.The study took a very conservative approach to assessing potential savings, for example only treating young people who had previously had two or more exclusions from school as being at risk of permanent exclusion.

The evaluation concluded that every £1 spent resulted in cost savings of £2.48. The biggest future savings are through the following average per participant savings over the course of their working lives:

  • £768 to schools/local authorities due to young people being less likely to be permanently excluded
  • £544 to Department for Work and Pensions through reduction in Jobseekers Allowance
  • £410 to criminal justice system through reduction in crime-related costs due to lower likelihood of young people being involved with criminal activity

This evaluation was undertaken by an Executive Masters of Public Administration candidate in Global Public Policy and Management offered jointly by UCL in London and NYU in New York as their Capstone project. This was supervised by an expert in Cost Benefit Analysis at NYU.

 

ThinkForward Evaluation: outcomes study

We have completed two studies into the outcomes that ThinkForward achieve.

Education Endowment Foundation Randomised Control Trial:

ThinkForward took part in the randomised control trial (RCT) to help the EEF test the effectiveness of two different RCT methodologies which would then inform the design of a larger randomised control trial. The RCT was conducted over a two-year period across two academic years. Randomisation took place at both school and pupil level. Regarding the methodology, the trial found that the most effective randomisation should be at a school level. On the educational attainment of young people, the study did not find evidence of improvement in GCSE attainment. However, teachers, Coaches and young people from across both schools reported that they believed the programme was beneficial and the study notes improvements in young people’s behaviour.

For more information on the RCT please click here.

London School of Economics Outcomes Study:

As a result we ran a second outcomes study exploration the correlations between behaviour, attendance, attainment, Ready for Work capabilities and baseline factors such as gender and ethnicity.

We ran this across three Thinkforward schools in which we have implemented an updated programme design in which our Progression Coaches work with the most at-risk ten young people per year group and begin doing so from the beginning of year 9. This change was based on our initial analysis of the first two years of delivery of ThinkForward and a “Theory of Change” process to improve the way the programme was designed to provide even better outcomes for young people through lessons learned.

The key findings of this study were:

  • statistically significant increase in attendance in year 9 of +8.5% (79.3% to 87.8%)
  • statistically significant improvement in behaviour points in year 11
  • statistically significant worsening of behaviour at one school in year 9 (we are currently exploring the reason behind this with the Progression Coach in the school, including whether there were particular factors which contributed to the observed changes or the school’s approach to behaviour shifted during the period)
  • increases in all Ready for Work capabilities in Y9

This evaluation was undertaken by a postgraduate student studying Management Science at London School of Economics, supervised by an experienced statistician and with input from Project Oracle. We are currently working with Project Oracle to get this evaluation validated against their ‘standards of evidence’.

EEF FAQ

ThinkForward took part in the randomised control trial (RCT) to help the EEF test the effectiveness of two different RCT methodologies which would then inform the design of a larger randomised control trial. The RCT was conducted over a two-year period across two academic years.

Randomisation took place at both school and pupil level. Regarding the methodology, the trial found that the most effective randomisation should be at a school level. On the educational attainment of young people, the study did not find evidence of improvement in GCSE attainment. However, teachers, Coaches and young people from across both schools reported that they believed the programme was beneficial and the study notes improvements in young people’s behaviour.

1. Why did ThinkForward participate in the Education Endowment Foundation’s (EEF) randomised control trial (RCT)?

ThinkForward took part in the RCT to help the EEF test the effectiveness of two different RCT methodologies which would then inform the design of a larger randomised control trial. ThinkForward has always been focused on ensuring their programme design has a long term and sustained impact on young people’s transitions from school to work. At the end of our first two years of delivery, the EEF’s trial provided us with a unique opportunity to take part in a cutting edge evaluation, to learn from our initial programme design and to make improvements ahead of further evaluation. Shortly after the start of the RCT in our third year we went through a rigorous Theory of Change process which led to refining our mission to focus on young people most at risk of unemployment and reducing each Coach’s caseload by half (from 100 to 50).

2. How was the RCT designed?

The RCT was conducted over a two year period across two academic years. Randomisation took place at both school and pupil level. Within the two randomly assigned schools, eligible pupils were randomly allocated either to a ThinkForward group or a within-school control group. Across the four schools, there were 181 pupils in the year 11 group, 40 of whom received ThinkForward support, and 160 pupils in the year 10 group, 37 of whom received ThinkForward. The study focused on the first two years of ThinkForward’s five year programme and young people’s educational outcomes.

3. What were the key findings?

Regarding the methodology, the trial found that the most effective randomisation should be at a school level. Due to the ‘spill-over effects’ on other pupils in the school it does not work to have the control group from the same school.

On the educational attainment of young people, the study did not find evidence of improvement in GCSE attainment. Year 11 pupils received 6 months of support from their Coach before their GCSE examinations, in preparing for their GCSEs young people’s primary focus is on their revision and exam preparation and it is often harder for Coaches to work one-to-one. According to ThinkForward’s refreshed programme design, participants would have received 2.5 years of support before taking their GCSEs.

Teachers, Coaches and young people from across both schools reported that they believed the programme was beneficial and the study notes improvements in young people’s behaviour.

More work needs to be done on the effective collection of data when conducting a trial of this kind and at greater scale. ThinkForward welcomes the recommendation that in a future trial, data collection is undertaken by a third party rather than Progression Coaches.

4. What has ThinkForward learned from the findings?

ThinkForward’s approach is to be open to the lessons we can learn from different types of evaluations and their findings. We are focused on making changes in two key areas covering programme design and delivery and future evaluation.

In terms of our programme design, following our Theory of Change process two years ago, we now enroll young people at the start of year 9 so that there is more time (2.5 years) to work with them ahead of their GSCEs, increasing the likelihood of impact on attainment at 16. We have also halved the caseloads of coaches, to increase the intensity of the support that young people receive. To strengthen our delivery we have captured our refined programme design in practical materials to support delivery, including our Coach Handbook.

We are also focused on improving our partnerships with schools, all schools have Service Level Agreements in place to ensure effective data sharing and access to dedicated and confidential space for Coaches to work with young people.

In future we would like to conduct evaluations using a much larger sample size. This was a small study, only 77 young people received the programme vs. 264 in the control group which meant that the required level of impact to show significance was very high.

5. What happened to the young people who participated in the trial?

Half the young people who participated in the trial graduated from ThinkForward in August 2016 with impressive results. The other half will graduate from ThinkForward in the summer of 2017. Across the two groups 92.4% are in positive destinations including education, employment or training. 6 young people progressed to university, 17 into employment and the remainder are continuing their studies.

Five young people (7.6%) who took part in the trial are currently not in education, employment or training (NEET). This is below the NEET rate for the geographical area in which the trial schools are located. This is an impressive outcome given that the young people were selected as being at high risk of unemployment.

ThinkForward continues to give targeted support either through targeted support by WorkForward, ThinkForward’s employment brokerage and one-to-one support from their Progression Coach.