Implementation Assessment and Outcome Evaluation Planning for the Restorative Action Program (RAP)
RAP's program monitoring process was developed in 2012 based on extensive consultation with the RAP workers and program administrators. The program monitoring system allows RAP workers to report details of their day-to-day activities and services in a systematic and standardized manner. This provides a baseline of RAP's routine service delivery and allows overall trends in how the program operates to be tracked.
By Carolyn Camman & J. Stephen WormithThis report is the third in an on-going, multi-phased evaluation of the Restorative Action Program (RAP). It includes two components:
- An implementation assessment examining the functionality and sustainability of the new program monitoring system as well as assessing the consistency of RAP's implementation based on these performance indicators.
- An assessment of RAP's outcome evaluability examining the options for evaluating RAP's effectiveness as a program and providing recommendations on how to proceed.
The focus of the current report was to:
- An implementation assessment was done to examine the functionality and sustainability of the new program monitoring system as well as assess the consistency of RAP's implementation based on these performance indicators. The results of this year's data collection have been analyzed and summarized and recommendations have been made for how to further refine and make use of this system.
- Concurrently, a specific assessment of RAP's outcome evaluability was undertaken to examine the options for evaluating RAP's effectiveness as a program. Based on consultation with program administrators and school division stakeholders as well as a review of the available program data and status, recommendations were made for how to move forward in this process.
Conclusion
At the present time, a specific and detailed evaluation plan for RAP cannot be generated. This is because there are considerable unaddressed questions with regard to access to data, particularly the archival data required for the school-level quantitative analysis and to some extent access for surveying and interviewing current and past students for the qualitative success case analysis.
However, based on the current status of RAP, the evaluation steps taken thus far, and the expectations around potentially available data, the above-described methodology represents the recommended approach to the initial outcome evaluation. In addition to the methodological recommendations above (i.e., that the evaluation be theory-driven; cost-effective, timely and sufficiently rigorous; and methodologically-mixed; and specifically that it incorporate two particular methods which meet these criteria, the interrupted time-series and the success case analysis), the following are the recommended next steps for moving forward with RAP's outcome evaluation:
- Make outcome evaluation an explicit priority
- Plan an extensive consultation period with school partners
- Prepare for the cost of the evaluation