Program Evaluation and Quality Assurance in Vocational Rehabilitation: An Introduction to the Special Issue

In the era of accountability, all human service agencies, including those in the public sector, emphasize program evaluation and internal quality controls to monitor and demonstrate the effectiveness of services they provide. Given that agencies serve differing target populations and provide a variety of services, it is critical to set up appropriate method and standards and validate them continuously. For the state-federal Vocational Rehabilitation (VR) programs, the 1973 Rehabilitation Act and amendments established unique evaluation standards and performance accountability measures (e.g., employment status, median wages, long-term follow up on employment status), aligned with their target populations and outcomes.

To comply with the mandated requirements, each state unit is charged with hiring or training personnel who understand the policies, and report the information correctly and in a timely manner. In addition to pre-service training related to knowledge on research methodology and evidence-based practice (Leahy et al., 2018), Sherman and colleagues (2018) reported that having highly trained staff and resources allocated to training their staff to implement innovative and promising practices was a key organizational and cultural factor that promoted best practices in the public VR program. In reality, however, agencies often select internal candidates who have extensive VR experience, but with no formal training or expertise in program evaluation; thus, they may face significant challenges in fulfilling their new roles in program evaluation and internal quality control (Sabella et al., 2018). This need for capacity building at the state level was one of the key recommendations from an earlier study (Leahy et al., 2014) that paved the way for the current program evaluation and quality assurance (PEQA) training effort.

In 2015, the Rehabilitation Services Administration (RSA), Office of Special Education and Rehabilitative Services, U.S. Department of Education funded the Vocational Rehabilitation Technical Assistance Center on Program Evaluation and Quality Assurance (PEQA-TAC) grant, a collaboration of partners with extensive experience in program evaluation, including the University of Wisconsin-Stout Vocational Rehabilitation Institute, Michigan State University, and the University of Wisconsin-Madison. The primary purpose of the Center was to assist state VR agencies to improve performance management by building their capacity to carry out high-quality program evaluations and quality assurance practices that promote continuous program improvement.

Over the five-year grant, a total of 37 of the 78 state VR agencies participated in the PEQA certificate program. The program consists of online coursework and the participant’s capstone project. The knowledge domains of the online courses included the principles of evaluation, research design and methods, data analysis, and report preparation for multiple stakeholder audience. In addition, each participant was to design and complete their own capstone project, as a culminating learning experience to synthesize knowledge and skills acquired through the online courses within their state agency. A total of 26 capstone projects were successfully completed.

Prior to presenting the actual capstone projects completed by the PEQA participants, the article, “The Importance of Building Capacity for Program Evaluation and Quality Assurance Within Vocational Rehabilitation” by Anderson et al., provides an overview of the PEQA-TAC, including background needs regarding staff development and training program within the state VR agency, a conceptual model, and the curriculum of the certificate program.

This special issue features capstone projects completed by PEQA participants, including those conducted by individuals and teams of two to three staff members within states. After meeting with the PEQA team at Michigan State University, each participant/team was assigned to a faculty mentor who provided guidance and mentorship for their study design, data collection, data analysis, and report writing. Considering the differences in educational background and experience as a program evaluator or quality management specialist, the capstone projects varied in terms of topical areas, methods, and results. To help readers better understand this special issue, the projects are organized into three sections: (1) evaluation of an existing program, (2) VR outcome studies, and (3) other (e.g., development and validation of a case review instrument, new program development). The following capstone projects evaluated the effectiveness of an existing program:

• Jacob Chorey (South Carolina VR Department): “Evaluation of the South Carolina Vocational Rehabilitation Department (SCVRD) Information Technology Training Center”

• Chris Clause (Missouri VR): “The Impact of Early Work Experience on VR Outcomes”

• Nicole Fraedrich (Utah State Office of Rehabilitation): “An Outcome Analysis of Community Rehabilitation Programs”

• Kim Gee (Missouri VR): “Evaluation of Employment Services Plus”

• David Higginbotham (Louisiana Rehabilitation Services): “The Effect of Introductory Training on the Use of Motivational Interviewing in Vocational Rehabilitation”

• Eve Lo (Minnesota DEED-VR): “A Pilot Study Evaluating the Effectiveness of Person-Centered Planning”

• Claudia Pettit (Michigan Rehabilitation Services): “The Effect of Benefits Counseling on Increasing Knowledge of Social Security Work Rules and Work Incentives”

• Graham Porell, Lola Akinlapa, & Bill Noone (Massachusetts Rehabilitation Commission): “The Effectiveness of Integrated Resources for People With Mental Health and Employment”

• Regina Rice (Florida DVR): “Pilot Evaluation of the VR Works Training and Implementation”

• Jordan Trumbo (South Dakota DVS): “Customized Employment Training Needs in South Dakota”

The projects below reflected studies to identify any factors (e.g., individual characteristics, services) relevant to VR outcomes for general or special target populations:

• Margaret Alewine (South Carolina VR): “Comprehensive Needs Assessment for Pre-Employment Transition Services”

• Crystal Anderson (Arkansas Division of Services for the Blind): “Examining Correlations between Pre-Employment Transition Services and Vocational Rehabilitation Progression”

• Deborah Collard (Virginia Dept. for the Blind): “A Closer Look at Vocational Rehabilitation College Training Services at the Virginia Department for the Blind and Vision Impaired”

• Carrie Marsh, Karla Eckhoff, & John Yang (Minnesota DEED-VR): “Repeat Customers: Minnesota Participants who Return for Multiple Series of Case Services”

• Andrew Grey (Idaho DVR): “Rural and Urban Counties in Idaho: Differences in Vocational Rehabilitation Service Delivery and Outcomes”

• Kira Hollywood (Texas Workforce Commission – VRD): “A Critical Examination of Cases Closed Unsuccessfully After Application for VR Services”

• Mary Otiato (Massachusetts Commission for the Blind): “Exploring the Relationship Between Cost of Purchased Services and Employment Outcomes”

• Kim Pieczynski (Illinois Dept. of Health Services): “Pre-ETS Success, Pre-ETS Standardized or Teacher-Created Curriculum: Does the Structure Determine Transition Success?”

• Danielle Russell (Wisconsin DVR): “Identifying Best Practices for Long-Term Success in Supported Employment”

• Kellie Scott (Kentucky VR): “Evaluating the Long-Term Effectiveness of the SGA Enhanced Services Model in Kentucky: A Follow-Up Study on the SGA Demonstration Project”

The remaining capstone projects were classified as the “other” category. Several projects developed or modified an internal case review scoring form for use in quality assurance and attempted to validate the instruments using a qualitative approach:

• Carol Anderson, Felicia Johnson, Jerry Neel, & Phyllis Hoover (South Carolina VR): “Improving Quality Assurance at SCVRD: Developing a New, Electronic QA Tool”

• Amanda Arnold & Alice Porter (Vermont VR): “Vermont Case Review Process Guide: Assessing and Delivering High-Quality Services to VR Customers”

• Brittney Downing (Indiana Bureau of Rehabilitation Services): “Case Service Reporting: Meeting Guidelines for WIOA and Supporting Documentation”

• Elaine Robertson (South Carolina Commission for the Blind): “A Pilot Study to Develop VR Case Review Instrument for WIOA Performance Measure Data Collection”

Finally, two participants initiated the development of new programs: one focused on designing an internal data dashboard system and the other a measurable training program. Due to complexities caused by the Covid-19 pandemic, implementation and evaluation of these projects were implausible. However, the information on these projects is still important, and the knowledge and lessons learned is shared to help inform future efforts:

• Kevin Harris (New Jersey Commission for the Blind and Visually Impaired): “Developing a Dashboard: Preliminary Steps and What We Learned”

• Andrzej Walz-Chojnacki (Wisconsin DVR): “Plan Twice, Measure Once: Designing a Financial Capability Service with Evaluation in Mind”

Collaboration among the PEQA-TAC was key, with partners involved in designing, implementing, and monitoring all phases of the project. In addition to the university partners mentioned earlier, the Council of State Administrators of Vocational Rehabilitation (CSAVR) and members of the Summit Group (https://vocational-rehab.com/) were instrumental in ensuring the program was designed and implemented in a manner that pragmatically addressed the knowledge and training essentials of VR personnel.

For the capstone projects, special thanks are extended to the 12 faculty mentors—Cayte Anderson, Malachy Bishop, Jina Chun, Todd Honeycutt, Bill Hoyt, Hung-Jen Kuo, Michael Leahy, Andrew Nay, Sukyeong Pi, Scott Sabella, Purvi Sevak, and Timothy Tansey—for their enthusiastic support with topic selection, study design, data collection, instrument development, data analysis, report writing, and preparation for the PEQA presentation. Also, thanks to the online course instructors at the University of Wisconsin-Stout (i.e., Justin Sullivan and Libby Smith) and coaches (i.e., Terry Donovan, Darlene Groomes, Debra Homa, Christine Johnson, Erin Nierenhausen) for their individualized support to participants as they completed the online courses.

In addition, we would like to thank the PEQA participants for their diligence and passionate participation. Although most were able to complete the certificate program with the support of their supervisors and state directors, taking online coursework and conducting the capstone project created an additional workload. Accordingly, thanks to the state directors who highly recommended their staff to participate in the PEQA Certificate Program and provided support for them throughout the program.

Finally, we would like to thank Dr. Douglas Zhu, our RSA officer, for his ongoing insight and support throughout the design and implementation of the PEQA-TAC program. His expert advice regarding contract management and understanding of the evaluative and quality assurance needs within state agencies was instrumental in ensuring sound guidance throughout the course of the project.

Author Note

The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.