Background and Purpose

Effective quality assurance in the public vocational rehabilitation (VR) program is important for continually assessing and informing opportunities for improvement (Leahy et al., 2009; Millington & Schultz, 2009). Prior to 2014, the Rehabilitation Services Administration (RSA) required every state VR program to have a formal case review process in place. However, the Workforce Innovation and Opportunity Act (WIOA) of 2014 further elevated the importance and scope of quality assurance, and expanded priorities beyond compliance. And given that quality assurance (QA) is intended to be a continuous and dynamic process, participatory methods are preferable when developing or revising instruments used during evaluation and QA processes (Baron-Epel et al., 2004; Glasgow et al., 2019).

With the inception of WIOA and common performance measures (CPM), the Vermont Division of Vocational Rehabilitation (DVR) took the opportunity to examine more closely how customers were served, and explore options for re-envisioning and shifting current practices to better support high-wage careers. This initiative, known as the “Careers Initiative”, identified five leading indicators believed to positively impact the customer experience, as well as improve program outcomes. The five indicators required a shift in VR counselor practice and, to support this practice change, the internal quality assurance team needed to revise the case review tool and develop an accompanying process guide. Specifically, the case review process needed to shift to support counseling practices that would help VR customers obtain and maintain high-wage employment, and achieve postsecondary credentials.

Vermont DVR is committed to continuously improving a standardized and consistent statewide case record review process for use by regional managers and senior counselors. The case review process provides guidance to both counselors and supervisors regarding best practices for case work. The original case review process was implemented in August 2018, with the current revision and implementation process taking place in 2020. The commitment to continuous quality improvement helps DVR address systemic compliance challenges identified through audit findings, as well as provide guidance to counselors around best practices within the VR program’s case management system. The intent is to provide VR staff with helpful data and guidance regarding legal and regulatory compliance, and provide a clear framework for the implementation of Careers Initiative activities.

The purpose of this project was to revise the extant Vermont DVR case review tool to reflect the shift in counseling practices and common performance measures, and positively impact the customer experience. The project aimed to (a) convene a workgroup responsible for developing case review questions to assess both compliance and quality, (b) revise the extant case review tool and accompanying process guide, and (c) achieve at least 75% interrater reliability with the revised tool.

Development of New Case Review Tool

Focus Group Procedures

Focus groups consisting of 48 field staff and manager/supervisor participants were convened to solicit feedback regarding questions and elements of the case process that were important to consider when revising the tool. The larger group was divided into six subgroups of eight participants. Each participant was given one of three cases to review; and each member of the subgroup reviewed the same case. A small, representative workgroup then reviewed the feedback and used it to revise the case review tool and develop an accompanying process guide. Participants then compared results and discussed any differences in ratings. The workgroup intentionally framed questions in a way that authentically, yet systematically, addresses the process VR counselors use with consumers, while providing flexibility to accommodate counselor preferences. The tool was then tested with a diverse group of staff at an expanded management meeting to determine interrater reliability.

Organization of the Case Review Tool

The case review tool is organized into two parts based on case type, with two sections in each, covering both compliance and qualitative elements. Slight variations are suggested for Transition and Supported Employment (SE) cases, as these cases also receive a separate review by program managers with additional questions specific to SE and Transition populations.

Regulatory Compliance

Part 1, Section A of the revised tool is designed to review cases for basic compliance with RSA regulations across various aspects of the VR process including (a) application, (b) eligibility, (c) IPE development, and (d) case closure. Within each area, reviewers assign a rating of yes, no, or not applicable for milestone items (basic compliance for auditing), and data quality items (inaccurate or incomplete data may affect reporting and/or common performance measures).

Case Quality

Section B of Part 1 of the revised tool highlights the new additions and examines the qualitative aspects of the casework. Guidance is provided to reviewers in using the rating scale of unsatisfactory, satisfactory, and excellent, as defined for each of the eight key quality indicators. See Table 1.

Table 1:Guidance for Qualitative VR Case Review
Questions used to guide qualitative review and rating
  1. Does the case record contain documentation of an initial interview that gives a holistic picture of the consumer’s barriers, strengths, and interests?
  2. Does the case show consistent contact/engagement efforts by VR and/or VABIR team?
  3. Does the case show evidence that the counselor helped the consumer explore and/or engage in activities (assessments, progressive education, progressive employment, etc.) that informed the identification of a career goal?
  4. Do the case notes tell a clear story of both the efforts to support the consumer’s identified career goal and possible barriers to that goal?
  5. Is there clear evidence in the case that the counselor supported the consumer in making an informed choice about their career goal, beyond entry-level employment?
  6. Is the work of internal and external partners (teaming) well documented in the case narrative?
  7. Is the information gained through progressive employment and progressive education activities documented in case notes by VR and/or VABIR staff?
  8. Are consumers who are employed receiving regular check-ins consistent with the Division’s Follow-Up guidance?

Validation of Revised Case Review Tool

The revised tool for reviewing cases comprises 22 questions focused on regulatory compliance, and an additional eight qualitative questions used to rate quality regarding case development and management. An interrater reliability score of 68% was calculated based on a comparison of reviewer ratings across six cases.

Recommendations and Implications for Practice

Development of a revised case review tool and accompanying process guide encouraged more frequent, thoughtful conversation about counseling practices within the state VR program. In particular, the process guide provided a common training tool all staff can reference to enhance understanding of case compliance and quality expectations. New staff now have a reference guide that supports the Division’s Careers Initiative and ensures a better understanding of how the common performance measures can be positively impacted through case work. Based on this study, the following recommendations and reflections are offered:

  • Incorporating feedback from all levels of staff within the VR agency greatly increased buy-in of the new review tool and allowed for questions to be developed in a manner that directly addressed counselors’ day-to-day practice.

  • Piloting the tool with a diverse group of staff allowed for the most generalizable interrater reliability results.

  • More consistent case practices statewide were achieved and attributable to the accompanying process guide, serving as a consistent and universal training tool for staff.

  • Developing a revised tool and accompanying process guide has encouraged thoughtful and more regular discussion about casework best practices within the state VR program.

  • Qualitative questions are subjective, which can make it difficult to achieve a high level of interrater reliability.

Author Note

The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.