Background and Purpose

In 2015, the Rehabilitation Services Administration (RSA) began implementing the Workforce Innovation and Opportunity Act (WIOA), which was signed into law in 2014. This hallmark legislation changed the way public vocational rehabilitation (VR) services were defined and provided across state agencies. WIOA also introduced new performance measures used to verify compliance and monitor outcomes, adding an evaluative focus on quality and requiring substantial changes in data collection, interpretation, and reporting.

Quality assurance (QA) and program evaluation have been important elements for VR programs to use in supporting ongoing continuous improvement efforts for years (Leahy et al., 2009). Similarly, it is important to consider the influence that factors such as organizational culture play in successfully implementing QA strategies (Millington & Schultz, 2009). Challenges related to navigating ongoing agency compliance needs, organizational culture shifts, and the quest for continuous quality improvement through program implementation are not unique to VR, but are increasingly relevant, as awareness to adopt evidence-based strategies in our programs develops (Durlak, 2015; Lockett et al., 2018; Nilsen, 2015). A growing body of evidence suggests that development of a framework and consistent use of related tools are key in supporting the advancement of QA efforts within disciplines, including VR (Green et al., 2019; Hurlocker et al., 2020; Lockett et al., 2018).

In light of the recent legislative changes and need to integrate quality assurance tools to meet contemporary needs, the South Carolina Commission for the Blind (SCCB) aimed to develop a new instrument to support this process. This project focused on developing an instrument to incorporate performance measures into case documentation, thus, providing a consistent method for interpreting quality and gauging compliance with WIOA performance indicators, as well as state and internal agency policy and regulations. In developing and validating the new case review instrument, the following aims were pursued:

  1. Identify the WIOA performance indicators pertaining to case documentation, and

  2. Determine whether the new case review instrument is reliable and valid.

Case Review Tool Development and Validation

A defined series of steps were used as a framework to guide the development and evaluation methods and procedures, which included (1) a review of existing materials, (2) the execution of a needs assessment, (3) review of individual cases, (4) development of the instrument, and (5) a validation and application study.

Methods and Procedures

Step 1: Material Review

First, existing materials, including the RSA Monitoring and Technical Assistance Guide, existing case review instruments used by other agencies, and monitoring visit reports for other states, were reviewed.

Step 2: Needs Assessment

Next, a needs assessment was conducted using a survey to measure 16 VR counselors’ level of understanding about the WIOA Performance Accountability Measures. Approximately two-thirds of the 16 respondents provided needs for more adequate and consistent training on the documentation process. A focus group with supervisors and the VR Director was also held to elicit their perceptions on quality documentation. As an agency serving individuals with blindness and visual impairments, the participants stressed an importance of the disability-specific documentation (e.g., updated vision report, assessments). They also provided issues/items that determine quality documentation, such as the comprehensive intake summary, case notes that tell the story of consumers from the intake appointment to closure, eligibility documentation with clear functional limitations and barriers, comprehensive assessment of rehabilitation needs (CARN), job readiness documentation, employment documentation aligned to IPE goal, and closure documentation with all service information and employment-related information.

Step 3: Individual Case Review

Ten individual consumer cases were then reviewed to determine the availability of data needed for RSA reporting and monitoring, which indicated needs for a great deal of improvement to satisfactory levels.

Step 4: Instrument Development

Information gathered during Step 1 through Step 3 then informed the development of the SCCB Case Review Instrument, which featured 25 questions across four categories, each containing multiple questions. The four main sections and sub-categories are as follows:

Section I: Quality of Counseling Services & Documentation

  1. Intake information, including documentation of benefits and referral to benefits counseling

  2. Evidence of informed choice and comprehensive assessment of rehabilitation needs (CARN)

Section II: Quality of Vocational Preparation, Goal, Training, & Placement

  1. Vocational exploration, labor market information, and measurable skill gains

  2. Referred to Employment Consultant and job placement information, including wages

Section III: Quality of Closures

  1. Federal Compliance when closed successfully

  2. Sufficient documentation and review when closed without employment

Section IV: Compliance with Regulations, Laws & Policies

  1. All timelines met for eligibility and IPE; all signatures in place and dated correctly

  2. Comparable benefits documented; expenditures follow all policies

  3. All goods and services included on IPE

  4. If a transition or supported employment case, all documentation follows policy

Step 5: Validation Study

To test the newly developed case review instrument, two validation studies were conducted by agency staff. Each reviewer was asked to rate the instrument using a Likert scale composed of four rating options: minimally/not compliant (1), mostly compliant (2), compliant (3), and not applicable. The rating scale was developed to measure compliance within each category, as depicted in Table 1.

Table 1.SCCB Case Review Instrument Compliance Rating Scale
Rating Description
Minimally or not in compliance (1) Multiple important aspects were omitted and/or the omission(s) have a major impact
Mostly compliant (2) The majority of important aspects were completed, and omissions have no more than a mild to moderate impact
Compliant with no major omissions (3) If any omission, they were minor and have no or only a mild impact
Not Applicable

At first, a total of 77 cases, representing all active caseloads and a diverse variety of current case statuses, were reviewed by a counselor trainer and the current evaluator. For this review, all cases were solely on the electronic system, not the hardcopies of case files. The next case review was completed by the three regional VR supervisors, the VR Consumer Services Director, and this evaluator. A total of 20 cases were pulled at random and each was reviewed utilizing the instrument. At this time, each case was reviewed within the electronic case management system and the physical hard copy to capture all documentation. For further review and to limit interpretation bias, each case was independently reviewed by two members of the team, which showed 100% agreement in the rating scale.

In addition to investigating how similarly each reviewer rated the cases, this study also used the analysis to identify areas of case documentation to be improved. Merging data from the two validation studies, case review results for 97 cases were analyzed and the compliance rate (mostly compliant or compliant) was computed. The compliance rates were: (a) medical/eye report in file (100%), (b) Individual Plan for Employment (IPE) developed in 90 days (89.7%), (c) eligibility determined in 60 days (74.2%), (d) qualified CARN was in the file (55.7%), (e) case notes that “tell the story” of the consumer (46.4%), (f) impediments to employment documented (16.5%), (g) informed choice noted (14.4%), (h) assessment completed (13.4%), and (i) functional limitations listed (11.3%). See Figure 1.

Figure 1
Figure 1.Case Review Compliance Rates

In addition, the following areas were identified as needing improvements in case documentation: comparable benefits, employment, transition services, low vision referral, benefits counseling, comprehensive intake summary, presumptive eligibility (for SSA beneficiaries), job ready form, closure, and services to employers.

Recommendations and Implications for Practice

As a result of these findings, the new case review instrument was adopted, and implementation has created a path to develop training to better inform and prepare VR staff of compliance needs. The instrument is another tool for counselors, supervisors, and QA staff to use in targeting specific data collection areas and utilizing the information for supervision and guidance. Case documentation procedures within the agency’s system were revised to reflect needs expressed by the counselors and encourage stronger case documentation. Increased use of the instrument will ultimately increase agency compliance with the mandatory federal performance indicators. Based on this study, the following recommendations and considerations are offered:

  • Implementation of a consistent case review instrument within VR is recommended to increase WIOA compliance rates and provide a better overall picture of agency performance.

  • Modification of case management software may be necessary to gather required data. Software training and a desk reference guide can increase accuracy of data entered.

  • Targeted case reviews should be completed monthly for specific measures and the review instrument itself should be examined annually to ensure all RSA changes are implemented, as needed.

  • VR agencies should consider employing dedicated Quality Assurance personnel to support case review procedures.

  • The review instrument can be used by supervisors to prepare annual performance reviews with counselors.

Author Note

The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.