Background and Purpose

Dashboards are frequently used by government agencies to drive and support decision-making, initiate action, focus on performance, inform policies, communicate with the public, foster discourse among users, and restore financial stability and eliminate backlog (Gorcester & Reinke, 2007; Lewis, 2014; Matheus et al., 2020). Within the public vocational rehabilitation (VR) program, tracking variables such as employment rates and outcomes, hours worked, wages, time in service, race, gender, and other participant characteristics can provide staff with important data to inform decision-making; provide support for staff, as well as hold them accountable; and offer a path forward for making improvements. Previous research has identified service delivery and outcome differences based on consumer race, gender, and other demographic variables within VR (Rosenthal et al., 2005; Wilson et al., 2002). In this regard, dashboards can serve as key tools to help agencies identify gaps and discrepancies and bolster an agency’s capability to plan and improve programs, and measure their impact (Groomes et al., 2012).

While data dashboards are often used to monitor and communicate program performance, little research has been published on their effectiveness in helping VR agencies improve outcomes. However, the need for data-driven decision-making and the ability to effectively communicate key performance measures and respective changes has made establishing a dashboard/feedback loop a priority for CBVI. Specifically, a dashboard/feedback loop focusing on exit outcomes provides overall performance outcomes data and consumer retention rates following application for VR services. Furthermore, real-time process and outcomes data will be available at the office and counselor levels as a management tool for supervisors and staff to evaluate and improve results.

As part of ongoing internal performance management and improvement processes, the New Jersey CBVI sought to refine their quality assurance methods. A key element involves development of a new dashboard/feedback loop regarding consumer exit outcomes. The dashboard will provide current data on performance outcomes and retention of consumers following applications for VR for services, can be used to identify trends and outliers, and serve as a strategic and analytic tool to monitor performance indicators. The dashboard aims to provide data detail by counselor, supervisor, and field office to identify trends and provide relevant feedback to counselors and supervisors. The overarching goal is to improve the agency’s processes and outcomes regarding the WIOA performance measures.

The purpose of this study was to inform development of the dashboard/feedback loop by better understanding the types of employment outcomes consumers are getting, where and why consumers exit the program in the early stages of the eligibility process, and to identify the leading exit reasons for consumers who exit after an Individualized Plan for Employment (IPE), but prior to attaining employment. The results of this study are intended to provide a blueprint for developing a fully functional dashboard and enable the agency to pilot the use of data for problem solving and quality assurance.

The study involved several key evaluation questions important in informing development of the dashboard:

(a) what types of employment outcomes consumers obtain?;

(b) how do outcomes compare between White and non-White consumers?;

(c) how do outcomes compare between male and female consumers?; and

(d) how do outcomes compare across the three regional offices?

Methods

Data for this study was drawn from the 2019 RSA-911 second quarter data set for 553 consumer closures during the prior 18-month period (April 1, 2017 to September 30, 2018). In order to examine the relationship between individual characteristics and competitive, integrated employment (CIE) outcomes, the following three independent variables were used: gender, race (i.e., White vs. non-White), and regional office (i.e., Northern, Central, and Southern). Data were analyzed across the three sites, for further investigation. Evaluation variables included employment outcomes (i.e., CIE) and exit reason. Descriptive statistics are used to illustrate findings.

Results

Competitive Integrated Employment Outcomes

Overall, 251 (45.4%) of 553 consumers achieved competitive integrated employment (CIE). As displayed, there were relationships between individual characteristics and CIE rate. For example, White consumers (49.2%) were more likely to have CIE at exit, compared to non-White (40.4%). Male consumers (46.4%) showed a slightly higher rate of CIE at exit than females (44.1%). Differences were observed in the regional offices, with the Southern Office showing a CIE closure rate of 52.3%, the Central office a CIE rate of 34.8%, and the Northern Office a CIE rate of 47.2%. See Table 1.

Table 1.CIE Rates by Consumer Characteristics
All consumers Consumers with CIE
Number Percent
Race
White 313 154 49.2%
Non-White 240 97 40.4%
Gender
Male 317 147 46.4%
Female 236 104 44.1%
Regional office
Northern 197 93 47.2%
Central 161 56 34.8%
Southern 195 102 52.3%

A large gap was observed between White and non-White consumers achieving CIE among the regional offices. The Central office showed the largest gap in CIE closure at exit (White at 39.3% and non-White at 29.2%). The Northern office CIE closure rate was White (50%) and non-White (44%), and the Southern office had a CIE closure rate of White (55.9%) and non-White (46.8%). See Table 2.

Table 2.CIE Rates by Race and Regional Office
Region Number of White Consumers Consumers with CIE Number of Non-White Consumers Consumers with CIE
Number CIE Rate Number CIE Rate
Northern 106 53 50.0% 91 40 44.0%
Central 89 35 39.3% 72 21 29.2%
Southern 118 66 55.9% 77 36 46.8%
Total 313 154 49.2% 240 97 40.4%

Reasons for VR Program Exit of Consumers Without Employment Outcomes

It was also important to better understand the reasons noted when consumers exited the program following plan development, but without successfully attaining employment. No longer interested in services was the leading exit reason for consumers after accounting for those that either exited after an IPE without achieving CIE. Unable to locate or contact and all other reasons were the other leading categories noted for exiting after IPE development, and prior to employment. These three reasons accounted for 90% of the 314 consumers that exited without achieving an employment goal. Results are described in Table 3.

Table 3.Reasons Consumers Exit from VR Program Prior to Employment
Exit reason Percentage Number
No longer interested in services 38.9% 122
Unable to locate or contact 30.6% 96
All other reasons 20.4% 64
Transferred to another agency 4.8% 15
Death 3.2% 10
Extended employment 1.3% 4
No disabling condition 0.6% 2
Disability too significant to benefit from services 0.3% 1

Recommendations and Implications for Practice

This study served as a pilot to inform development of a data dashboard. In the short-term, identifying where and why CBVI was losing consumers, racial and gender outcome gaps, and differences in CIE rates observed among the regional offices was intended to help the agency focus efforts on what can be done to better retain consumers. Further investigation is needed to identify how we can keep consumers engaged, particularly those who exit as no longer interested in receiving services. The study also highlighted the need to close the racial gap for those successfully achieving CIE. More research is needed to better understand employment outcomes by race, providing a detailed breakdown of the non-White category. As the agency transitions to a new case management system, development of a user-friendly dashboard for staff to use in tracking key performance measures and service will be integral to improving outcomes, enhancing transparency, and reducing disparities.

Based on the findings from this study, the following recommendations are offered to VR counselors and managers to improve VR program processes and performance:

  • A user-friendly dashboard can be used to identify program gaps and strengths and help identify where practice and process improvements can be made.

  • The VR agency needs to further evaluate why consumers are exiting after an IPE, but without an employment outcome, to identify opportunities to strengthen engagement.

  • The agency needs to work on closing the racial gap for those successfully achieving Competitive Integrative Employment outcomes.

  • Further research and evaluation in this area will help inform understanding of how dashboards can be used to communicate performance measures, improve outcomes for VR service recipients, and ensure equal access for consumers.

Author Note

The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.