The Effectiveness of Integrated Resources for People With Mental Health and Employment

Individuals living with mental health disabilities frequently encounter challenges in obtaining and maintaining employment. According to the National Alliance on Mental Illness (NAMI), the national unemployment rate for individuals receiving public mental health services is approximately 80%, despite approximately 60% of the 7.1 million receiving public mental health services expressing that they want to work (National Alliance of Mental Illness, 2014). Individuals who receive mental health services frequently face more significant barriers to work than do people with other disabilities. Prior findings indicate that vocational rehabilitation (VR) applicants with psychiatric disabilities experience longer periods of unemployment than individuals with physical disabilities, and are significantly more likely than individuals with physical disabilities alone to report nonhealth reasons, such as getting fired and lacking skills, as barriers to employment (Sevak & Khan, 2017).

However, research also indicates that this population can be successfully employed in jobs that are socially integrated and coordinated with both clinical and vocational rehabilitation services (Cook & Razzano, 2000). Studies have also demonstrated linkages between employment and improved parts of well-being, such as mental health (Bond & Drake, 2014). Furthermore, rapid response and engagement, accelerated entry into competitive employment, and on-the-job support services appear to contribute significantly to employment outcomes (Bond & Drake, 2014; Del Valle et al., 2014; Tashjian & Hayward, 1989).

The Massachusetts Rehabilitation Commission (MRC), in collaboration with the Massachusetts Department of Mental Health (DMH), recently launched a new program providing employment services to DMH job seekers participating under the Adult Community Clinical Services (ACCS) program. The program is designed to use an intensive service model for individuals with mental health disabilities seeking employment. The model is based on the concept of rapid engagement and the use of an Integrated Resource Team (IRT) approach. IRT is recognized as a promising practice designed to enhance cross-program collaboration across multiple systems and service providers for individuals with significant disabilities (Anderson et al., 2021).

The short-term goals of this program were to rapidly engage individuals with mental health challenges who are DMH clients in the Vocational Rehabilitation (VR) Program, utilize an IRT approach to provide intensive and more one-on-one supports to ACCS MRC VR clients to assist them in finding competitive integrated employment, and use Key Performance Indicators (KPIs) and other data points to make decisions and evaluate the effectiveness and efficiency of the MRC ACCS program. Specific evaluation questions addressed in this project were:

1. Do consumers served through the MRC ACCS program achieve waypoints/steps in the VR process faster than consumers in the regular VR program?

2. Does consumer satisfaction differ among those participating in the MRC AAC intervention and comparison groups?

3. Does the MRC ACCS program model lead to reduced dropout rates and more prompt delivery of services through rapid engagement compared to the regular VR program?

4. Does the MRC ACCS program model intervention lead to a higher employment rate for individuals with mental health challenges compared to the regular VR program?

Method

MRC utilizes qualitative and quantitative methods for data collection. Analyzing both program data, as well as stakeholder input, allowed for a richer understanding of the operation and outcomes of the MRC ACCS program. The primary data source was the MRCIS Case Management system, an internal comprehensive case management system containing personal history (i.e., contact details and demographics), types of services/supports received, and employment outcomes on all MRC VR consumers. Data was pulled for both treatment and comparison groups, representing outcomes and services received during July 1, 2019 to June 30, 2020. The treatment group consisted of shared MRC/DMH consumers applying to MRC Vocational Rehabilitation Program for the ACCS program between July 1, 2019 and December 31, 2019. The comparison group consisted of consumers (a) applying to the MRC Vocational Rehabilitation program during the same time period of July 1, 2019 to December 31, 2019; with (b) mental health disability as their primary impairment (defined as RSA Disability Impairment Code 18, Psychosocial Impairments) and assigned by MRC’s Order of Selection process to Category 1 (individuals with the most significant disabilities); and (c) were not part of the MRC/ACCS program.

Program data was analyzed based on a series of defined Key Performance Indicators (KPIs) developed for the program. The KPI cover multiple points in the VR process, including (a) engagement with counselors; (b) timeliness of the eligibility determination; (c) individualized plan for employment development; (d) utilization of services, such as benefits planning and job placement services; (e) employment placement and outcomes; (f) wages and number of hours worked; (g) and the dropout rate/non-employment exit of participants early in the VR process.

Survey data was gathered from consumers in June 2020 by the MRC Analytics and Quality Assurance Department using Qualtrics, a web-based data collection tool. The purpose of the survey was to assess satisfaction with Individualized Plan for Employment (IPE), level of engagement with staff and service providers, service types received, and service delivery. The MRC VR Employment Consumer Survey consisted of 15 questions, and the survey was sent to consumers who actively received services between July 1, 2019 and December 31, 2019 as a participant of either (a) the MRC-DMH ACCS Individualized Resource Team (IRT) group, or (b) the General VR employment services group. While a total of 807 individuals received services (306 treatment group and 501 comparison group), valid email addresses were only available for 407 participants. The demographic composition for both program participants were similar.

Results

Data Analysis of Key Performance Indicators

Results for each KPI were calculated for both the treatment (n = 306) and comparison groups (n = 501). Both groups applied to VR services between July 1, 2019 and December 31, 2019, with outcome data through the end of June 2020. Across the groups, 17.8% of the treatment group were closed successfully or unsuccessfully by June 30, 2021 compared to 23.4% in the comparison group; the remainder of the groups remained active. The results for each KPI and the difference in outcomes between the treatment and comparison groups were then analyzed for variance and/or statistically significant findings. Overall, the analysis of the KPI illustrated several key themes and areas of success. There were KPI where the treatment group (MRC ACCS consumers) experienced higher performance than the comparison group, specifically in the areas of faster eligibility determination, benefits planning services in IPE, successful employment placement, reduced time from VR application to job placement, increased rate of post-employment support, and decreased rate of exit prior to employment. However, the comparison group experienced higher hourly wages and higher number of hours worked than the treatment group. See Table 1.

Timeliness in Eligibility Determination

The data analysis showed that the MRC ACCS consumer treatment group had a higher rate of timeliness in eligibility determination at both 30 days after application for VR services (66% versus 51% for the comparison group), and at 60 days from application (96% compared to 93.6% for the comparison group). The mean time from application to eligibility determination was 24.8 days for the treatment group compared to 30.5 days for the comparison group—a statistically significant difference (p < .001). An eligibility determination timeframe of up to 60-days is the federal standard for VR programs.

Timeliness in Individualized Plan for Employment Development

In terms of timeliness from eligibility determination to IPE development, 47% of MRC ACCS consumers had their IPE developed in 45 days or less compared to 43.5% for the comparison group. An IPE development timeframe of up to 90 days is the federal standard. The mean time for IPE development amongst the treatment group was 51.4 days compared to 53.5 for the comparison group. The difference was not found to be statistically significant.

Utilization of Benefits Planning Services

A key aspect of the MRC ACCS program involved the provision of wraparound supports and services, such as benefits planning, to assist participating consumers in successfully obtaining employment. Benefits counseling and planning services are specifically designed to assist consumers who receive disability benefits, from the Social Security Administration and other public entities, better understand the interplay between earnings and their benefits. This service can assist consumers in both understanding and utilizing the array of work incentives available through public benefit programs that encourage and support employment activities.

The data analysis showed that 53.9% of consumers in the treatment group had benefits counseling as a service provided in their IPE compared to only 20% of consumers in the comparison group, a statistically significant difference (p < .001). Further analysis regarding specific demographic and benefits utilization data did not yield significant differences, indicating that the program model is likely the main factor in this key finding.

Job Placement and Job Retention Outcomes

Overall, there were both a higher number and rate for initial first day job placements of treatment group of participating MRC ACCS consumers versus the comparison group. As such, 15.3% of the treatment group were placed prior to June 30, 2020 compared to 7.8% in the comparison group, a statistically significant difference (p = .004). For 90-day retention outcomes, 8.5% of the treatment group achieved 90 days of employment compared to 5.2% of the comparison group; however, this difference was not found to be statistically significant. Sixty-seven percent (67%) of those placed in the treatment group achieved a 90-day outcome by June 30, 2020, compared to 64% of those placed in the comparison group. These findings suggest that the initial employment rate for the first year of the program was higher amongst the treatment group.

Time to Placement Into an Employment Opportunity

Individuals in the treatment group who were successfully placed in employment by June 30, 2020 took an average of 164.5 days from the date of application to the first day of employment, compared with an average of 188 days for individuals in the comparison group. Comparison of the means indicated that the difference in the means between the two groups was statistically significant at a 95% confidence interval (p = .006).

Utilization of Post-Employment Support Services

The MRC ACCS consumer population in the treatment group had a significantly higher utilization of post-employment support services than the comparison group. Thirty-eight percent (38%) of consumers in the treatment group who achieved a 90-day employment outcome were receiving ongoing employment supports compared to only 8% of those in the comparison group. This difference was found to be statistically significant (p = .001). It is recommended that this be further evaluated as the project continues to mature and more consumers obtain employment, yielding a larger sample for analysis.

Hourly Wages and Hours Worked Per Week for Placed Job Seekers

The ACCS treatment group had an average wage of $14.04 per hour and a median of $12.75 per hour compared to $14.76 average and $13.00 median hourly wage for the comparison group. This was the only area in our findings where the comparison group had higher outcomes than the ACCS treatment group.

In regard to the number of hours worked per week, the treatment group worked an average of 20.1 and a median of 20 hours per week compared to an average of 26.7 hours and a median of 25 hours per week for the comparison group. The difference between the comparison and treatment group for wages was not statistically significant; however, the difference in hours worked per week was found to be statistically significant (p = .005).

Unsuccessful Closures and Dropout Rate From the Program

Consumers participating in the ACCS program had notably lower unsuccessful closure numbers and lower dropout rates from the program compared to the comparison group. Only 14.5% of ACCS program referrals dropped out prior to eligibility determination compared to 39.2% of those in the comparison group. This difference was found to be statistically significant (p = .001).

After service initiation following IPE completion, a total of 2.2% of ACCS consumers in the treatment group were closed unsuccessfully with services without employment compared to 2.4% of the comparison group. This difference was not significant, but should be monitored to assess trends and patterns as the program continues to evolve.

The overall dropout rate of ACCS job seekers from the program between July 1, 2019 and June 30, 2020 was 11.1% compared to 18.2% for the comparison group. For those dropping out prior to IPE completion, the rate was 8.8% for the treatment group compared to 17.4% for the comparison group. The difference between the two groups in terms of the dropout rate was found to be statistically significant (p = .009).

MRC VR Employment Consumer Survey

Of 407 surveys sent, a total of 107 survey responses were received, representing a response rate of approximately 26%. Overall, 77% of consumers indicated they were satisfied with the services provided under the VR program, with a majority of those (39%) being very satisfied. Most consumers (81%) indicated being very satisfied with the level of engagement with their counselor, while the remaining responses indicated some levels of dissatisfaction. Consumers were generally satisfied (83%) with their IPE or employment plan and their counselor.

Overall, most respondents seem to be satisfied with MRC services. There were no significant differences between both groups in terms of satisfaction or perceived usefulness ratings, as both groups seemed to respond similarly regardless of receiving or not receiving ACCS services. In terms of timeliness of services, 80% of consumers indicated satisfaction while 20% indicated being dissatisfied. When reporting on benefits planning services being received, a high level of satisfaction (89%) was reported from consumers. The survey further asked consumers to rate their experience in the VR program relative to specific services received, including (a) job engagement, (b) job placement, (c) integrated service delivery, and (d) support and communication from agency staff. See Table 2.

Consumers were also asked to rate the perceived usefulness of specific VR services, such as Career Exploration and Engagement, Employment-Based Skills Training, Job Development and Placement, Initial Employment Supports, Ongoing Supports, and Interim Flexible Supports, in supporting their employment process and outcomes. See Table 3.

Conclusion, Recommendations, and Next Steps

Overall, analysis of the preliminary data and survey results suggest that the first year of the MRC ACCS program model appears to be an effective intervention to serving individuals with mental health challenges in the VR program. The second half of the first year of the program was significantly impacted by COVID-19, but despite this challenge, the program still achieved employment outcomes and continued to serve consumers. Our sampling approach attempted to control the impact of COVID-19 to the data analysis by looking at a cohort of individuals who applied to VR in both the treatment and comparison group prior to the pandemic. Furthermore, consumers in both program groups noted overall satisfaction and usefulness with the VR program and services received. However, future evaluation may want to qualitatively follow-up with those noting lower levels of satisfaction and perceived usefulness to better understand the context and inform further programmatic improvements. Findings on the early effectiveness of the model are demonstrated and the following recommendations are offered for other VR programs considering similar program implementation:

• Close collaboration between VR and state mental health programs can decrease the amount of time needed to determine VR program eligibility, as well as help reduce the amount of time between application and successful employment placement in support of a rapid engagement model.

• Collaboration between VR and state mental health programs can help intentionally increase the employment rate of individuals with significant mental illness and help reduce the VR dropout rate with this population.

• Further research and analysis of the program going forward will provide longer-term outcomes data, which will help inform further policy and practice.

• It is recommended that further research be conducted on consumer satisfaction to see if the results change as the program continues to mature going forward.

• The initial data suggests that the IRT approach and the use of wraparound supports from DMH are contributing factors toward supporting better employment outcomes for participating consumers. It is recommended this be closely evaluated going forward as the project continues to mature to monitor trends and patterns in this area, and to determine longer-term outcomes.

• One area for further evaluation and an area for improvement pertains to the number of hours worked per week. MRC ACCS consumers work fewer hours per week than individuals with similar disabilities served during the time period. There are several possible reasons for the higher hours per week worked among the comparison group compared to the ACCS consumer population. One may be that the ACCS group may have more significant needs than the comparison group; another may be that this group has less work history or needed extra supports or time to adjust to working more hours on the job. This is an area recommended for investigated further and may be an area that MRC should focus on for performance improvement as the program evolves.

• The finding of lower numbers of front-end closures and a reduced dropout rate among MRC ACCS participants is significant, as this is an area with which VR agencies often struggle, particularly with the mental health population. This suggests that the model used for the ACCS program, focusing on rapid engagement, smaller caseload size, and the IRT approach, is very effective in improving consumer engagement with VR, resulting in reductions in the number of front-end closures and dropout rate. MRC has seen similar findings with reduced front-end closures and lower unsuccessful dropout rates with youth with disabilities in two federal demonstration grants. Both of these grants also focused on increased engagement with consumers and utilized specialty caseload concepts, with one using a similar concept to the integrated resource team.

• It is recommended that MRC continue to closely evaluate the MRC ACCS program to see if these initial findings on the success of the model continue as the program evolves, and to see if longer-term trends continue to support these initial findings.

Author Note

The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.