Background and Purpose

Identifying factors associated with the discrepancy in the postsecondary employment rates between students with disabilities and their peers without disabilities has been the subject of decades of research (see Fourqueran et al., 1991; Mazzotti et al., 2016; Test et al., 2009). Despite the expanded knowledge base generated, a need to better understand the mechanisms by which these factors influence outcomes remains. Predictive factors, such as high parental expectations for youth holding a job after high school graduation, family involvement, life skills instruction, interagency involvement, communication skills, independent living skills, job skills training, social skills, and early work experience, are all correlated with postschool employment of youth with disabilities (Carter et al., 2011; McDonnall, 2011; Wehman et al., 2015). Recently, research regarding the role of early work experience in postsecondary employment outcomes has begun to shift from simply identifying associated factors toward investigating causal relationships between specific interventions and employment outcomes (Balcazar et al., 2018; Fraker et al., 2018; Langi et al., 2016; Luecking et al., 2018).

The passage of the Workforce Innovation and Opportunities Act of 2014 (WIOA) renewed interest in developing effective interventions and service delivery models to increase postsecondary employment outcomes among students with disabilities. As such, Missouri Vocational Rehabilitation (MOVR) developed a summer work experience (SWE) program in early 2015, with the first cohort of participants starting in May of the same year. Participants were required to be VR-eligible students with disabilities who had no prior work experience and would be entering their final year of high school after completing the summer work experience. The SWE was delivered locally through an existing statewide network of community rehabilitation programs (CRP). This network consisted of established VR service providers who had previously been delivering a variety of other employment-related services. All CRP service locations in Missouri were offered the opportunity to develop a summer work experience site and provide this service locally to students with disabilities. Once established, the available SWE opportunities were publicized to local VR counselors for outreach and recruitment purposes.

The purpose of this study was to better understand the mechanisms underlying the impact of the summer work experience on postsecondary employment outcomes of student participants. Specifically, this study sought to answer: are summer work experience participants more likely to exit VR with a successful employment outcome than those who did not participate?

Methods

Participant Sample Selection

The population for this study involved students with disabilities who participated in the MOVR summer work experience program during the years 2015, 2016, and 2017. Selection of participants was not random, as the local VR counselor, school, and CRP staff, together, identified potential participants who were determined eligible for VR services, attended a school with an established SWE site nearby, and expressed an interest in participating. A comparison group was identified based on (a) VR application date; (b) participant age at application; and (c) participation in VR services under an IPE. Applying the sampling parameter strategies resulted in a sample of 2,821 total participants, with 816 of these individuals having participated in the summer work experience program. Administrative data from the Missouri VR case management system was the sole source of data for this study.

Dependent Variable

Successful exit from the VR program is measured as individual exited after an IPE in competitive and integrated employment or supported employment (Rehabilitation Services Administration, 2017). For students with disabilities, this requires that (a) the individual be employed in a competitive and integrated work setting for a minimum of 90 days after receiving VR services; (b) is satisfied with and no longer requires VR services; and (c) is no longer enrolled in high school. The dependent variable was dichotomous indicating whether the case was closed successfully (1) or not successfully (0).

Independent Variables

The primary independent variable was whether or not an individual completed the Missouri VR summer work experience during the summer of 2015, 2016, or 2017. Summer work experience participation was represented by a dichotomous variable indicating whether or not the student completed the summer work experience (1= yes, 0 = no). Furthermore, the participant’s gender was represented by a dichotomous variable for gender (1 = male, 0 = female). Race and ethnicity were included as White, Black, and Other. While all participants included in the sample were age 16, 17, or 18, participant age was ordered to control for the influence of age on VR outcomes. Five disability type categories were included: (a) sensory impairments, (b) physical impairments, (c) cognitive impairments, (d) psychosocial impairments, and (e) other mental impairments; the students’ level of disability severity was represented by a dichotomous variable (1 = Most Significant Disability [MSD] and 0 = not MSD).

Data Analysis

Demographic variables were used to derive a propensity score (PS) via a logistic regression model (Wright, 1995), using SWE participation as the outcome of interest. A stratification approach was applied to estimate how summer work experience participants and non-SWE participants compared in terms of successful outcomes. Following an initial descriptive comparison of participant characteristics, linear probability modeling (LPM) was used to examine differences between the treatment and comparison groups’ relationship with successful employment outcomes, while controlling for the previously identified observable characteristics (i.e., gender, race/ethnicity, age, disability type, and MSD status).

The PS population model was specified as:

  • Summer work experience participation = β0 + β1 (gender) + β2 (race/ethnicity) + β3 (age) + β4 (primary disability) + β5 (MSD) + µ

The LPM population models were specified as:

  • Successful Employment Outcome = β0 + β1 (summer work experience participation) + µ

  • Successful Employment Outcome = β0 + β1 (summer work experience participation) + β2 (gender) + β3 (race/ethnicity) + β4 (age) + β5 (primary disability) + β6 (MSD) + µ

Results

Among the overall sample (n = 2,821), 816 participants (29%) completed the summer work experience during the years 2015, 2016, and 2017. Of the overall sample, 1,827 (65%) participants were male and 994 (34%) were female. One hundred ninety-five participants (7%) included in the sample were age 16 at application, 1,201 (43%) were age 17, and the remaining 1,425 (50%) were age 18. In terms of race/ethnicity, 2,098 (74%) of the sample was White, 594 (21%) Black, and the remaining 129 (5%) participants were identified as Other. Most students had cognitive (63%) or psychosocial impairment (24%), while 94 (3%) were participants with a sensory disability, 157 (6%) with a physical impairment, and the remaining 127 (4%) indicated a disability described as “other mental impairment”. The majority-1,720 (61%) participants in the sample—were individuals coded at eligibility as having a Most Significant Disability (MSD). One thousand five hundred sixty-four (55%) participants exited VR with a successful employment outcome, and the remaining 1,257 (45%) participants’ VR cases were closed unsuccessfully. Table 1 describes proportions of participants for all included variables for those who completed the summer work experience, those who did not participate in the summer work experience, and the sample as a whole.

Table 1.Participant Characteristics
Variable SWE
(%)		 Non-SWE
(%)		 Full sample
(%)
Successful employment outcome 50 58 55
Age at application
16 17 3 7
17 62 35 43
18 21 62 50
Male 62 66 65
Race
White 74 74 74
Black 19 22 21
Other 7 4 5
MSD 63 60 61
Disability type
Sensory 3 3 3
Physical 3 6 6
Cognitive 67 61 63
Psychosocial 20 25 24
Other mental 6 4 4

Note: Percentages in bold represent variables that, based on t-tests, had statistically significant differences between the summer work experience participant (SWE) and non-participant groups (non-SWE) at the p < .05 level

Linear Probability Model

A linear probability model was utilized to predict the probability an SWE participant would exit the VR program with a successful employment outcome. When summer work experience participation is the lone predictor variable and successful employment outcome at program exit is the dependent variable, the model is statistically significant at the p < .05 level, but explains only a small amount of the variance associated with the dependent variable, F (1, 2819) = 15.76, p = 0.0001, Adjusted R2 = 0.005. The coefficient representing summer work experience participation is statistically significant at the p < .05 level, and negatively associated with the independent variable of interest, β = -0.08, p =0 .000. The model, which includes all described independent variables including summer work participation, explains more of the variance and is significant at the p < .05 level, F (11, 2809) = 2.90, p = 0.000, Adjusted R2= 0.02. Taken individually, four predictor variables, being male as compared to female, β = 0.09, p = 0.000, being Black, as compared to White, β = -0.08 p = 0 .001, having a physical disability as compared to a cognitive disability, β = -0.10, p = 0 .01, and having a psychosocial disability as compared to a cognitive disability, β = -0.05, p = 0 .027, are each statistically significant at the p < .05 level, while holding all other variables constant. Table 2 provides a summary of the two linear probability models described above.

Table 2.Predicted Probability of Exiting the VR Program With a Successful Employment Outcome
(1) (2)
Summer work experience participant -0.08 -0.09**
Male 0.09**
Age at application
16 0.06
17 -0.009
Race/ethnicity
Black -0.08**
Other -0.04
MSD 0.02
Primary disability
Sensory 0.04
Physical -0.10**
Psychosocial -0.05**
Other mental -0.07

** p <.05
Note: “18 years old” served as the omitted variable for Age at Application, “White” as the omitted variable for Race/Ethnicity, and “Cognitive” as the omitted variable for Primary Disability.

Propensity Score Stratification

Propensity scores were derived using logistic regression modeling to assess the differences between the treatment and comparison groups due to the lack of random assignment. As observed in Table 1, individual characteristics of the two participant groups were different, which indicates that the program assignment was not done at random, and two groups might not be comparable. Figure 1 compares non-summer work experience participants (0) to summer work experience participants (1) based on derived propensity score. The propensity score matching procedure produced significant overlap in scores, suggesting that any derived effect of SWE participation is the result of participation in the summer work experience, and not due to differences between the groups themselves. As illustrated in Figure 1, a low proportion of the participants showed the common area in terms of the propensity score. Looking at the participants whose propensity scores were overlapped (n = 341), 50% of SWE participants (versus 58% of non-SWE participants) achieved an employment outcome at exit.

Figure 1
Figure 1.Non-Summer and Summer Participant Work Experiences Box Plot

The propensity score generation process was successful in creating more balanced summer work experience and non-summer work experience participant groups (n = 2,805), as shown in Table 3.

Table 3.Group Characteristics After Propensity-Score Balancing
Variable SWE participants Non-SWE participants
Successful employment outcome 50% 58%
Age 16 at application 17% 13%
Age 17 at application 62% 60%
Age 18 at application 21% 27%
Male 62% 58%
White 74% 75%
Black 19% 18%
Other race 7% 7%
MSD 63% 64%
Sensory 3% 4%
Physical 3% 4%
Cognitive 67% 69%
Psychosocial 20% 17%
Other mental 6% 7%

Based on the derived Linear Probability Model estimates and propensity score stratification analysis, findings suggested that the specialized summer work experience program did not result in significantly improved VR outcomes when compared with those who received other services. In fact, results suggested that summer work experience participants were 8-9% less likely to exit the VR program successfully when compared to non-SWE participants.

While study results did not reveal improved outcomes associated with the summer work experience program, this does not mean participating students did not benefit in other ways, which were not addressed here. Additionally, the lack of statistically significant difference in outcomes between the two groups may be explained by other causal mechanisms not accounted for in the current study.

Recommendations and Implications for Practice

This study aimed to evaluate one specific type of student-focused early work experience program offered through the Missouri State VR agency. While the results offer mixed support in answering the research question, some unexpected, yet interesting results emerged which can inform future efforts, including:

  • Explore the impact of post-summer work experiences (job placement, supported employment, training, etc.) on employment outcomes between summer work experience participants and non-participants.

  • Investigate the effect of evaluation timing (e.g., at the completion of SWE or after high school graduation) on employment outcomes.

  • Consider other dependent variables of interest to better understand other ways SWE is impacting participants, beyond only successful employment outcome measures.

  • Consider the use of an instrumental variable to address Linear Probability Model bias concerns.

  • Continue to pursue answers about the causal mechanisms influencing the summer work experience on VR program exit in an effort to move toward a more effective and meaningful intervention for youth with disabilities.

Author Note

The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.