Background and Purpose
For decades, the development and advances of computer technologies have given rise to new occupations that are increasingly accessible to people with disabilities (Hunt & Berkowitz, 1992; Petrick, 2015; Vedder, 1988). Specialized internal training programs, such as SPR Consulting’s nAblement division in 2003 (Phillips et al., 2016), were created to recruit, train, mentor, network, and support qualified candidates for employment and professional development in information technology (IT), both within the organization, as well as contracted positions with other companies in Chicago (Henthorn, 2017). Phillips et al. (2016) identified challenges to this model, including the limited number of qualified workers with disabilities entering the program, the resulting difficulty in providing employees to business clients in a timely manner, and a lack of proactive professional development by trainees to keep up with rapidly changing IT environments.
Alternatives to internal programs offer options to develop customized training partnerships with technical and community colleges, or IT apprenticeships. However, these training programs have historically focused primarily on basic IT skills, such as those required to work in a call center or provide helpdesk/desktop support (e.g., i-Ability Program; Hire-Ability Vocational Services, n.d.-a, n.d.-b).
The SCVRD ITTC program evaluated for this project is rather unique in that it is directly operated by the vocational rehabilitation designated state unit and offers lengthy and in-depth courses to prepare consumers for skilled positions in fields such as programming and computer-aided design (CAD). Part of the context for this evaluation involved assessing the varying circumstances that exist now, compared to when the program began in 1984. While the program initially provided training opportunities that would not have been accessible elsewhere, people with disabilities can now readily obtain training, certifications, and degrees in IT fields via technical colleges, which have disability services and ADA coordinators to support the identification and implementation of accommodations. Therefore, this study aimed to evaluate the ITTC operated by the South Carolina Vocational Rehabilitation Department at its West Columbia State Office campus to determine effectiveness in delivering training to support participants in obtaining competitive, integrated employment in occupations in the IT sector. The program was originally created in 1984 (South Carolina Vocational Rehabilitation Department, n.d.), prior to the passage of the Americans with Disabilities Act and subsequent amendments, and offered a single curriculum in computer programming to agency consumers with physical disabilities, with content consistent to similar programs initially funded by the International Business Machines Corporation (IBM) (du Laney, n.d.; Hunt & Berkowitz, 1992; Nelson, 1994; Petrick, 2015; United States Department of Education – Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration, 1991; Vedder, 1988). The ITTC program also provides lodging, meals, and personal care support on-site.
With guidance from a Business Advisory Council (BAC) composed of executives of various local corporations, governmental entities, and educational institutions, the program evolved to include courses on business applications, CAD, and network and server support. The additional offerings expanded occupational preparation options for participants, which, in turn, expanded the number of agency consumers who were appropriate for referral to the program. The ITTC’s training programs are lengthy and range from approximately six (6) months to 14 months in duration, with a limited number of spaces available. Therefore, it was important for the agency to determine its value to consumers and potential employers, and better understand who benefits from this program, to inform the investment of public funds.
Given the lack of contemporary research regarding the effectiveness of internal IT training programs when compared with external options for people with disabilities, this project aimed to evaluate the ITTC to learn whether or not the program was cost-effective and necessary after years of operation under a set of historical assumptions. The findings were intended to extend knowledge of the program’s strengths and limitations, improve design and implementation, and inform referral and selection of consumers for participation. The evaluation questions for the project were as follows:
-
What ITTC courses are leading to Successful Employment Outcomes (SEOs) related to the training?
-
How do participant outcomes compare to outcomes of non-participant consumers with the same or similar vocational objectives?
-
What are the ITTC program’s costs per consumer served and how do these compare to the costs for non-participant consumers?
Methods
The evaluation used a non-experimental design, quantitatively analyzing existing program data to answer the evaluation questions. The study population included all consumers who received training from the SCVRD ITTC between July 1, 2015 and June 30, 2018 (N = 114). Data were collected via a query of the agency’s Case Management System (CMS) to identify Case Service Report (RSA-911) data elements for program participants with open and closed cases. Additional data elements were appended to the file and included (a) program admission and exit dates, and (b) certifications earned during the training. Due to variations in course length and start dates, the duration of services ranged from a minimum of just a few days for those who enrolled shortly before the study period ended to a maximum of 14 months. The query identified 114 program participants enrolled across 16 combined sections of the four (4) courses offered at the ITTC.
The combined dataset was reviewed to identify all occupations in which program participants achieved successful employment outcomes related to the training received, with a second query producing a comparative sample of agency consumers. The criteria for consumer inclusion in the comparative sample were as follows: (a) had an IPE developed and received services; (b) case closed successfully or unsuccessfully during the specified period; and (c) did not receive training from the ITTC, but had a vocational objective in common with program participants. The comparative sample included 2,149 consumers.
Measures
For the study population, counts were produced by course and aggregated for the consumers enrolled in ITTC. ITTC participants whose cases had been closed constituted the target sample, as these consumers had outcome data and were most similar to the comparative sample. Furthermore, rehabilitation rates, calculated as the number of Successful Employment Outcomes (SEO) divided by the total number of case closures, were computed and compared for the target and comparative samples.
Descriptive statistics pertaining to hourly wages and weekly hours worked at case closure were generated for the full target sample. Additionally, F-tests and t-tests were conducted to investigate the differences in wages and weekly hours, as well as differences within the target sample for those who earned certifications and those who did not.
Costs
To calculate the study population and non-participant costs per consumer served, expenditure data from the agency’s accounting system for the 3-year period were obtained from agency fiscal staff, and data regarding total number of consumers served were pulled from existing statistical reports. Expenditures included personnel costs (salaries and contributions), contractual services, supply and material costs, fixed charges, travel expenses, utilities, and capital, as well as non-capital assets and equipment. Consumer service expenditures were excluded from the calculation for both the ITTC participants and non-participant consumers, with the rationale that those costs are consumer-specific (variable by year based on the individual needs of the consumers served) and may not be directly related to the program. Most consumer service expenditures would be incurred whether the consumer was enrolled in the ITTC program or not; and therefore, these costs were not included in the comparison data. After deducting consumer service expenditures from total expenditures, the resulting adjusted expenditures figure was used for comparison of ITTC cost per consumer served and non-participant cost per consumer served.
Results
As of February 28, 2019, all but one of the courses (section 008 of Computer-Aided Drafting with nine consumers enrolled) had concluded. Sixty-six (62.86%) of the 105 participants whose course had concluded had completed their training. Overall, 55 had exited the VR program, while 59 of the 114 served in ITTC remained in an active status (i.e., still had an open case with the agency). The 55 ITTC participants whose cases had been closed were the target sample for quantitative analyses, as this subset of the study population had outcomes data available to analyze and was most similar to the comparative sample. Thirty-three (33) members of the target sample had at least some post-secondary education at the time of application (60% vs. 58% of the comparative sample). Of those with closed cases, 32 achieved a successful employment outcome, and 23 had their cases closed unsuccessfully. This represented a 58.18% rehabilitation rate, compared with 50.49% for the comparative sample and 56.65% for general program cases during the study period. Average weekly hours worked at case closure was 36.59 for the target sample (37.18 for occupations related to training). Average hourly wage at closure was $13.58 for the target sample ($14.66 for occupations related to training). Of the 32 SEO closures from the target sample, only 22 (68.75%) were closed in an occupation related to training received at the ITTC. The number of completers as a proportion of total capacity (i.e., seats available across all sections) was 57.89%; see Table 1.
Courses Leading to Successful Employment Outcomes Related to Training
Of the courses offered at the ITTC, Business Applications Plus (BAP) produced the most SEOs, representing 13 of the 32 total consumers trained during the study period. This was not unexpected given that the course is the shortest in duration and, consequently, enrolled and trained more consumers than did the other courses during the study period. BAP also had the highest ratio of completion as a percentage of capacity (72.50%). The CAD course had the second highest enrollment and produced the second highest rate of SEOs. Though the Programming course had the highest rehabilitation rate (66.67%), only one third of those SEOs were in occupations related to the training. BAP had the highest percentage of SEOs in occupations related to training (84.62%), followed by the CAD course (70%). Network and Server Support (NSS)—which along with CAD is one of the two courses that offers the opportunity to earn certifications—had the second lowest enrollment, produced the second fewest SEOs, and had the lowest rehabilitation rate (50%).
ITTC Outcomes vs. the Comparative Sample
ITTC participants had a higher rehabilitation rate (58.18%) than consumers in the comparative sample for the 3-year study period (50.49%). Furthermore, ITTC participants had higher average weekly hours worked (36.59 overall; 37.18 for jobs related to training) and higher average hourly wages at case closure ($13.58 overall; $14.66 for jobs related to training) than did consumers in the comparative sample (34.61 weekly hours worked and $13.02 in wages).
Prior to conducting a series of independent t-tests, F-tests were conducted to determine whether differences between the ITTC and comparative sample variances (for occupations related to training) were statistically significant. The p value of each was < .05, indicating a statistically significant difference in wage distributions. Figure 1 depicts the differences in wage distributions for the two groups. Having established unequal variances, independent samples, two-tailed t-tests were then conducted to determine whether the means of the two groups on hourly wages at case closure and weekly hours worked at case closure were statistically significant. The $1.64 difference in means for hourly wages was not statistically significant at the p < .05 level; however the 2.57 hours difference in the means of weekly hours worked at case closure between the two groups was statistically significant at the p < .05 level.
Average Expenditures Per Consumer Served
The ITTC’s adjusted (i.e., non-consumer services expenditures) cost per consumer was $18,947, or 4.84 times the general program’s cost per consumer served for the same period, which was $3,915. The cost to serve ITTC consumers was higher than average cases in the general program, which is not surprising as it is a comprehensive program, with lodging, meals, and nursing provided on-site. This study revealed the difference in actual cost, which is important for the agency to understand when considering alternatives to direct training provision.
Recommendations and Implications for Practice
The nature of the program and outcomes (including earnings) data suggest that, like SPR Consulting’s nAblement program (Phillips et al., 2016), the ITTC program serves consumers with fewer barriers and greater potential than the general program on average, but the programmatic costs to achieve the outcomes were higher. While cost per consumer served for the ITTC was anticipated to be higher than for the general program because it is a comprehensive program, the agency should nonetheless consider whether it may be more cost-efficient to sponsor consumers for IT training via technical college programs or other training providers, rather than provide and deliver the training on-site. Technical college training programs offer accommodations and opportunities for integration with the population of students without disabilities, who will be peers in the IT workforce.
Furthermore, small class sizes at the ITTC increase the cost per student for instruction; and completion rate and completers as a percentage of capacity during the study period were underwhelming to the agency’s administration. It is challenging to admit an adequate number of qualified candidates into the program to effectively cover costs and meet outcome goals. This is not unique to the ITTC, although fiscal sustainability of the program is more challenging now that consumers have other viable options for accessing similar training, as certifications can be obtained more quickly via short-term, external programs. Given that the certifications which study participants obtained while at the ITTC were not demonstrated to have a statistically significant impact on earnings, it is understandable for consumers to pass on the program in favor of a shorter program elsewhere.
The ITTC previously achieved an impressive 81% placement rate through its first 11 years of operation (South Carolina Vocational Rehabilitation Department, 1995a), which was 15% higher than the national average for similar programs (South Carolina Vocational Rehabilitation Department, 1995b). By fiscal years 2016-2018, however, the rehabilitation rate had dropped to 58%. Moreover, many ITTC participants still had open cases several months after completing their training, suggesting a lag in finding suitable employment. Additionally, of those consumers who received training at the ITTC during the study period and achieved a successful employment outcome, only 68.75% were in an occupation related to the training received. Average hourly wages for participants of all ITTC courses except BAP were also lower than those of the comparative sample for the same clusters of occupations related to those trainings. The following recommendations are offered based on lessons learned through this study:
-
VR agencies can use evaluation findings to inform strategic planning and decisions regarding how to best invest public resources into quality services that produce outcomes.
-
If another state opts to conduct a similar evaluation, it is recommended that data about participants’ previous IT experience and training be collected to help strengthen the outcomes data.
-
When conducting a cost-benefit analysis of agency-provided training, VR agencies need to consider overall employment outcomes, training field-specific employment outcomes, case closures, and the cost of operating the specific training program in the assessment.
-
As a result of the study, viable alternatives to internal provision of IT training will be considered. One possibility is to co-sponsor training via technical colleges or other training providers, although additional options will be explored.
-
SCVRD’s ITTC program staff have deep industry knowledge and are involved in job development and placement efforts for the consumers served in the program. This may be a key factor in the finding that ITTC consumers who achieve a successful employment outcome in an occupation related to the training received earn more on average than consumers in the comparative sample.
Author Note
The contents of this paper were developed under a cooperative agreement with the U.S. Department of Education, Technical Assistance Center for Vocational Rehabilitation Agency Program Evaluation and Quality Assurance (PEQA-TAC) (Grant Award Number: H263B150004). However, the contents and views expressed in this publication do not necessarily represent the positions or policies of the U.S. Department of Education, and you should not assume endorsement by the Federal government.