An assessment of vocational rehabilitation

Draft Study Plan – Subtask 4.1

Submitted by The Study Group Inc. to Rehabilitation Services Administration, U.S. Department of Education in partial fulfillment of requirements under ED-04-CO-0042, An Assessment of Transition Policies and Practices In State Vocational Rehabilitation Agencies.

A. Introduction

This report outlines the Draft Study Plan (Subtask 4.1) of An Assessment of Transition Policies and Practices in State Vocational Rehabilitation Agencies (ED-04-C0-0042). The Draft Study Plan is designed to meet the study objectives set forth in the procurement. These procurement objectives are: (a) describe and classify state policies and practices for identifying and serving youth with disabilities who are transitioning from school to work; (b) identify and describe policies, promising practices, and other factors that facilitate effective collaboration, transition planning, and provision of services; (c) identify major obstacles to collaboration and early intervention in transition planning; and (d) examine the influence of financial management considerations including provisions of complementary or matching funds to the VR agency by educational or education-related agencies.

The Draft Study Plan is based on the summary findings and information obtained in relation to the completion of Subtasks 3.1, literature review, 3.2, interviews with selected federal agency staff, and 3.3, review of state and federal documents and reports associated with the transition-age students with disabilities. The procurement objectives are further detailed in Attachment A. This report is also framed and supported by careful review of current Vocational Rehabilitation Act regulations, and joint OSEP/RSA monitoring criteria and guidelines.

This Draft Study Plan consists of: (a) the data collection instrument, which will collect information from all 80 state vocational rehabilitation agencies, (b) a plan for pre-testing the instrument, and (c) the data analysis plan. The Draft Study Plan also demonstrates how items on the data collection instrument address each objective of this procurement. The work for Subtask 4.1 is carried out by The Study Group Inc. (TSG) and its subcontractors.

B. Use of Information from the Preliminary Activities of Subtasks 3.1-3.3

The Draft Study Plan builds upon the knowledge gained in Subtasks 3.1-3.3. The following summarizes how and what aspects of this information have been used to support the development of the draft data collection instrument.

1. Subtask 3.1, Literature Review

More than 90 documents were reviewed, including 70 articles published in peer-reviewed journals between 1990-2004 that are contained in the What Works in Transition: Synthesis Review Project database housed at Colorado State University. This included national longitudinal studies focused on postschool employment outcomes, national policy reports and commissioned studies, and special topic reports focused on state vocational rehabilitation agency policies and practices impacting transition-age youth with disabilities. The literature review found that the knowledge base about successful transition of students with disabilities from secondary school to postsecondary environments has grown considerably over the past two decades. Available research confirms the value of well designed, well-coordinated transition activities involving VR, education, and other adult service agencies to support transition success, while also clearly documenting the constant need for further improvement in transition services and supports nationally.

Based on the literature review, several broad categories of findings were identified; these have been incorporated in the Draft Study Plan, serving as the basis for the survey development. These categories include:

Nature, attributes, and impact of interagency agreements on transition outcomes – emphasizing distinctions between state and local interagency agreements; content or focus of the interagency agreements; effectiveness/impact of such agreements; and how such agreements are administered, monitored, and enforced.
Role and involvement of vocational rehabilitation personnel in transition planning – emphasizing the challenges and barriers relating to VR personnel participation in a students’ IEP/transition plans; sharing information with students, family members, and educators on VR eligibility and service information; value of VR participation in IEP/transition planning; criteria for when (at what age, grade, or circumstance) VR should participate in the development of IEP/transition plans; and extent of use of school/student assessment data in making VR eligibility determinations (including the presumptive eligibility of students receiving SSI/SSDI benefits).
Availability of personnel trained in transition services and support – emphasizing the nature of training and preparedness of VR personnel to participate in the IEP/transition planning of transition-age youth with disabilities; extent of continuing education and professional development programs that support VR personnel participation in IEP/transition planning meetings; examination of the relative importance of cross-training between VR and special education personnel; and use and value of collaboratively/jointly-funded positions between vocational rehabilitation, special education, and other agencies in coordinating and delivering transition services for youth with disabilities.
Funding arrangements supporting transition – emphasizing the existence and value of jointly-funded positions in schools and adult service agencies; examining cost-sharing and resource-pooling options and their value; and examining state VR agency policies and practices concerning when (at what age, grade, or circumstance) case-service funds are dedicated to transition-age youth prior to leaving school.
Factors associated with transition success – emphasizing the types of school-based academic and non-academic (work-based learning, career and technical education, service learning, others) courses and services are available to prepare transition-age youth for postsecondary education and employment following school completion; and the relative value and importance of these school-based services in achieving future employment outcomes.
Availability of appropriate programs in secondary schools – emphasizing the importance of gaining an understanding of the current priority placed on academic development (as now required by the No Child Left Behind Act) and the impact on secondary school programs; and the need to identify and examine educational and VR programs for transition students which balance vocational and adult living skills with academic preparation.

2. Subtask 3.2, Interviews with Selected Federal Agency Staff

Interviews with seven senior federal officials from OSERS, RSA, OSEP, the Office of the Under Secretary, and the Social Security Administration were conducted. The following are areas of topical/thematic consensus among the respondents:

Identification of all transition students eligible for VR services – emphasizing the need to identify Section 504 and other students with disabilities who do not have an IEP and are not currently being identified as potential VR clients; and the impact of state Order of Selection requirements on transition-age youth with disabilities.
Impact of interagency agreements on collaboration at the service delivery (local) level – emphasizing the role and impact of interagency agreements at the local level in achieving results; the need to understand quality (the impact of agreements on services and subsequent transition outcomes for students) and quantity (the number of states and local communities that have agreements in place); and the extent to which these local interagency agreements minimize or eliminate service duplication.
Influence of trained VR personnel at the state and local levels – emphasizing the impact of VR personnel specifically assigned to high schools; and examining co-funding of VR counselor positions in schools as a means of improving service planning and coordination.
Dilemma of academic v. vocational community participation in transition programs – emphasizing the impact of increasingly limited school-mediated access to vocational and community-based programs due to an increased emphasis on academic development under the No Child Left Behind Act.
VR costs for transition programs and services – emphasizing the development and understanding of current costs for serving youth in transition and what these fiscal resources are supporting.

3. Subtask 3.3, Review of State and Federal Documents and Reports Associated with the Transition-Age Students with Disabilities

The final component of the preliminary activity was a review and analysis of state VR agency documents and the RSA 911 FY 2003 data. A total of 177 state-specific documents – 68 state Section 107 monitoring and technical assistance reports for FY 2003 and summary data from FY 2002; 72 state VR attachments 4.9(c)(2), describing coordination with education officials; and 37 state VR interagency agreements with the state education agency. Information obtained from this activity yielded the following themes of consistent importance: VR eligibility, timeliness of services, substantiality of services, nature of employment outcomes, transition services provided, interagency agreements, identification and referral procedures, and information concerning staffing patterns.

C. Subtask 4.1, Draft Study Plan

1. Draft Data Collection Instrument

The data collection instrument developed for the purposes of this procurement is based on the following considerations: (1) delineation of areas/domains that will function as an organizing framework for specific survey items; (2) development of an item pool based on the work concluded in relation to Subtasks 3.1-3.3 (and revised based on the pre-test); (3) measurement of selected demographic characteristics that provide descriptive information about individual respondents and state VR agencies; and (4) emphasis areas that address the interests of the procurement to identify barriers, best practices, and the extent to which VR policies and practices are addressing the needs of transition-age youth with disabilities. The survey is designed to answer questions that cannot be answered through a review of the extant data and is thus focused on obtaining new insights on state VR agency policies and practices. The following briefly describes each of these four design considerations that were addressed in relation to the draft survey (see Attachment A).

Areas/Domains to Assess. Several factors were taken into account in deciding specific “areas” or “domains” to include within the survey instrument. The functional purpose of creating areas/domains is to provide an organizing framework for displaying individual items within the survey. That is, the framework provides a means of creating a system of logic in presenting diverse items to potential respondents. Considerations in developing the areas/domains included: (1) basing the areas/domains on the findings derived from Subtasks 3.1-3.3, (2) discussing the relevance of the area/domain selections with the project study team, and (3) reviewing other relevant and recent national survey research instruments to examine similar area/domain structures. As identified in Attachment A, the survey instrument is organized into five major areas/domains. Again, subtasks 3.1-3.3 served as the primary basis for the selection and development of these specific areas/domains. The areas/domains include:
- State and local collaboration
- Identification and referral
- Role of VR personnel
- Provision of transition services
- Resources and funding
Survey Item Development/Selection. Based on the earlier preliminary activities, consultations with the study team, and reviews of other pertinent national surveys focused on transition-age youth with disabilities, a pool of items was initially developed for consideration. Individual items were then cast into one of the five areas/domains and reviewed by the study team for relevance, accuracy, and the extent to which each item addresses one or more of the procurement objectives. The items currently found in the draft survey instrument represent project staff consensus on the most meaningful and relevant items at this stage in the development of the instrument.

Attachment B provides a crosswalk showing each survey item and its relationship to each of the procurement’s objectives. The attachment illustrates how each procurement objective relates to the specific areas/domains and individual items found within the survey. The study team will continue to review and revise items and examine their relative relationship to each of the objectives of this procurement.

Respondent Characteristics. In order to adequately describe respondents, obtain basic context information about the 80 VR agencies, and facilitate analysis of data, specific demographic variables have been included in the draft survey instrument. These are fully noted in Attachment A. Demographic data of interest include: state VR agency administrative position within state government, number of state VR counselors, average caseload size, Order of Selection status, existence and/or nature of waiting lists maintained, specific caseloads dedicated to transition-age youth, relative percentage of state VR counselors serving transition-age youth, (estimate of transition-age youth served as a percentage of a state’s total VR client enrollment), and general information addressing the age at which VR services begin for transition-age youth.
Emphasis Areas. The draft survey is also organized to examine three primary emphasis areas: (1) range and variation in state VR policies and practices (extent and importance of items), (2) obtaining information on best practices, and (3) identification of barriers that limit or inhibit the adoption of best practices at the state and local levels. Varied scaling procedures are used to elicit responses to individual items. That is, a range of closed-ended responses requests to “check all that apply,” “rate the relative importance of items,” and “rank or order items according to your individual perceptions of importance.” In addition, several open-ended questions further probe and elicit responses and information that might not be addressed in the current survey. The latter will be used to provide clarification, extended comments, and other observations, as necessary to supplement the overall data collection process.

2. Data Collection Procedures

Respondents include all 80 state VR agencies situated throughout the 50 states, U.S. Territories and Pacific Rim entities, and the District of Columbia. The survey will be distributed to state VR directors for completion directly or reassigned by the state director to an informed state agency staff member. That is, we envision that state directors may delegate the task of completing the survey to other knowledgeable persons, including state VR agency transition specialists, personnel responsible for VR counselor assignments statewide, and others. Three options will be extended to respondents for completing the survey. Choices include: completing an online internet survey, completing a printed copy of the survey and returning it by mail, or requesting a phone interview with study team research staff. Individual responses will be routinely tracked, and email and phone reminders will be provided to individuals who have not completed the survey within a specified period of time. A 90% or better response rate is anticipated to this survey. Individual agency responses will be carefully reviewed by the study team for accuracy, completeness of responses, and other quality checks. When responses are found to be unclear and/or absent, we will request a phone consultation with the respondent to clarify their responses.

3. Plan for Pre-Testing the Instrument

The draft survey instrument will be pre-tested on-site with three state VR agencies. Several criteria will be used to select state pre-test sites. These will include (1) interest in and willingness to participate, (2) study team access to key state VR agency personnel, (3) study team knowledge of and familiarity with personnel from selected states, (4) state complexity and size (large v. small state), and (5) geographical representation (East, West, South, and North) of regions of the United States. An initial selection of three states was made by the study team and will be reviewed with COR for final approval. The specific states meeting these criteria include: Minnesota, Alabama, and Colorado. Again, final approval from COR will be obtained prior to contacting these individual state VR agencies.

At each of the state VR agency sites, three individuals (nine total) will respond to the survey. A study team member will be on-site to record feedback from the pre-test respondents. The study team member will spend up to three days on-site testing the instrument, with another day of travel allocated. The initial feedback from respondents will focus on the instrument’s overall technical adequacy. Questions will be addressed regarding: (1) the length of time required to complete the survey and its reasonableness, (2) clarity of the questions posed, and (3) comments on the length and ease of use of the instrument. Input from respondents will be entered into a database and coded accordingly to identify needed changes and modifications to the draft survey instrument. In addition, we will compute a Cronbach Alpha (i.e., KR-20) raw score reliability to determine reliability properties of the survey instrument and individual items.

In addition to these specific aspects regarding the technical adequacy of the instrument itself, we will evaluate issues regarding the social validity, utility, and importance of the questions addressed within the survey for the end user. We will elicit comments from the pre-test respondents concerning: (1) relevance of the survey to the objectives outlined in this procurement, and (2) likelihood of yielding useful information that state and local VR agencies can use in planning and in strategic efforts to improve programs and services. When all of the pre-test data have been collected, the study team will be convened for the purposes of reviewing responses, identifying needed modifications or changes, and identifying new items that need to be developed based upon state respondent input. These changes will constitute the basis for the revised draft study plan and for the subsequent feedback that will be obtained by members of the Panel of Experts, which will meet in Washington, D.C. in July 2005.

4. Data Analysis Plan

Descriptive statistics (percentages, means, and standard deviations) will be used to depict overall survey results. Such statistics are helpful in providing policy audiences with the “big picture” in the context of the magnitude and range of the data. These statistics will be presented in tables, charts, and graphs (e.g., bar, line, area, pie, box-plot, as appropriate) to provide a visual representation of variables in areas such as interagency agreements, patterns of VR counselor participation in IEP/transition planning meetings, depictions of the type and level of transition services provided, and other pertinent data and information. Procedures such as t-tests, ANOVA, chi-square analysis, and others may be used to examine the inter-relationship of variables and their statistical significance. Additional analyses might employ a correlational design where independent variables are measured – but not “manipulated,” as commonly associated with a controlled, experimental design.

One purpose of this study is to aid in the identification of sites associated with promising policies and practices for serving transition-age youth with disabilities. For example, multiple regression analysis may be used to assess the relationship between VR caseload and federal disability category, average age when VR services began, and whether the state has developed early intervention and interagency policies. Much of this later multivariate analysis would be more appropriately applied in Subtask 3.3 in the further analysis of the extant data supplement.

Perhaps one of the most important data analysis considerations is whether the end user will be able to readily use the resulting information in service delivery planning and decision-making. That is, descriptive and cross-tabular analysis of the survey data would likely yield optimal representation of findings for end-user purposes. Displaying the findings in highly graphic and visual formats would, we believe, expedite the further use of this information at the state and local levels. Presenting findings in this manner would make the data readily accessible and presentable in formats such as Power Point presentations, Briefs, and Executive Summaries that can be used by state and local VR agency personnel in sharing and disseminating the findings from the survey data.

Promising practices will be identified through analysis of relevant survey items. In each group of survey items about VR practices, respondents are asked to identify the frequency with which they employ each practice and the importance of each practice. Subsequently, respondents rank the three most important practices from the group of items. Promising practices will be identified through an analysis of responses to the importance and ranking items. Model states/territories will then be identified based on their frequency responses to these same items. Thus, if a practice is highly rated (high importance and ranking), those who say their state/territory “always or almost always” or “frequently” employs this practice will be considered as a model. Choice of promising practice models may be influenced by clustering of related promising practices and by the apparent applicability (or generalizability) of specific approaches to other states and territories.

ED-04-CO-0042 is a collaborative effort among TSG as the prime contractor and Colorado State University (Colorado Center for School and Work for Special Populations) and the University of Minnesota (Institute on Community Integration) as subcontractors.

The information summarized in this section of the Draft Study Plan is derived from the Report of Preliminary Activities – Subtasks 3.1-3.3, submitted by TSG in April 2005.

Activities | Interim Reports | Survey | Draft Final Report | Home

For additional information, please contact:

Michael Norman, Ed.D., Principal Researcher
The Study Group Inc.
209 Sir Walter Raleigh Drive
Kill Devil Hills, NC 27948
Phone: 252-441-2788
Fax: 252-441-9663
E-mail: Studygroup@aol.com

The VR Transition Study is managed by The Study Group Inc. in North Carolina, and its subcontractors (the Institute on Community Integration, University of Minnesota, and the Colorado Center for School and Work for Special Populations, Colorado State University). The study is funded by the Rehabilitation Services Administration (RSA) within the Office of Special Education and Rehabilitation Services (OSERS), U.S. Department of Education (ED). However, the contents of this site do not necessarily represent the policy of ED. You should not assume endorsement by the Federal government.

This Web site is housed at the Institute on Community Integration, University of Minnesota, and was last updated on October 11, 2007. The University of Minnesota is an equal opportunity educator and employer.