Title II, Part D of the ESEA (P.L. 107-110) has the explicit purpose “to enhance the ongoing professional development of teachers, principals, and administrators by providing constant access to training and updated research in teaching and learning through electronic means” as well as “to support the development and utilization of electronic networks and other innovative methods, such as distance learning, of delivering specialized or rigorous academic courses and curricula for students in areas that would not otherwise have access to such courses and curricula, particularly in geographically isolated areas.” Online learning is a growing part of educational offerings at every level of the education system, and policymakers and practitioners need to understand the conditions and practices associated with effective use of online learning.
Numerous studies have established the fact that students can learn at a distance—typically as well as students meeting with their teachers face-to-face—but much of the research examining the relative effectiveness of different online learning approaches and practices is riddled with poorly designed studies and a lack of objectivity. While good studies do of course exist, they are often found in literature related to training or adult populations and are rarely connected to K-12 populations. Moreover, the increasingly common practice of blended learning—in which online learning activities supplement face-to-face instruction—has not been separated from studies of learning taking place entirely online in a meta-analysis or other multi-application study of effects on learning. Despite the absence of a well-established evidence base, the use of online learning in K-12 education and teacher education is expanding astronomically. Administrators choosing to invest in online learning and the practitioners responsible for its implementation need guidance concerning the conditions and practices associated with effective use.
The proposed case studies will build on the contractor’s literature review by investigating approaches found to be differentially effective in prior research, in order to provide more detailed descriptions of the practices identified as associated with effectiveness in the literature review. The case studies will provide rich descriptions of the context of use and implementation practices in order to substantiate and expand upon findings from both a quantitative analysis of online student achievement and a literature review of teacher professional development research. This work will also be distinguished by inclusion of input from a wide range of stakeholders, including developers, administrators, instructors, and students.
The data collection activities to be conducted by this study will provide three types of products for the education community:
reader-friendly research syntheses and a final evaluation report,
recommendations for future research, and
tools and instruments for use by schools, districts, and states in evaluating online courses.
The research syntheses and evaluation report developed through this study will highlight effective, research-based practices that educational practitioners and policymakers can use to guide implementation of online offerings. It will include illustrative case studies to convey rich details associated with successful practices. The study will also produce checklists and other tools to support the training, implementation, and evaluation activities of districts and states. The protocols for virtual site visits and associated coding rubrics will be of particular use to districts and states, as many places are grappling with how to ascertain the quality of online offerings. Finally, this study will help guide future research. It will employ statistically rigorous methodology to determine objective student outcomes associated with online learning. These findings, supplemented by descriptive information about online learning implementation fidelity, conditions, and practices, will inform future investigations of online learning.
The contractor will use a variety of advanced information technologies to maximize the efficiency and completeness of the information gathered for this evaluation and to minimize the burden that the evaluation could potentially place on respondents. For example, members of the study team will collect demographic and other descriptive data about online implementations and the schools and districts that use them by accessing Websites and online databases. This practice will significantly reduce the amount of information that will need to be gathered through interviews.
During the data collection period, an e-mail address will be available to permit respondents to contact the contractor with questions or requests for assistance. The e-mail address will be printed on all the data collection instruments, along with the name and phone number of a member of the data collection team.
Finally, in order to gather data for online applications that occur entirely online and outside of traditional school settings, the contractor will collect observation data online. If the application is entirely online, there is no geographic place to conduct the observation. As the action takes place online, researchers will log in and conduct site visits in the virtual, online space. If instructors for a single online application are geographically dispersed, interviews will be conducted by telephone or over email, as it is inefficient to travel for a single interview. In all cases, the contractor will make objective observations of each online activity to make independent evaluations of the functionality of the technology and its use.
The case study work will be informed by a systematic review of the research literature, including a meta-analysis that will provide an empirical basis for many of the conditions and practices that will be examined in the case studies. K-12 case study sites will be selected in Florida, after the contractor performs analysis of FLVS and Florida state data to obtain information on student outcomes in comparable online and face-to-face courses while controlling for prior achievement. Teacher professional development case study sites will be identified based upon the findings that result from a comprehensive review of research on how teachers are prepared to teach online. This review will point the contractor toward specific professional development organizations, programs, or applications which provide this type of training.
We are also working to minimize burden by excluding potential case study schools and districts that are also part of other Department educational technology evaluations. Instrumentation will be coordinated across Department studies to prevent unnecessary duplication (e.g. no repeating questions for which sufficient data are already available).
We have not yet selected the final set of FLVS implementation sites. It is possible that one or more entities will be a small entity. Participation in the study will be voluntary, and the developer and associated school sites will be free to decline participation if doing so is determined to be of too high a burden. In addition, as mentioned above, the contractor will make every effort to gather available information on the Web and through electronic means in order to reduce the burden on all respondents, including those from small entities.
As already noted, online learning practices are prevalent and evolving quickly. In addition, current purposes expressed in Title II, Part D of ESEA promote the use of distance education for K-12 courses and teacher professional development activities. If information from this study is not collected, policymakers and educators will not have adequate information to inform the conditions and practices in which online implementations are most likely to be effective. This could result in inefficient use of resources for the design and implementation of online learning applications. The evaluation report developed through this study will highlight effective, research-based practices that educational practitioners and policymakers can use to guide implementation of online offerings. This type of guidance on how to make instructional decisions based on evidence helps accomplish the NCLB goal of providing technical assistance for state and local educational authorities.
None of the special circumstances listed apply to this data collection.
A notice about the study will be published in the Federal Register when this package is submitted to provide the opportunity for public comment. In addition, throughout the course of this study, the contractor will draw on the experience and expertise of a technical working group (TWG) that provides a diverse range of experience and perspectives, including representatives from the district and state levels, as well as researchers with expertise in relevant methodological and content areas. The members of this group and their affiliations are listed in Exhibit 3. The first meeting of the technical working group was held on January 18, 2007, the second was held on February 15, 2008, and the third is planned for spring 2009.
Exhibit 1. Technical Working Group Membership
Member |
Affiliation |
Bob Bernard |
Concordia University |
Barry Fishman |
University of Michigan |
Dexter Fletcher |
Institute for Defense Analysis |
Karen Johnson |
Minnesota Department of Education |
Susan Patrick |
North American Council for Online Learning |
Kurt Squire |
University of Wisconsin-Madison |
Bill Thomas |
Southern Regional Education Board |
Bob Tinker |
Concord Consortium |
Julie Young |
Florida Virtual School |
No payments or gifts will be provided to respondents.
The contractor has worked with the Institutional Review Board at SRI International to seek and receive approval of this study. All proposals for studies in which human subjects might be used are reviewed by the contractor’s Human Subjects Committee, appointed by the President and Chief Executive Officer of SRI. For consideration by the reviewing committee, proposals must include information on the nature of the research and its purpose; anticipated results; the subjects involved and any risks to subjects, including sources of substantial stress or discomfort; and the safeguards to be taken against any risks described.
The contractor recognizes the following minimum rights of every subject in the study: (1) the right to an accurate representation of the right to privacy, (2) the right to informed consent, and (3) the right to refuse participation at any point during the study. The draft forms in Appendix B show the school staff informed consent, the developer notification and informed consent, the teacher participant informed consent, the parent waiver of consent, and child assent. The information on confidentiality assurances is central to the contractor’s IRB process and will be conducted in full compliance with Department regulations. Data collection activities will also be conducted in compliance with The Privacy Act of 1974, P. L. 93-579, 5 USC 552 a; the "Buckley Amendment, " Family Educational and Privacy Act of 1974, 20 USC 1232 g; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants.
Responses to this data collection will be used only for research purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. The contractor will not provide information that identifies a subject or district to anyone outside the study team, except as required by law.
The contractor’s project staff has extensive experience collecting information and maintaining confidentiality, security, and integrity of interview and survey data. In accordance with the contractor’s institutional policies, confidentiality and data protection procedures will be in place. These standards and procedures for case study data are summarized below.
Project team members will be educated about the confidentiality assurances given to respondents and to the sensitive nature of materials and data to be handled. Each person assigned to the study will be cautioned not to discuss confidential data.
Respondents’ names, addresses, and confidentiality forms will be disassociated from the data as they are entered into the database and will be used for data collection purposes only. As information is gathered on individuals or sites, each will be assigned a unique identification number, which will be used for printout listings on which the data are displayed and analysis files. The unique identification number also will be used for data linkage. Data analysts will not be aware of any individual’s identity.
Participants will be informed of the purposes of the data collection and the uses that may be made of the data collected. All case study respondents will be asked to sign an informed consent form (see draft in Appendix B). Consent forms will be collected from site visitors and stored in secure file cabinets at the contractor’s office in Washington, DC. Any wavier of consent form returned will be carefully noted to remove the participants information from the data analysis.
Access to the database and case study notes will be limited to authorized project members only; no others will be authorized such access. Multilevel user codes will be used, and entry passwords will be changed frequently.
Access to hard copy documents collected from respondents will be strictly limited. Documents will be stored in secure file cabinets. All identifiable data (e.g., interview notes) will be shredded as soon as the need for this hard copy no longer exists.
Reports to the Department or any employee of the Department concerning case study activities will contain no individual or school or district identifiers. Participating schools will be acknowledged in the final report for their cooperation, but they will not be identified in the text of any report unless model practices are highlighted, in which case permission will be obtained from administrators or course developers before the information is included in reporting.
All case study participants will be assured of confidentiality to the extent possible in the initial invitation to participate in the study (see drafts of notification letters in Appendix C), and this assurance will be reiterated at the time data collection begins (i.e., when each respondent is presented with an informed consent form). While most of the information in the final report will be reported in aggregate form, as noted above, there may be instances where specific examples from the case study data will be utilized to illustrate “best practices”. In these instances, additional permission will be obtained from the administrator or course developer and the specific report text will be reviewed by the instructor or development staff prior to publication. This is an approach frequently used in developing technical assistance materials developed by the Department.
No questions of a sensitive nature will be included in the site visit protocols.
As described above, several types of data collection are intended including document analysis, interviews, observations, and researcher interaction with the applications themselves. In this section, we focus only on those parts of the data collection that add to respondent burden. The estimates in Exhibit 4 reflect the burden for the developer and site selection and notification of study participants, as well as the case study data collection activities. Course developers will be notified of selection and asked about their willingness to participate.1 Course developers will also be interviewed about the characteristics of their online implementation.2 Similarly, administrators of selected sites will be notified and asked to participate in the study. Once assent is secured, administrators will be interviewed about the online implementation as it occurs at their site. Instructors will also be interviewed, and student perspectives will be collected during focus groups.
K-12 site visits will be conducted with developers from FLVS and a large suburban school district. Up to ten FLVS implementation sites will be visited. The contractor will conduct teacher professional development site visits with up to 12 developers or providers. Therefore, up to 24 site visits will be conducted in total.
Exhibit 2. Estimated Burden for K-12 and TPD Site Selection and Notification
Type |
Total No. |
Hr. per participant |
Total number of hours |
Cost Per Hour |
Estimated burden |
K-12 Course Developers (notification) |
2 |
0.5
|
1 |
$40 |
$40 |
TPD Course Developers (notification) |
12 |
0.5
|
6 |
$40 |
$240 |
K-12 Course Developer (interviews – 3 per developer) |
6 |
1 |
6 |
$40 |
$240 |
TPD Course Developer (interviews-3 per developer) |
36 |
1 |
36 |
$40 |
$1440 |
K-12 Administrator (notification) |
10 |
0.5
|
5 |
$40 |
$200 |
K-12 Administration (interviews-2 per selected site) |
20 |
1.0
|
20 |
$40 |
$800 |
K-12 Online Instructors (interviews – 3 per selected site) |
30 |
1.0 |
30 |
$25 |
$750 |
K-12 Students (focus groups – 1 per site with average 8 students per group) |
80 |
1.0 |
80 |
$5 |
$400 |
Total |
196 |
6.25 |
184 |
$270 |
$4,010 |
There are no additional respondent costs associated with this data collection other than the burden estimate provided under Exhibit 4 above.
The annual costs to the federal government for this study, as specified under contract, are:
Fiscal Year 2007 $ 429,291
Fiscal Year 2008 $ 316,770
Fiscal Year 2009 $ 871,980
Fiscal Year 2010 $ 407,309
Total $ 2,025,350
This request is for a new information collection.
During the summer of 2009, the data collection team will complete analysis of case study data. These data will provide a more in-depth look at how online learning is implemented in two particular instances, as well as the conditions and practices associated with emerging promising practices in online implementations. The contractor will analyze instructional observations and developer, administrator, instructor, and student data. The contractor will code data for conditions, including characteristics of the learners and learning content, and for practices, including synchronicity, technology media and delivery, learning experience type, setting, and duration and intensity using formal protocols that have been developed with expert input from the TWG. Coding the observational data for conditions and practices will allow the contractor to categorize the online offering within the three dimensions of the original conceptual framework (learning experience type, synchronicity, replacement of face-to-face learning, and enhancement of face-to-face learning). Case study data will ultimately be incorporated into the project’s final evaluation report.
All data collection instruments will include the OMB expiration date.
No exceptions are requested.
1 The term “course” is used throughout this section for the sake of brevity. In actuality, the online learning offering under study could be an entire program or a supplemental resource rather than a single, stand-alone course.
2 As mentioned above, developer reports will be verified through independent assessment of application features and capabilities during both instructional observations and through independent interaction with the application by researchers.
| File Type | application/msword |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |