Carnegie Mellon Open Learning Initiative Evaluation Studies

The OLI project will conduct a rigorous evaluation plan aimed at describing the nature of student learning and documenting the process of development and implementation of the online courses. The evaluation process will substantiate the relative effectiveness of the courses, their underlying pedagogical rationality, the soundness of the assessment strategies and tools, and the features that make unique effective online learning. It will supply rich, usable information concerning the context of teaching and learning, the socio-cultural and institutional conditions that favor an adequate implementation of the courses, and the routes and detours that the design team experienced throughout the development of the courses. In this sense, the evaluation process will go beyond a mere validation of the courses’ effectiveness, and become the telling of an educational experience, for both professional and scientific audiences.

Types of Studies

For each of the online courses, there will be converging evaluation studies that vary in their analytical granularity, methodological approach, limitations, and targeted audience. We will design evaluations based on think aloud and verbal protocols, self-reports, log-file data, test scores, technical documents, institutional information, personal narratives and observational data. The variety of methods, experimental control, and grain sizes brings converging evidence to the resulting analyses.

The different evaluation studies can be sorted into seven categories, according to the goals they set to achieve and the information they provide:

1. Contextual studies
2. Effectiveness studies
3. Component studies
4. Learning studies
5. Assessment studies
6. Design studies
7. Dissemination studies

Contextual Studies

Context encodes the conditions under which students and faculty make sense of learning and teaching. A careful description of the context in which the online courses are delivered should prevent us from what has been referred to as the anonymity and invisibility of teachers and students in most educational studies. Thus, far from trivial, contextual studies provide information about the institutional, sociocultural, and curricular contexts of teaching and learning.

By institutional context we mean the nature of the institution where the courses are delivered (e.g., private, public, large universities, community colleges). The sociocultural context refers to the educational and cultural characteristics of the student population, as well as to engagement levels of the students and faculty that participate in the on-line courses. Finally, the curricular context refers to the place within the college/university curriculum where the course is located.

Effectiveness Studies

These studies aim at documenting the nature of the student competencies as a result of their transit through the online courses. At a basic level, effectiveness studies should provide evidence of “no harm”, that is, show equivalence or slight improvement of student performance on posttest measures relative to lecture formats.

A different kind of effectiveness studies concerns the student subject matter competence relative to disciplinary criteria. That is, are the students able to account for the concepts and principles that the online course conveys? Are they capable of using knowledge to tackle domain significant problems? How do they compare with tough competitors.

Component Studies

Each of the courses consists of a complex array of learning objects, ranging from instructional texts to virtual labs and discussion boards. Information regarding the functioning of course components will guide the design process, and can promote the export of learning objects to other environments. These studies neatly complement learning studies that trace student behavior with respect to the objects or environments in question.

Learning Studies

Learning studies are designed to show how and when learning takes place. Unlike the effectiveness studies, these studies unfold the occurrence of learning in its complex relationships with the task environment. Given the extent of the existing literature on effective learning, OLI will focus on learning processes associated with task environments or learning objects that are peculiar to the online courses, such as virtual labs, interactive examples, hypertext use, discussion boards, and case study analysis. This emphasis implies a move from the traditional question about the effectiveness of and the unique skills associated with online learning, to the more enlightening question about the advantages that online learning environments offer to learners and teachers.

Assessment Studies

The OLI courses currently include multi-choice questions as the privileged form of evaluation of student learning. With regard to this type of questions, assessment studies use standard psychometric techniques to estimate: (a) the difficulty of individual items, (b) the internal consistency of item pools, (c) and the equivalence of alternate test forms.

However, as other forms of evaluation are incorporated, we will assess both their instructional and assessment properties. Thus, for example, having the students go through short essay questions or case analyses may well be a learning event, in that students are likely to explain even in the absence of feedback.

Design Studies

Design aim at showing that the course design was rational, in that the courses were developed in the light of sound instructional goals, and with the assistance of a theory of learning guiding the decision made at every step of the way. Design studies help potential users understand the logic behind the courses, and make explicit the rationale that for designers is already shared knowledge. They provide information about the conceptual divide that is embodied in the course modules, about the development process, and about the practices that became a substantial part of the design and implementation processes.

Dissemination Studies

Dissemination studies provide information about the conditions under which the designed courses can actually be implemented in institutional environments other than the ones in which they were originally developed. A reflection on both course desirability and distance effectiveness should give us a typology of school fit.