APPENDIX
A RESEARCH SUMMARY*
by Kathleen Fulton
A FRAMEWORK FOR CONSIDERING TECHNOLOGY'S EFFECTIVENESS
Policymakers, parents, and the general public all ask the same question: do educational technologies work? It's a fair question, and an important question, but the answer is a complex one. While it may be tempting to take stacks of research reports and say, "This many say 'technology works,' this many say it doesn't, and this many suggest the results are inconclusive"; this scorecard approach does not give policymakers the information they need for thoughtful analysis and strategic planning . Simple answers never do.
In considering research on technology, several caveats must be taken into account.
- the technology keeps changing; as hardware and software evolve, new educational opportunities appear
- educational technologies are used in classroom settings which rarely provide optimal conditions for their use
- it is inappropriate to take research results and generalize their findings across grade levels, students, subject matter, types of technologies, and applications
- the teacher is a key variable in technology implementation and effectiveness
- and, finally, technology's impact on teachers and their practice should be considered as important as student effects, for students move on but teachers remain to influence many generations of students.
In other words, to ask if technology works is almost the equivalent of saying "Do textbooks work?" Yes, some textbooks "work," in some conditions, with some teachers, with some students, but these same textbooks may not "work" in another educational context. Clearly the question of technology effectiveness requires us to be clear in what results we seek, how we measure success, and how we define effectiveness.
An Overview: Technology Can Play a Positive Role
Despite these caveats, and aware of the public's frustration with researchers' tendency to obfuscate with disclaimers when simple answers are sought, several recent national reports have dealt with the question of technology effectiveness. A RAND study for the U.S. Department of Education, which became the basis for the Department's influential report Getting America's Students Ready for the 21St Century: Meeting the Technology Literacy Challenge, made the following four points:
- numerous studies of a wide variety of specific applications of technology show improvements in student performance, student motivation, teacher satisfaction, and other important educational results
- there are examples of technology-rich schools that report significant improvements in student motivation, academic outcomes, and other results such as improved problem-solving or collaboration
- traditional ways of assessing the effectiveness of educational programs (e.g. test scores alone) are generally deficient for assessing the contributions of technology
- good implementation is crucial to the successful application of technology in education.
Breaking down the Research
This review gives an overview of research on the impacts of technology in three areas of focus: basic skills instruction, the development of higher-order thinking skills, and in support of what have been called information age skills. It also describes research on the effectiveness of distance education.
Technology's Role in Enhancing Basic Skills
We start with the area of basic skills, because this is most often the area of greatest immediate concern to policy makers. There is also the most research available in this area. However, it should be noted that most of these studies are based on computer-based instruction (CBI) or computer assisted instruction (CAI) built around the "drill and practice" models of developing isolated skills, often embodied in network-based systems known as integrated learning systems.. Furthermore, the majority of these studies consider software programs that were developed prior to 1990.
In comprehensive meta-analyses aggregating several hundred studies conducted by research teams at many different research centers, looking at a variety of computer uses with different populations, researchers presented the following conclusions:
- students usually learn more in classes in which they received computer based instruction
- students learned their lessons in less time with computer based instruction
- students liked their classes more when they received computer help in them
- students developed more positive attitudes toward computers when they received help from them in school.
An important caveat was noted:
- computers do not, however, have positive effect in every area in which they were studied
Another more recent meta-analysis, looking at 176 studies, including research from 1990 to 1995, found positive effects at all school levels, subject areas, and for both regular and special education students. The conclusion:
- the use of technology as a learning tool can make a measurable difference in student achievement, attitudes, and interactions with teachers and other students
The choice of words is important: "can make a measurable difference," not "will make a measurable difference."
As noted in another comprehensive research review: "The new technologies can contribute in several ways to better learning in various subject and to the development of various skills and attitudes. The nature and breadth of learning depends on previously acquired knowledge and on the type of the learning activities using technology."
As new skills are acquired in the motivating context of the computer-based instruction (which provides immediate feedback and private, personalized pacing allowing a student to progress at his or her own speed), these skills are reinforced in ways that fit the behaviorist model of teaching and learning that has been the foremost instructional approach in classrooms over the last century. Key elements in the motivating factors of technology are not just the bells and whistles of positive feedback, but the opportunities for self-pacing and self-regulation, although the authors of this study suggest that low- achieving students often required more structure.
Writing achievement is another key "basic skills" area where technology has provided positive benefits: students writing more and with greater proficiency. Because word-processing software can make writing less of a handwriting chore and more of a process of creation, teachers can help students focus more on the content of their text. When editing is a less onerous and more continuous part of the writing process, multiple drafts can be produced and shared for critiquing with peers. Technology does not diminish the need for instruction in basic skills of spelling, grammar, writing structure, tone, voice, and organization, but it can facilitate the structured application of these skills. When a sound model of teaching writing is used, researchers have found that students using word processing have demonstrated higher levels of achievement than those writing without the support of word-processing.
Technology's Role in Developing Higher Order Skills
While progress in basic skills via technology should not be discounted, some have expressed concern that these skills can be developed by other means that do not require the expense of technology. These educators maintain that technology may offer the most promise as a tool for advanced skill acquisition and for supporting new models of teaching and learning. The use of educational technology to help students develop higher order skills of problem-solving and the ability to access, organize, display, and communicate information are the components least likely to be measured on traditional pencil and paper standardized tests. As the tests become more sophisticated in assessing student performance on problem solving tasks, it will require close analysis of how technology use and higher level thinking are correlated.
Because these cognitive applications of technology are more difficult to evaluate, the research is this area is less extensive and data are more difficult to aggregate and compare. The situation is compounded by the fact that much of today's school improvement efforts challenge teachers in multiple ways, calling on them to transform their practice by requiring high standards for all their students, to adopt new curricula emphasizing higher-order skills, and to use constructivist, student-centered teaching methods. These learning environments place greater demands on teachers; applying technologies to this environment adds to the challenge. Assessing the impacts is even more complex.
A recent national study of teacher practice in technology- intensive classrooms, however, suggests that teachers' use of computers can play a role in shifting their instructional practice to a more "constructivist" approach that advances these school improvement goals. Becker (1998) defines practices that support this constructivist model:
- learning activities connected to real-world problems in which students have an interest
- tasks that require longer blocks of time to complete and deeper understanding of content
- curriculum that is thematic and interdisciplinary with greater emphasis on depth over breadth of coverage
- a teaching style emphasizing collaborative learning, student initiative, and student independence
- teachers learning and reflecting along with students, no longer seen as the only source of information.
Becker's research found that the teachers who report having changed their instructional practice from traditional fact transmission models to the knowledge construction model are the same teachers who have most thoroughly employed computers in their teaching. They are also the teachers who most often incorporate the Internet into their instruction. He suggests that computers encourage and even demand such practices which, in turn, change the pedagogical beliefs of teachers who use them.
Another large national study confirms this view of the interaction between technology use and changed teaching practice. Nine national case studies of technology-supported school change efforts, undertaken by SRI'S Center for Technology in Learning, found that "learning skills in the context of meaningful projects elicited greater student interest and understanding, as well as higher self imposed standards for quality." This research suggests why, among possible reasons, technology can be a catalyst for fundamental shifts in teaching:
- teachers see complex assignments as feasible
- technology appears to provide an entry point to content areas and inquiries that might otherwise be inaccessible until much later in an academic career
- technology can extend and enhance what students are able to produce, whether the task at hand is write a report or graphing data
- use of software tools enabled students to go farther than previous classes had without technology in a whole variety of curriculum areas
- teachers reported that students made greater use of outside information sources, showed more consideration of multiple perspectives, and improved understanding of audience needs
- perhaps as a result of these effects, teachers felt that use of technology enhances creativity, improves design skills and the ability to present information well, and promotes better oral communication skills
- the most common - in fact, nearly universal - teacher-reported effect on students was an increase in motivation
- most of the case study teachers described an increase in collaboration and more peer teaching among their students.
Another large-scale study of 500 students in 14 schools, comparing students with Internet access and those without, found similar outcomes. Student learning in this case was measured by outside evaluators assessing students' research projects. Overall, students with on-line access to the Internet produced "better projects than students without access, scoring higher in all nine learning criteria with statistically significant scores for 5 of the 9 learning measures." Students with on-line access demonstrated greater abilities to:
- bring together different points of view
- present a full picture (e.g., who, what , when, where, why, and how)
- effectively present their ideas and information
- effectively state an issue
- produce a complete project
One of the most comprehensive, long-term research studies has been the work conducted by the Apple Classrooms of Tomorrow (ACOT) research program. ACOT studied what happens in classrooms in which every teacher and student has access to a computer both in the classroom and at home. While far from an typical situation, this research gives a window into what occurs when technology is pervasive in the classroom and when teachers are trained and encouraged to use technology across the curriculum. Over 20 universities and research institutions conducted ACOT-supported research (ACOT had its own research thinktank independent of the company's sales and marketing division), spanning ten years. Summaries of these studies include the following findings:
- Test scores indicated that, despite time spent learning to use the technology, students were performing well-and some were clearly performing better.
- The students wrote more, more effectively, and with greater fluidity.
- Some classes finished whole units of study far more quickly than in past years.
- Access to technology actually encouraged them to collaborate more than in traditional classrooms.
- Technology was becoming more interesting to students as they began using it for creating and communicating.
- Students became socially aware and more confident.
- Students communicated effectively about complex processes when using computers.
- Students started using technology routinely and appropriately.
- Students became independent learners and self-starters.
- Students worked well collaboratively.
- Technology helped all students, including those at risk for failure and those with disabilities.
The technology had benefits that went beyond its value as an instructional tool, for example, as an assessment tool to provide information on demand about students' progress and accomplishments. It also provided new ways for families to increase their involvement in their children's education, especially as computers were increasingly integrated in home activities.
Technology's Role in Producing Information Age Skills
How well does technology become a vehicle for students' developing the very skills that the technology itself requires, that is, in developing the technological fluency that will enable them to work and thrive in the information age? Because technologies change so rapidly, students do not need to be trained to use a specific piece of hardware or software. Rather, research tells us that what is necessary is general understanding of technological applications, enthusiasm, and confidence to try new things, and the ability to "think with technology"--to know when technology can help solve a problem or complete a task, and when other means are more appropriate. It also means being able to use the tools of the technological age in ways that experts use them.
Here too, there is less hard research data to prove effectiveness, but promising examples. For example, in Project ICONS, an International Communications and Negotiation Simulations, pioneered at the University of Maryland, high school students take on the roles of decision-makers and negotiators on issues such as human rights, nuclear proliferation, international debt, or conflicts in the Middle East or other regional conflict areas. Telecommunications link them with teams in other countries around the world. Research suggests they learn not just content but also new skills in technologically supported negotiation, collaboration, and communications--skills increasingly necessary to conduct business or diplomacy in the shrinking 21st century global community.
In science projects like Global Lab, students don't just study science, they do science, focusing on a study site in their local community, using technological tools for collecting, analyzing, and sharing environmental data worldwide. In activities like these, students are learning to become facile with technology, but in the context of learning the skills, content, rules, ethos, and behaviors of the discipline. Technological resources give them authentic contexts, tools, and collaborative opportunities to work as historians, scientists, economists, scholars, entrepreneurs, and even politicians! As they develop skills of "telecollaboration," they are using what is one of the most sophisticated deployments of classroom telecommunication.
Technology for Distance Learning
Distance learning, like technology, is a term that is broadly used. We use it here to refer to the use of technologies to connect learners with teachers and learning resources located at another physical location. It can involve a range of video , audio, and text-based technologies with varying levels of one or two-way interactivity . It can consist of a whole course packaged and delivered from one school to another, or pieces of courses or information sharing between various experts and learners and between learners. The most common model is a course taught "live" at one site and delivered to learners at multiple distant sites.
How effective is distance learning? One of the most comprehensive analyses of distance learning research makes the following statement:
"Comparing the achievement of learners (as measured by grades, test scores, retention, job performance) who are taught at a distance and those taught in face-to-face classes is a line of research going back more than 50 years. The usual finding in these comparison studies is that there are no significant differences between learning in the two different environments."
While many studies have looked at distance learning effectiveness in training and professional development, results are also positive when reviewing K-12 applications, especially in secondary schools. For example, at the high school level, Martin and Rainy (1993) compared the results of a course in anatomy and physiology taught to seven conventional classes with the results of teaching the same course to seven classes by video-conference. While there were no differences in the students' pre-test scores, there were significant differences in the post-test scores, in favor of the distance learners.
Much of the research is no longer asking if distance education courses can be as effective as conventional classroom instruction, but rather, who learns best in distance learning settings and why. Whether the students learn or not depends less on the medium than on the characteristics of the learners, and on numerous other variables of program design, instruction, and administration, as well as content variables. Most studies suggest that the absence of face-to-face contact is not in itself detrimental to the learning process and what makes any course good or poor is a consequence of how well it is designed, delivered, and conducted, not whether the students are face-to-face or at a distance.
Moore and Kearsley (1996) list a number of key variables that determine the effectiveness of distance education courses:
- Number of students at learning site (individual, small group, large group)
- Length of class/course (hours, days, weeks, months)
- Reasons for students taking the class/course (required, personnel development, certification)
- Prior educational background of student (especially with self-study and/or distance learning)
- Nature of instructional strategies used (lecture, discussion/debate)
- Kind of learning involved
- Type of pacing
- Amount and type of interaction/learner feedback provided
- Role of tutors/site facilitators
- Preparation and experience of instructors and administrators
- Extent of learner support provided
Conclusion
While there is much research to be done to provide a better understanding of why and how technology benefits the educational process (both teaching and learning), the body of evidence to date suggests that new technologies provide powerful vehicles for educational improvement. It would be inappropriate to wait until tomorrow for documentation in all areas if this delay kept today's students from the opportunities technology can offer in a range of educational applications.
* The research review was conducted by Kathleen Fulton, Associate Director of the Center for Learning and Educational Technology, University of Maryland, College Park, for the Maryland Business Roundtable for Education Committee on Technology in Education 1999-2003. Reprint permission granted to Indiana Department of Education, December, 1998.