James McMillan, Editor

Online Assessment

There is little doubt that technology can change and improve education. With technology advances, particularly the Internet and World Wide Web, it is anticipated that many aspects of schooling, including assessment, will become better and will be done more efficiently, much like other areas of our lives such as entertainment and communication. On-line assessment, which is completed at individual computer terminals through a centralized source, holds particular promise, especially when coupled with demands for high-stakes testing. Indeed, computer-based assessment has existed for many years, without dramatic effects on schools. What is unique now is the on-line aspect of the technology. This coupling of technology and testing is noteworthy because policy makers believe that on-line assessment can be used to administer standardized tests to large numbers of students, and to score student performance as well as return test results quickly, at lower cost. Requiring students to use technology also helps them develop technology skills. However, a number of very important policy decisions need to be made concerning the implementation of on-line assessment. Will the assessments be different from paper-and-pencil versions? Will the assessments be fair to all students, including those with limited technology skills? How will students with special needs be affected? How will test security be maintained? How will teaching and learning be affected? What teacher training and in-service is needed? Will technology change how students are assessed?

What is “On-Line” Assessment?
Computer-based assessment has been around for at least twenty years. In this kind of testing, students answer questions that are included in software used in each computer. The assessment and scoring is self-contained in each computer. “On-line” assessment is delivery and scoring that is completed through the Internet. Test questions that are read or heard by students originate from a central source outside of the school, transmitted through a wire entering the school. Students transmit answers back to the central source through the same wire, where the responses are scored. Typically, students or teachers log onto the designated Web site, using established passwords. The system at the site verifies the password and link, then shows a screen with test directions. One to several questions are presented on a screen at one time, and students use a mouse or keys to signal their answer. Answers can be changed. Students navigate forward or back as needed to answer all the items, then submit answers for scoring. While currently on-line assessments are used primarily for large-scale testing, such as national certification exams and the Graduate Record Examination, there is strong evidence that on-line capabilities will transform most kinds of large-scale assessment, as well as classroom assessments.

Advantages of On-Line Assessment
On-line assessments have several advantages over paper-and-pencil tests:

• Reduces burden for schools in managing tests by eliminating the need for test booklets, answer sheets, and other conventional materials.
• Allows students to work at a faster pace.
• Is less costly to administer and score, after initial development expenses.
• Allows for customized reports with suggestions for further learning.
• Allows for advanced types of test items, such as simulations and web-based interfaces.
• Allows for multimedia use such as voice overlays, different languages, and animation.
• Increases student motivation.
• Provides flexibility in scheduling when and where assessments are taken, especially for retesting.
• Allows for immediate score feedback to students, parents, teachers, administrators, and state officials.
• Students can see rapid or instant feedback with explanations of correct answers.
• Allows for easy changes in answers.
• Allows for the efficient preparation of class, school, division, and state databases of student performance.
• Allows for virtually an unlimited number of different kinds of test items teachers can access for formative classroom assessment.
• Offers easy-to-use colors and interactive graphics.
• Allows for greater security because tests are stored by a server and are not left at the workstation or on disc (computer adaptive tests provide even greater security because each student takes a different set of items).
• Allows for testing students at enrollment for appropriate placement.

Disadvantages of On-Line Assessment
Disadvantages include:

• Student anxiety over using technology may interfere with and limit their performance.
• Unforeseen negative consequences.
• Students need training in working with on-line assessments.
• Minimum hardware capabilities must be met.
• Inconsistent connectivity.
• Difficulty ensuring that students are not cheating.
• End-user identification.
• Access to other Internet resources could be used by students.
• Maintaining test security.
• Maintaining session security in local schools.
• Advantaging students who have stronger computer skills.
• Loss of control of conditions in which students take the test.
• Tendency to emphasize multiple-choice item format.
• Appearance of test items presentation being affected by local computer capabilities.

Currently, there are two major perspectives concerning on-line assessment: 1) an efficiency perspective, and 2)using technology to change the nature of assessment. The first view (cheaper, faster) stresses the economy of on-line assessment and the ability of the technology to provide immediate results. The emphasis is on using technology to better deliver existing paper versions of tests, whether selected-response, such as multiple-choice, or constructed-response, like writing samples. This is not unimportant, especially in the context of large scale standards-based assessment, where on-line assessment can save money. Cost savings accrue because it is to the extent that on-line easier to make and revise test items, to deliver tests (eliminating the costs of printing, warehousing, shipping, and sorting tons of paper), to deliver results (eliminating further costs of printing and mailing), and to score constructed-response test items.

However, the cost savings are projections, not demonstrated reality. Established on-line testing programs such as the Graduate Record Examinations, the Graduate Management Admissions Test, and the Test of English as a Foreign Language, have found that the computerized versions of the assessments are actually more expensive than paper versions. As noted by an on-line testing expert from ETS, “for these programs, assessment by computer costs far more than assessment by paper.” (Bennett, 2001).

Providing immediate results is important because it allows testing to be completed later in the school year and helps teachers and parents make decisions about summer school and course placements for the next school year. For some kinds of testing, timing of the assessment is not necessarily an issue. In situations where the test covers one or two years of specifically targeted standards, such as end-of-course testing in Virginia’s mandated assessment program, being able to spend a few weeks more on instruction can impact student performance significantly.

A different perspective is provided by those who suggest that technology makes it possible to change the nature of the assessments. The design of current on-line assessments is based largely on a behavioral view of student learning in which test specifications are broad and general, and items are selected randomly to represent larger domains of knowledge and skills. These tests tend to measure isolated bits of knowledge. Computers make it possible to design assessments that are more consistent with recent research in measurement and student learning. In particular, computers allow new assessments to look like what cognitive science research has documented as the manner in which knowledge and understanding are represented in cognitive networks. The emphasis is on connected understanding and representation, rather than on isolated facts and concepts. More emphasis is also given to problem solving and other “higher-order” reasoning skills.

Another change will be concerned with the format of test items. With computers, it is possible to use simulations. These have already been used extensively in medicine. Students can meet patients, perform a physical examination, ask questions, listen to responses, request laboratory tests, make diagnoses, make decisions about treatment, and receive immediate feedback, all through a computer program. On a more simple basis, computers allow students to solve problems by asking questions. Computers allow for hearing and seeing video, expanding the skills that can be tested.

Computers are already being used to transmit writing samples for scoring. There are also several programs that will automatically score writing samples (Bennett, 1999), though it is complicated to design a system that approximates the perspective and judgment process used when a person reads and evaluates a writing sample. It is only a matter of time before other kinds of student constructed-response answers can be automatically scored. This will allow efficient and reliable scoring of performance assessments, essays, chart construction, and other skills that demonstrate what students can do. One testing advance that has been implemented with success is called Computerized Adaptive Testing (CAT). CAT is an approach to testing that customizes each test for each student, depending on the performance of the student. That is, the difficulty of subsequent items is determined by whether the student answered questions correctly. This approach allows the computer to select items with the best match, which results in a shorter but more reliable measure. CAT tests are all different, so test takers do not see the same items on retesting. This reduces the likelihood that scores will rise based solely on item familiarity.

Recent research has demonstrated that, for students who have gone to school with computers, paper-and-pencil constructed-response tests underestimated the performance in comparison with the same questions administered by computer (Russell & Haney, 2000; Russell & Plati, 2001).

• Research is mixed on whether greater access to or experience with computers impacts online assessment performance (Sutton, 1997).
• Research demonstrates that students who have above average typing speed score higher on constructed-response items than students with average or below average typing speed (Russell & Haney, 2000).
• Research on multiple-choice items suggests that mode of administration, paper or computer, does not significantly affect the test taker’s performance (Russell, 1999).
• In Oregon, the Northwest Evaluation Association electronically transfers tests via the Internet to computers of 300 school districts.
• The nation’s largest commercial test processor, NCS Pearson, scored more than 37 million tests in 2000, including 77 million constructed-response test items.
• Oregon estimates that on-line reading and mathematics assessments will eventually save over $2,000,000 each year (Neuburger, 2001).
• Research shows that, in general, most students have positive attitudes about using computers for assessment, though little research has been done with high-stakes testing (Sutton, 1997).
• Oregon will have over 400 schools participating in reading/language arts and mathematics on-line assessments in 2001-2002, in grades 3, 5, 8, and 10.
• Vantage Learning, Inc., estimates that it will give between 18-20 million on-line assessments in 2001.

Currently, six states have formalized initiatives to use an on-line approach in statewide testing - Virginia, Oregon, Georgia, Washington, Maryland, and Florida - and each is about at the same point in development. In Virginia, on-line assessment is being planned as a part of a broader project, the Web-based Standards of Learning Technology Initiative. The goal of the Virginia initiative is to improve student achievement through the use of statewide Web-based computer resources. Funding has been targeted to high schools to obtain the following:

• A ratio of one computer for every five students.
• Internet-ready local area network capability in every school.
• High-speed, high-bandwidth capabilities for instructional, remedial, and testing needs.

On-line testing, then, is part of an overall strategy to enhance the use of technology for instructional purposes. However, the emphasis is now focused on online assessments, with the goal that high school SOL on-line testing will be implemented in 2003. On-line assessments will then be developed for middle and elementary schools. During the winter, 2001, vendors from three companies demonstrated on-line assessment systems. The state will select one or more of these companies to develop complete on-line assessments for high school end-of-course tests.

In Virginia, as well as in other states, on-line assessment is being implemented primarily for efficiency (one exception is Oregon, where computerized adapted tests are being developed). Paper tests are simply put in an electronic format, with no changes, and distributed through the Internet. There is no emphasis on using the capabilities of the Internet to create new kinds of assessments, and it may be some time before this occurs, given the cost and complexity of getting current forms of existing tests on-line. Since state on-line assessments have yet to be administered, issues arising from implementation are more speculative than actual. From the experience of other types of on-line assessments, such as the GRE, there are several issues that can be anticipated. Some of these are concerned with traditional assessment technical standards, others identified because of the use of technology.

From the standpoint of testing standards, three issues must be addressed - validity, reliability, and fairness. Validity is concerned with the extent to which inferences from the test scores are appropriate and reasonable. In general, on-line assessments are subject to the same validity requirements as paper assessments. Since the major difference between on-line and paper assessments, especially high-stakes ones, is medium of reading the same items and response mode with the same alternatives, some kinds of evidence for validity of the on-line tests are needed. Evidence based on test content, such as what is used for the Virginia SOL, is adequate if indicated for the paper versions. When inferences are made about student placement, however, additional validity studies are needed for the on-line assessments. For example, it would be important to verify that a student’s level of computing and keyboarding skills does not affect their performance. It would also be desirable to include research on unintended consequences of on-line assessment, such as effects on use of the computer by students and teachers, or whether computer-based tests that mimic the high-stakes ones are used in the classroom.

Reliability refers to the degree of error in test scores. Because reliability is concerned with the scores that result from an assessment, and not the assessment itself, an on-line format needs to be researched to establish the degree of error that is likely in both individual and group scores. According to the recent Standards for Educational and Psychological Testing (1999), “When significant variations are permitted in test administration procedures, separate reliabilit analyses should be provided for scores produced under each major variation” (p. 36). This is especially important because the different format may contribute a new source of error. This degree of error would need to be considered when classifying students on the basis of their scores. If students are given the option of using on-line or paper assessments, research is needed to make sure scores are interchangeable.

One of the most important considerations with on-line assessment is making sure that technology use skill does not influence scores. That is, the medium of the test should not benefit or disadvantage any student. This is accomplished by using very simple commands and procedures, giving students easily understood directions, and giving students practice with responding to questions. Even with these safeguards, however, it is hard to believe that students with few or no computer skills can respond to the questions as efficiently and accurately as students who do have such skills. This is essentially a fairness issue. In practice, it is important to conduct research so that some students are not unfairly penalized because of the need to use the computer. Since level of computer awareness and skill are related to socioeconomic status, care must be taken to ensure that low socioeconomic students are not unfairly penalized by their lack of computer skills.

Fairness could also be an issue with the type of hardware used. On-line assessments should be developed and implemented so that they are not influenced by the type, power, or sophistication of the hardware. The technical requirements of implementing on-line assessments are considerable and costly. Designing a statewide system for hundreds of schools with differing technology capabilities is complicated, and bringing all schools up to needed levels of technology is time-consuming. Inevitably, there will be disruptions, additional development costs, unforeseen difficulties, and barriers. This suggests that policymakers must be patient and be willing to be flexible.

A major concern in implementing on-line assessment is security. Security is complicated because, along with all the standard security issues with high-stakes tests, such as making sure tests are not copied, and student cheating, additional factors must be considered. Since test items are stored by a central server, with special programming, it is very difficult to copy the items either to paper or to a hard drive or disc. The tests disable functions like print, print screen, e-mail, copy, paste, and other commands that could be used to breach security. Elaborate password control and “firewalls” also enhance security by limiting access and preventing what is called “footprinting.”

Footprinting is used by hackers to create a profile or map of the system. However, there is still the possibility that someone will be able to access on-line assessments before or after the testing window to know specific questions. Consequently, extraordinary efforts must be operational to be certain that there has not been unauthorized access to the assessments. In the end, there are probably no completely fool-proof security systems that will protect against every possible mechanism to compromise, corrupt, or destroy tests or data. A complete risk analysis is needed to catalog and devise ways to prevent corruption. It is also helpful to have a sound security policy in place.

Another major concern is making sure that all testing locations have adequate hardware and band width in communication links. There are obvious policy implications concerning hardware, as well as implications for staff development and the need for technical support to address technology difficulties in a timely fashion. What happens, for instance, if during a test power to the computers is lost? Will the students lose their work?

One of the exciting possibilities of on-line assessment is for assessments that teachers use continuously throughout the year in the classroom. These assessments are both formative (helping to diagnose weaknesses) and summative (end of chapter, unit, or semester). There are many companies that now advertise on-line assessment capabilities. While some of these have been tested in the field, the rhetoric about possibilities is stronger than the evidence of use by teachers. On-line assessments, theoretically, allow the teacher to:

• Easily comprise tests and make-up tests from a large bank of items.
• Allow for multiple forms of the same test as well as practice tests.
• Allow for the use of multimedia.
• Allow option branching.
• Provide immediate feedback to students with explanations of wrong answers.
• Provide immediate scoring services and item analyses.
• Allow for continuous assessment as an integral part of instruction, incorporating recent theories and principles of effective teaching and student learning and motivation.
• Enhance student self-assessment.

Allow for creative assessments such as electronic portfolios, case studies, and simulations. Some of what on-line assessment makes possible for the classroom teacher facilitates traditional models of teaching, making assessment more efficient. Other aspects open entirely new ways of teaching, though, despite “user friendly” claims, this requires a relatively high level of technology skills in addition to motivation to learn about another teaching tool. To the extent that teachers use on-line assessment to continuously monitor student progress, data are provided to modify instruction on an ongoing basis to better suit student needs. Some well-known companies that offer on-line assessments for teachers include PLATO Learning, Inc. (www.plato.com), Test Pilot (www.clearlearning.com), Ed Vision Performance Series: Computer Adaptive Internet Assessment for Schools (www.edperformance.com), and FasTEST Professional (www.assess.com).

On-line assessment will become a reality in all schools and classrooms. What is not known is when or how this technology can be used in ways that promote teaching and learning. Here is where policymakers can have significant influence, because funding, implementation strategies, and teacher training, will be major factors in determining whether the positive aspects of on-line assessment will be emphasized. The critical policy points that need to be addressed include the following:

• Do not allow technology to drive the nature of the assessment. Rather, let technology make possible more appropriate and effective ways of assessment consistent with cognitive research on learning and motivation, and new types of measurement (e.g., voice enhanced; graphics).
• Invest in research to provide systematic evidence on the psychometric requirements, as well as on the consequences of, high stakes on-line assessment.
• Minimize possible limitations and drawbacks to on-line assessment by providing:
  • adequate technical resources and hardware.
  • adequate on-site support in case assistance is needed.
  • sufficient practice in responding.
  • a failsafe system in case of equipment failure.
  • staff development.
  • a complete security policy and set of procedures.
• Provide training for teachers to use on-line capabilities in their classroom assessments.

Two additional considerations are important. The first is that children are more adaptable to echnology than are adults, so significant staff development is critical for teachers and administrators to keep up with students. Second, educators are already considerably behind other industries in the effective use of technology for all purposes, including assessment. With the speed of change in technology it is necessary for adequate support and resources to be invested if we hope to just stay even or not too far behind.

Click here for summary of recent Virginia Legislative history of Online/Web-Based Assessment.

Baker, E. L. (1998). Understanding educational quality: Where validity meets technology. The fifth annual William H. Angoff memorial lecture, Princeton, NJ: Educational Testing Service.

Bennett, R. E. (1999). Using new technology to improve assessment. Educational Measurement: Issues and Practice, 18(3), 5-12.

Bennett, R. E. (2001). How the Internet will help large-scale assessment reinvent itself. Education Policy Analysis Archives 9(5) (http://epaa.asu.edu.epaa/v9n5).

Kerka, S., & Wonacott, M. E. (2000). Assessing learners on-line. Columbus, OH: Center on Education and Training for Employment.

Behrens, J. Tl, Mislevy, R., & Bauer, M. (2001). Futrure prospects for assessment in the on-line world. Paper presented at the Annual Meeting of the American Educational Research Association, Seattle.

Faron, M., & Jennings, T. (2001). Measuring student knowledge in an on-line assessment system. Paper presented at the Annual Meeting of the American Educational Research Association, Seattle.

Lewis, A. (2001). New directions in student testing and technology. The CRESST Line. Winter. Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing.

Neuburger, W. (2001). On-line assessments: A state’s perspective. Paper presented at the 2001 Annual Meeting of the American Educational Research Association, Seattle.

Russell, M. (1999). Testing on computers: A Follow-up study comparing performance on computer and on paper. Education Policy Analysis Archives, 7(20). (http://epaa.asu.edu.eppa/v7n20).

Russell, M., & Haney, W. (2000). Bridging the gap between testing and technology in schools. Educational Policy Analysis Archives, 8(19) (http://epaa.asu.edu.epaa/v8n519).

Russell, M., & Plati, T. (2001). Effects of computer versus paper administration of a state-mandated writing assessment. Teachers College Record, January, ID# 10709. (www.tcrecord.org/Collection.asp?CollectionID=57).

Sephton, P. A beginner’s introduction to on-line web-based testing. (www.unb.ca/web/econ/sephton/testing).

Shermis, M. D., & Averitt, J. (2001). Security for web-based tests. Paper presented at the Annual Meeting of the American Educational Research Association, Seattle.

Standards for educational and psychological testing. (1999). Washington, DC: American Educational Research Association, American Psychological Association, National Council of Measurement in Education.

Sutton, R. E. (1997). Equity and high stakes testing: Implications for computerized testing. Equity & Excellence in Education, 30 (1), 5-15.

Organizations

Achieve, Inc., web site: www.Achieve.org

American College Testing (ACT): www.act.org

American Educational Research Association: www.aera.net

Assessment Systems Corporation (FastTEST Professional): www.assess.com

Association of Test Publishers: www.testpublishers.org.

The CAA Centre. Computer-Assisted Centre: www.caacentre.ac.uk.

ClearLearning: www.clearlearning.com

Center for the Study of Testing, Evaluation and Educational Policy: www.csteep.bc.edu.

Cisco Systems: www.cisco.com/warp/public/779/edu/

Computer Security Institute: www.gocsi.com.

The College Board: www.collegeboard.com

Education Commission of the States: www.ecs.org

Educational Testing Service (ETS): www.ets.org

FairTest (National Center for Fair & Open Testing): www.fairtest.org

Hacking Exposed: www.hackingexposed.com.

Harcourt Educational Measurement: www.hbem.com

National Center for Research on Evaluation, Standards, and Student Testing (CRESST): www.cse.ucla.edu

National Council on Measurement in Education: www.ncme.org

NCS Pearson: www.ncs.com/ncscorp/aoe/testing.htm

The University of Newcastle, Australia: www.newcastle.edu.au/department/so/assess.htm Vantage Technologies: www.intellimetric.com

Web-Based Education Commission: www.interact.hpcnet.org

Click cepi@vcu.edu to provide comments or additional information. Please indicate in an e-mail the copyright source and contact information for new inclusions.

Back to Top

Copyright © CEPI 2000
CEPI grants permission to reproduce this paper for noncommercial purposes if CEPI is credited.