Handout module6

When to Use Essay or Objective Tests (Continued from previous page) Either essay or objective tests can be used to: ! measure almost any important educational achievement a written test can measure. ! test understanding and ability to apply principles. ! test ability to think critically. ! test ability to solve problems. The matching of learning objective expectations with certain item types provides a high degree of test validity: testing what is supposed to be tested Conventional wisdom accurately portrays shortanswer and essay examinations as the easiest to write and the most difficult to grade, particularly if they are graded well. Matching Learning Objectives with Test Items Instructions: Below are four test item categories labeled A, B, C, and D. Following these test item categories are sample learning objectives. On the line to the left of each learning objective, place the letter of the most appropriate test item category. A = Objective Test Item (multiple choice, true-false, matching) B = Performance Test Item C = Essay Test Item (extended response) D = Essay Test Item (short answer) ____1. Name the parts of the human skeleton Certain item types are better suited than others for measuring particular learning objectives. For example, learning objectives requiring the student to demonstrate or to show, may be better measured by performance test items, whereas objectives requiring the student to explain or to describe may be better measured by essay test items. ____2. Appraise a composition on the basis of its organization ____3. Demonstrate safe laboratory skills ____4. Cite four examples of satire that Twain uses in Huckleberry Finn ____5. Design a logo for a web page ____6. Describe the impact of a bull market ____7. Diagnose a physical ailment ____8. List important mental attributes necessary for an athlete To further illustrate this principle, several sample learning objectives and appropriate test items are provided on the right. Match the most suitable test item with each of the learning objectives. ____9. Categorize great American fiction writers ____10. Analyze the major causes of learning disabilities Answers: 1-A, 2-C, 3-B, 4-D, 5-B, 6-C, 7-B, 8-D, 9-A, 10-C 5 Planning the Test… By definition no test can be truly objective: existing as an object of fact, independent of the mind. In general, test items should… ! Assess achievement of instructional objectives ! Measure important aspects of the subject (concepts and conceptual relations) ! Accurately reflect the emphasis placed on important aspects of instruction ! Measure an appropriate level of student knowledge ! Vary in levels of difficulty Criteria for Establishing Technical Quality of a Test* 1. Cognitive Complexity Standard: The test questions will focus on appropriate intellectual activity ranging from simple recall of facts to problem solving, critical thinking, and reasoning. Cognitive complexity refers to the various levels of learning that can be tested. A good test reflects the goals of the instruction. If the instructor is mainly concerned with students memorizing facts, the test should ask for simple recall of material. If the instructor is trying to develop analytic skills, a test that asks for recall is inappropriate and will cause students to conclude that memorization is the instructor''s true goal. Refreshing the old bloom… During the 1948 convention of the American Psychological Association, a group of educational psychologists decided it would be useful to classify different levels of understanding that students can achieve in a course. In 1956, after extensive research on educational goals, the group published its findings in a book edited by Dr. Benjamin S. Bloom, a Harvard professor. Bloom’s Taxonomy of Educational Objectives lists six levels of intellectual understanding: ! Knowledge ! Application Implying that one type of question is automatically objective and the other necessarily subjective is a faulty assumption, since bias can occur with either type of test. Analysis Synthesis ! Evaluation ! ! Comprehension ! These levels of understanding assist in categorizing test questions. Teachers tend to ask questions in the knowledge category 80% to 90% of the time. These questions are not bad, but using them all the time is. Try to utilize higher order level of questions. These questions require much more brain power. (See the next page for a definition and sample question frames for each level of learning.) *Adapted from material developed by the National Center for Research on Evaluation, Standards, and Student Testing (CRESST). 6 1. Cognitive Complexity (continued) See pages 59 & 60 for Cognitive and Affective Domain Guides. Knowledge Recognizing and recalling information, including dates, events, persons, places; terms, definitions; facts, principles, theories; methods and procedures Sample Question Frames Who invented the…? What is meant by…? Where is the…? Comprehension Understanding the meaning of information, including restating (in own words); translating from one form to another; or interpreting, explaining, and summarizing. Sample Question Frames Restate in your own words…? Convert fractions into…? List three reasons for…? Application Applying general rules, methods, or principles to a new situation, including classifying something as a specific example of a general principle or using a formula to solve a problem. Sample Question Frames How is...an example of... ? How is...related to... ? Why is...significant? Analysis Identifying the organization and patterns within a system by identifying its component parts and the relationships among the components. Sample Question Frames What are the parts of... ? Classify ...according to... Outline/diagram... Synthesis Discovering/creating new connections, generalizations, patterns, or perspectives; combining ideas to form a new whole. Sample Question Frames What would you infer from... ? What ideas can you add to... ? How would you create a... ? Evaluation Using evidence and reasoned argument to judge how well a proposal would accomplish a particular purpose; resolving controversies or differences of opinion. 7 Sample Question Frames Do you agree…? How would you decide about... ? What priority would you give... ? Criteria for Establishing Technical Quality of a Test (continued) 2. Content Quality To Achieve Content Quality… Standard: The test questions will permit students to demonstrate their knowledge of challenging and important subject matter. The first activity in planning a test is to outline the actual course content that the test will cover. A convenient way of accomplishing this is to take a few minutes following each class to list on an index card the important concepts covered in class and in assigned reading for that day. These cards can then be used later as a source of test items. Some important questions need to be answered concerning the content quality of the test. What are the test specifications? What skills do they indicate will be tested. How many questions and how many areas will be covered? How many sections will there be? What formats will be used to test? If an instructor has focused on the War of 1812 in the majority of the class sessions and activities, this emphasis should be reflected in the test. A test that covers a much broader period will be regarded as unfair by the students, even if the instructor has told them that they are responsible for material that has not been discussed in class. Students go by instructors'' implicit values more than their stated ones. 3. Meaningfulness An even more conscientious approach would be to construct the test items themselves after each class. The advantage of either of these approaches is that the resulting test is likely to be a better representation of course activity. Standard: The test questions will be worth students’ time and students will recognize and understand their value. To Achieve Meaningfulness… "In my opinion, students should not be forced to guess what will be on a test, or psych-out the teacher to decide what to study. Research shows that the less able students are heavily penalized by a failure to realize what is required for a test. The more able students seem to sense what the teacher wants, but the students most in need of help are likely to flounder even more painfully if they must guess what to study. "The obvious solution to this problem is to give students specific study questions, then draw the test from the study questions. Sometimes this is criticized as teaching the test, as if having study questions in itself encourages a superficial approach. That may be true if there are very few study questions. However, if a teacher offers questions for all of the most important ideas in an assignment, then teaching the test is teaching the course." Russell A. Dewey, PhD Georgia Southern University, Statesboro, GA 8 It is very easy to write items which require only rote recall but are nonetheless difficult because they are taken from obscure passages (footnotes, for instance). Criteria for Establishing Technical Quality of a Test (continued) Preliminary findings by the National Center for Research on Evaluation, Standards, and Student Testing (CRESST): Results of Applying Language Evaluation Criteria to Standardized Content Test Items Math and science subsections: 67% percent of items had general vocabulary evaluated as uncommon or used in an atypical manner; 33% of items had syntactic structures evaluated as complex or atypical in their construction. Reading comprehension: Same as above for vocabulary and syntax; 50% of items also had discourse level demands. To reduce frustration for good students, avoid all of these and none of these and both a & b answers. These items are acceptable from a theoretical standpoint, but most prepared test-takers dislike them! As an example, the more subject matter a student knows, the easier it is to make arguments in favor of answers that the teacher might regard as wrong. True-false questions are the worst of all in this regard. Often the truth value of an isolated statement is quite debatable! It all depends on how it is interpreted, the definition of a key term, or the context. 4. Language Appropriateness Standard: The language demands will be clear and appropriate to the assessment tasks and to students. Test questions should reflect the language that is used in the classroom. Test items should be stated in simple, clear language, free of nonfunctional material and extraneous clues. Test items should also be free of race, ethnic, and sex bias. Beyond these two qualifications, students'' language backgrounds impact their performance on tests. The vocabulary (uncommon usage; nonliteral usage) and the syntax of the test (atypical parts of speech; complex structures) may create language barriers. Modifications of the test for students that are limited English proficient include: assessment in the native language; text changes in vocabulary; modification of linguistic complexity; addition of visual supports; use of glossaries in native language; use of glossaries in English; linguistic modification of test directions; and additional example items/tasks. 5. Transfer and Generalizability Standard: Successful performance on the test will allow valid generalizations about achievement to be made. Presentations, scenarios, projects and portfolios add dimensions to assessment that traditional testing cannot. Teachers can make valid generalizations about achievement more easily using authentic and performance assessments. These generalizations may involve instructional placement decisions, formative evaluation decisions and diagnostic decisions. Well constructed tests—whether they are objective or performance oriented—allow teachers to understand what needs to be taught next. Teachers are also able to monitor a student’s learning, while instruction is underway, and can change the instruction program as needed. 9 Criteria for Establishing Technical Quality of a Test (continued) 6. Fairness Five hundred secondary and postsecondary students were surveyed for suggestions on how an instructor could grade fairly and accurately. Standard: Student performance will be measured in a way that does not give advantage to factors irrelevant to school learning; scoring schemes will be similarly equitable. Here are a few basic rules of fairness: test questions should reflect the objectives of the unit; expectations should be clearly known by the students; each test item should present a clearly formulated task; one item should not aide in answering another; ample time for test completion should be allowed; and assignment of points should be determined before the test is administered. Grading constructively requires the instructor to provide feedback (written and/or oral) that helps the students to appreciate what they achieved and did not achieve by taking the test. This feedback could include the following: encouraging comments on a test or paper that convey respect for what the student attempted to accomplish; praise for what the student did accomplish and suggestions for improving performance. 7. Reliability Here are the top 10 responses. ! Consider grading based only on mastery of material and not on personalities or perceived effort. ! Do not over emphasize grades. Emphasize learning over grades. ! Keep students informed of their progress throughout the term. ! Clearly state grading policies and procedures in the syllabus and review them with the class during orientation. ! Avoid modifying policies during the term. ! Provide plenty of opportunities for assessment. This will avoid unnecessary pressure and allow for some mistakes. ! Provide some choice in format or topic when assigning work. ! Keep accurate records of grades. Record numerical grades, rather than letter grades, when possible. ! Consider allowing rewrites on papers. ! If many do poorly on an exam, schedule an exam for the following week to retest the class. Standard: Answers to test questions will be consistently trusted to represent what students know. The whole point of testing is to encourage learning. A good test is designed with items that are not easily guessed without proper studying. It is possible to construct all types of test questions which are not readily guessed and therefore require a student to comprehend basic factual material. Multiple choice questions are widely scorned as multiple guess questions. The solution to this problem is to design multiple choice items so that students who know the subject or material adequately are more likely to choose the correct alternative and students with less adequate knowledge are more likely to choose a wrong alternative. (On the next page are suggestions on how to defeat the TEST-WISE strategies of students who do not study.) 10