TYPES OF DIRECT ASSESSMENT

4.8.3. Rubrics: Tools for Response and Assessment

Rubrics are tools teachers and students use to evaluate and classify writing, whether individual pieces or portfolios. They identify and articulate what is being evaluated in the writing, and offer "descriptors" to classify writing into certain categories (1-5, for instance, or A-F). Narrative rubrics and chart rubrics are the two most common forms.

4.8.4. TYPES OF DIRECT ASSESSMENT

This section is intended only as a brief summary of various types of direct assessment as background for subsequent discussions of reliability, validity, and other issues that at times are influenced by the type of assessment. More complete (and more precise) treatments of types of assessment described here, as well as other types, are given in a number of writings by members of the English teaching profession (see, for example, Cooper 1977; Lloyd-Jones 1977; Myers 1980; Odell 1981). The types of direct assessments commonly used may be classified with respect to task types and the method of evaluation used. At times the evaluation procedure is closely linked with the task, as in primary-trait scoring. Most scoring methods, however, can be applied to more than one specific task, though modifications may be necessary as the tasks vary.

A. Task Types : Task types are infinite in their variety, since they vary not only with the topic to be addressed but with the specific kind of prompt or stimulus used, the audience to be addressed, and the purpose intended. Prompts may be written, aural, or pictorial. The audience and purpose may be only implicit, as when a student writes something to be evaluated by his or her teacher or an anonymous teacher or group of teachers. A task may allow consultation of reference works, such as dictionaries, and time for revision, editing, and rewriting. Or, it may be a brief, impromptu task, which allows no consultation of reference works and no time for rewriting. Following are brief descriptions of some well-known types of writing tasks.

B. Letter : An examinee might be asked to write a letter of some type: to a friend, to the editor of a newspaper, to a potential employer, to a company complaining about a product or service, and so on.

C. Narrative:  An autobiographical account, a description of a vacation or other experience, or a historical description of some other type would all be narratives. These narratives could, of course, also be written in the form of a letter, and narratives can be either real or imaginary.

D. Descriptive:  Although a narrative is usually descriptive, the term implies the description of a series of events. A piece of writing may be simply the description of some object, how it looks, how it works, or some other aspect of it, or some other kind of description.

E. Argumentative:  In this type of task, the examinee is usually asked to take a position on some issue and argue persuasively for that position using evidence from his or her own personal experience or reading. It is probably the most common task type used because it requires the integration of several different writing skills. Sometimes this type of task is referred to as an "expository-argumentative" task.

F. Expressive:  Rather than argue persuasively, the task may be only to express one's opinion on some issue or event. While expository in nature, this kind of task is usually distinguished from a persuasive or argumentative exposition.

G. Role-Playing: One may be asked to assume a role in some situation and then to write something (such as a letter or a memorandum) for some specific purpose. Examples would include responding to an irate customer as a customer relations official, or writing a memorandum to a superior or a subordinate in an organization. For role-playing tasks, the audience and purpose are usually quite clear.

H. Precis or Abstract:  A real-life task of some importance is that of synthesizing a large body of information for transmittal to an audience different from that intended in the original piece. Scientists abstract complex scientific investigations for non specialists. Diplomats abstract current information about specific countries, at times originally written in other languages, for use by others. Lawyers synthesize case histories having legal precedents in making arguments. Therefore, a useful task is to ask students to read something and then to prepare a brief precis or abstract of it.

I. Diary Entry:  This could be similar to any of the preceding tasks, but the fact that it is written for personal use would probably change its tone.

J. Literary Analysis:  This is a common task used in literature courses and in the more difficult English examinations

K. Revision or Editing:  Any of the tasks above might be the subject of a task requiring revision or editing.

4.8.5 Evaluation Methods : Having obtained a response to one or more of the stimuli represented in the task types discussed in the preceding section, one can then usually choose among a number of different methods for evaluation of the response. As noted earlier, some evaluation methods are closely tied to the stimulus, namely, primary-trait methods. Thus, the task may predetermine the evaluation method. Among the several different approaches to evaluation, some are more widely used than others. The descriptions that follow, it should be cautioned, do not represent a consensus of opinion on the meaning of terms. Rather, they are an attempt to describe briefly methods about which there is often much disagreement.

1. Holistic Scoring:  According to Cooper (1977), in holistic scoring "the rater takes a piece of writing and either (1) matches it with another piece in a graded series of pieces, or (2) scores it for the prominence of certain features important to that kind of writing, or (3) assigns it a letter or number grade. The placing, scoring, or grading occurs quickly, impressionistically, after the rater has practiced the procedure with other raters." Holistic scoring is at times conducted using scoring guides, or rubrics. Some practitioners of holistic scoring distinguish it from impressionistic scoring, since the latter is viewed as a haphazard, noncontrolled, and unmonitored procedure. Holistic scoring is the most widely used evaluation procedure.

2. Focused Holistic Scoring:  This method is essentially the same as holistic scoring except that scores are produced for more than a single dimension of the writing sample being evaluated. For example, one might score for content and mechanics, or for some other specific aspects. The scoring might be done for each dimension after a single reading, or it might be done for each separately so as to minimize influences of one focus on the other. The number of focuses must of course be limited; otherwise, the procedure tends to be more like an analytical procedure. As in holistic scoring, no eounts or enumerations of any type are used. Scoring rubrics for each of the dimensions focused on, however, may be used.

3. Analytic Scoring:  This evaluation procedure is perhaps best exemplified by that associated with Diederich (1974). The Diederich procedure is based on a factor analysis of writing samples scored by experts representing several different academic disciplines. The factors derived were ideas, organization, wording, flavor. and mechanics. In some versions of the method, mechanics is further divided into usage, punctuation, spelling, and handwriting. Each factor is rated on a scale from 5 (high) to 1 (low), and two of the scales (ideas and organization) receive a double weighting. Thus it is possible to obtain a score as high as 50, or as low as 10. Other analytic procedures are described by Cooper (1977), Odell (1981), and Follman and Anderson (1967).

4. Atomistic Scoring: Somewhat akin to analytic scoring are methods in which detailed enumerations are made of quite a number of different features of a piece of writing. While certainly "analytic" in many senses, it is useful to distinguish atomistic scoring from analytic scoring, as described here, because it is very different with respect to the detail required. One example of an atomistic scoring procedure was described by Moss (1982). In this procedure, the total number of errors was counted in each of four categories: spelling, capitalization, punctuation, and expression. To develop a score from these counts, the total number of errors was divided by an index of paper length so as to avoid inappropriate penalties for writing more.

5. Primary-Trait Scoring:  Mullis (1980) explains that the rationale of primary-trait scoring "is that writing is done in terms of an audience and can be judged in view of its effects upon the audience." The primary, or most important, trait of a piece of writing will be the approach used by the writer to reach the audience intended. The primary trait of a set of directions, for example, "would be an unambiguous, sequential, and logical progression of instructions," according to Mullis. Another example given by Mullis is a piece of political campaign literature intended to persuade a reader to vote for a particular candidate. "A successful campaign paper will have certain persuasive traits that an unsuccessful one will not have, and these traits will differ from those necessary for a successful set of directions," Mullis notes. For any given task, the scoring directions must be prepared beforehand, and they are usable only with that specific task.

6. Syntactic Scoring:  Hunt (1977) has popularized a method of gauging syntactic maturity which is most often associated with the term, Tunit." AT-unit is defined by Hunt as a "single main clause plus whatever other subordinate clauses or non clauses are attached to, or embedded within, that one main clause." In other words a T-unit is a single main clause and whatever else goes with it. The T-unit is used, rather than the sentence, because it is empirically useful in describing the changes that occur in the syntax of writers as they mature.

7. Communicative Effectiveness:  In a sense similar in objectives to primary-trait scoring, this method of measuring the quality of prose is also concerned with the effects it has on an audience. But, operationally, the method is very different from primary-trait scoring. Hirsch and Harrington (1981) describe the theoretical basis for this new method and some of its advantages over traditional methods of scoring. The method is also similar in some ways to recent approaches being taken by cognitive psychologists, in which the theory and structures of reading comprehension research are applied to the analysis of text (see, for example, Bracewell et al. 1982; Bruce et al. 1982; Fredericksen 1983). Usually, an objective index of communicative effectiveness, such as reading speed or comprehension, is derived for the assessment. 8. Automated Scoring:  Another new method of evaluation that is of considerable interest is that done by computer. Frase et al. (1981) and MacDonald et al. (1982) describe a computer-based system developed at Bell Laboratories that is presently operational. A more sophisticated parsing system is under development by ffiM (see Heidorn et al. 1982). These methods will be discussed in more detail in a later section of this review.