The pressure on educators to be accountable for the learning outcomes of students is a key issue in current literature on authentic assessment. The following paper will critique the rubric for an authentic assessment developed by Hanlonen and colleagues (2003) for use with first year psychology students taking the subject, The History of Psychology.
A convenience sample of post-graduate and undergraduate students (n= 5) who are or have taken at least one semester of psychology, was recruited to collaborate with me on the critique and recommendation for changes to Halnonen’s rubric. Collaboration with students in the design of rubric has shown to be a critical way to enhance the cohesiveness of a group of people with common goals (Halnonen, Bosack, Clay & McCarthy, 2003; Newhouse-Maiden & de Jong, 2004; University of NSW, 2003). It is anticipated that this critique will enable me to better understand the complexities involved in designing rubric, valuing authentic assessment and recognising the critical need to involve students in the design process.
Student Requirements
The rubric provided by Halonen and colleagues lacked clear student requirements for undertaking the production of a newspaper publication for an audience of first year psychology students (see Appendix 1). To amend this, it is recommended that students at least have access to a computer with Microsoft Word. It is not necessary for students to use Word Publisher to create pages that resemble a newspaper format, Word will suffice. They will also need access to the internet, psychology databases for peer-reviewed articles and websites/blogs, a variety of newspapers to determine the layout desired for the final product, and access to a black and white printer. For disabled students an Eye-TV (from Access Ability Services) could be made available as this is a mobile item. To accommodate different learning styles, podcasts, video and transcripts of lecture and tutorial notes could be made available on Blackboard Learn.
Details of Conditions
Hanlonen gives few details of the conditions in which students will be assessed. The students are informed that some class time will be utilised for work on their project but that it is mostly an independent task requiring their own time in the library. This is ironic given that a purpose of the assessment is “to collaborate with others” which does take some skilled instruction to achieve (Cohen, 1994).
The students are also made aware by Halnonen that the final product will be displayed during a specified class time; however, it is not made clear if the display is similar to a presentation or simply the newspaper made available for class members to view. And whilst the student groups may be called on to discuss their assessment it is not known if this is as a formal or informal presentation. Thus, the conditions were deemed to be ambiguous and confusing.
To rectify this, it is suggested that the research and preparation for publication take place within tutorial times to enable students to have access to a tutor to guide them in team collaboration skills if they choose to seek them. As the work-life-study balance was determined to be a major stress for most group work projects, and making time available in class would enable different lifestyle circumstances to be less likely to impinge on accessing the group.
The final psychology newspaper publication will be made available to university students of all disciplines and community members at the cost of $1 which will be donated to a local charity to be negotiated by students prior to publication.
Instructions to the Examiner
The Halnonen example had no explicit instructions for the examiner. It is suggested that in the first two tutorials of term the rubric to be used is addressed with the class. A discussion will be held in which the students will contribute to critiquing and refining the rubric suggested here. This is an opportunity for the examiner to explain to students how the syllabus concepts of scientific inquiry in psychology are related to the assessment (Moskal & Leydens, 2000). In this way students are encouraged to be active learners.
Furthermore, the assessment strategy is to be divided into two distinct parts; a newspaper publication (completed as a group) and a critical reflection essay (completed as an individual). The written products are to be completed in the format described in the rubric (see Appendices 2 & 3). Students are to be given access to examples of psychology newspapers produced in the real world. Time in class could be used for students to investigate different forms of online and offline psychology newspapers with the examiner indicating strengths and weaknesses of the articles. This will enhance student self-efficacy and give them a better idea of how to monitor their own performance (Gulikers, Bastiaens, & Kirschner, 2004; Halnonen et al., 2003).
Halnonen also requires students to take part in a self-assessment of their performance in the group project. However, it is not evident that students will be exposed to critical reflection techniques and theories in class to enable them to self-assess effectively. Thus, the topic of critical reflection needs to be a part of the syllabus for the subject and tutorial time made available for students to discuss concepts and processes of critical reflection.
It is required by Halnonen that students receive peer feedback from group members as to perceived levels of collaboration within the group. This method of assessment was determined to be unfair due to the high subjectivity of the task and the numerous personal accounts given by the critique group of poor intra-group processes when left unsupervised. Instead it is recommended that peer reviews be summarised and analysed by students within their critical reflection essay. In this way, the psychology students can be better compare their judgements of goals met and practice the written communication to others of what they have learned from their experiences and how it is linked to theories they are learning about in class (Biggs, 2001).
Justification for the Assessment Strategy
Authenticity. An authentic assessment provides alternative measures of student performance, drawing on real-world tasks so that students actively take part in meaningful learning activities (Palomba & Banta, 1999). Assessment that is made relevant to the lives and career goals of students provides for evaluation of effective outcomes. Halnonen and colleagues state that the newspaper product is a real-life example which can be linked to student experiences to increase the likelihood that they will conceptualise and apply practical examples of problem solving and scientific thinking in the context of psychology. However, many aspects of the assessment could be considered to be irrelevant.
For example, are the students doing a psychology and journalism course or simply an undergraduate degree in psychology? The language, argument and style of communication required of an undergraduate require APA formatting and academic language. And although practicing psychologists may be asked to create products for lay persons, the assignment was to be delivered to first year psychology students by first year psychology students. Overall, the production of a newspaper article that, “Expresses events in language appropriate for a newspaper audience,” is unrealistic.
However, the critique group did feel that producing an academic-style psychology newspaper for an audience interested in psychology would be relevant, as it would mirror expectations in the workforce to produce newsletters for staff, clients or other academics.
Validity. The degree to which an instrument measures what it sets out to measure is its validity. One general form of validity used for an assessment instrument is content validity (Moskal & Leydens, 2000). Content validity draws on evidence which shows the extent to which an assessment instrument demonstrates the student’s knowledge of the content. The rubric used by Halnonen has very low content validity. There is only one content question, “Accurate identification of events within the assigned year”. As was pointed out within the critique group, “other newsworthy events” are irrelevant outside of the context of psychology for the undergraduate. Instead, students could be encouraged to focus on socio-political events that would inform the reader, and themselves, about how the psychology theories of the assigned year were embedded in the culture and thinking of the time.
Reliability. is the consistency of scores on an assessment, so that a student can reasonably expect to receive the same score regardless of the time, place or marker of the assessment (Moskal & Leydens, 2000). Inter-rater reliability is a common measure of assessment reliability, and a well-designed rubric enables students to be marked on an unbiased judgement, as different markers would give similar marks. Halnonen did not use well defined categories at all to clarify the scoring rubric of the assessment and so it has low reliability. There are no instructions provided for the examiner to determine what marks to give; are they to make a check mark or a score out of 10 for example?
It was determined within the critique group that the lack of well-defined categories meant that it was highly likely that different examiners would give different marks. It was suggested that the final rubric criteria that I decided upon could be tested and then tested again to determine reliability. However, for the current assignment this longitudinal design was not possible.
Another reliability issue is that of the appropriateness of the scoring rubric to the population of students it is given. Given that Halnonen and colleagues were scoring psychology undergraduates with questions pertaining to journalism, the assessment is viewed as being unsuitable for the population and thus unreliable. Making the changes as suggested in the modified rubric will rectify this as the items focus on conceptualisation, communication, problem-solving, group work and critical reflection within a psychology context.
Flexibility. The rubric set by Halnonen is not a flexible assessment in that the continuous assessment is attributed relative weights; 50% for the newspaper and class presentation; 30% for collaboration, and 20% for the self-assessment. This follows the norm in most universities for each form of assessment to be compulsory and given a fixed proportion of the overall assessment (Allsop, 2002). However, for the redesigned rubric, each stage of the assessment will be considered in isolation and potentially worth 100% of the assessment. Thus, a student will be awarded the higher grade from either the two pieces added together (each weighted at 50%), or from the one piece that has a higher grade. In this way, a student who did not perform well on one piece of assessment will not have their overall grade significantly affected and some of the stress of reaching academic goals can be removed (Allsop, 2002).
Fairness. Assessment is another critical factor that is used to minimise misinterpretation of results. The goal of fairness is to provide for students of both genders and all backgrounds an opportunity to do equally well in learning and demonstrating their learning. Bias is a factor that systematically affects an entire group of students as opposed to an individual student (Laboratory Network Program, 1993).
Halnonen’s rubric has bias in the context of the assessment, as some students may have done journalism subjects as part of their degree or in studies prior to their psychology degree. These students will know more about how to write in a journalistic manner and how to choose other newsworthy events. For this assessment task to be fair, its content, context and performance expectations need to be modified to reflect the knowledge, values and experiences that are equally familiar and relevant to all the students (Laboratory Network Program, 1993). Making the final product an APA style newspaper gives it an academic voice. Using a reflective piece of writing that draws on peer review is also a real-world expectation in the workforce for psychologists.
As the assessment is to evaluate understanding of new knowledge, ESL students will not be disadvantaged by potentially poor grammar as the tutor will be available to help them and other students to enhance their skills in academic English writing.
Task Efficacy
Halnonen et al. provide no details as the task efficacy of their rubric. However, as an ideal assessment is one that enhances the competence of students in their educational practices (Mentowski, 2000) it is concluded that Halnonen’s authentic assessment and rubric would have low task efficacy. The modified rubric has also not been tested; however, it is argued here that task efficacy will be high given the changes to content and context of the assessment and its rubric. The modified rubric is more likely to help students to make informed decisions about what they want to achieve as an undergraduate in psychology and how to go about achieving those goals because they will be involved in identifying their own levels of quality in real-world tasks.
For the present assignment, the task of collaborating with a group of students was very effective in that all could reflect on the goals of a rubric, authentic assessment for first year psychology and the value of student input in determining the criteria needed to develop knowledge, skills and competencies of a high quality suited to the workforce.
Conclusion
In conclusion, the design of an integrated and comprehensive rubric for an authentic assessment in first year psychology such as suggested here proved to be a complex though cooperative and collaborative adventure. Understanding and appreciating the processes involved in creating a rubric was made dramatically clear by the process of critiquing another’s rubric. The process of critique has provided for better articulation of the objectives and outcomes for rubric design for the entire critique group and promoted the desire to learn more about valid, reliable, fair and practical rubric designs for all forms of assessment. It is anticipated that enhanced inclusiveness of students in determining rubric and testing and retesting authentic assessments will contribute to avoidance of student comments, such as, “Grading and learning just don’t go together.”
References
Allsop, L. (2002). Flexibility in assessment. Retrieved April 17, 2010, from
http://www.informingscience.org/proceedings/IS2002Proceedings/papers/Allso068Flexi.pdf
Biggs, J. (2001). The reflective institution: Assuring and enhancing the quality of teaching and
learning. Higher Education, 41, 221-238.
Cohen, E. (1994). Designing Group work: Strategies for the Heterogeneous Classroom. Retrieved
April 14, 2010, from http://www.julieboyd.com.au/ILF/pages/members/cats/bkovervus /t_and_learn_pdfs/design_grpwork.pdf
Gulikers, J.T.M., Bastiaens, T.J., & Kirschner, P.A. (2004). Perceptions of authentic assessment:
Five dimensions of authenticity. Paper presented at the Second Biannual Joint Northumbria/EARLI SIG Assessment Conference, Bergen, Norway.
Halnonen, J.S., Bosack, T., Clay, S., & McCarthy, M. (2003). A rubric for learning and teaching,
and assessing scientific inquiry in psychology. Journal of Teaching of Psychology, 30(3),196-208.
Laboratory Network Program. (1993). A Tool Kit for Professional Developers: Alternative
Assessment. Portland, Oregon: Northwest Regional Educational Laboratory.
Newhouse-Maiden & de Jong, T. (2004). Assessment for learning: Some insights from collaboratively constructing a rubric with post graduate education students. Proceedings
of the 13th Annual Teaching and Learning Forum, Seeking Educational Excellence.
Perth: Murdoch University.
Mentowski, M. (2000). Learning that lasts: Integrating learning, development, and
performance in college and beyond. San Fransisco: Jossey-Bass.
Moskal, B.M. & Leydens, J.A. (2000). Scoring rubric development: Validity and reliability.
Practical Assessment, Research & Evaluation, 7(10).
Palomba, C.A. & Banta, T.W. (1999). Assessment essentials: Planning, implementing, and
improving assessment in higher education. San Francisco: Jossey-Bass.
The University of NSW (2003). Learn about rubrics. Retrieved online April 14, 2010, from
http://info.newcastle.edu.au/service/teaching-learning/projects/rubrictemplates/context.html
Appendix 1
Purpose of Assignment
To collaborate with others in the production of a newspaper
that summarizes the important events in psychology
along with other “current” events for a selected year. Students
sometimes struggle with understanding the significance
of historical events in psychology. This activity helps them
place the critical events of psychology in a broader cultural
and historical context as they refine their collaboration skills.
Directions
Join with a group of 4 or 5 students. Select a particular year
in psychology’s history and develop a newspaper page that
communicates the primary events of interest in psychology
during that year along with other events that happened during
the year. Some class time can be used to organize the tasks
of the group members, but the majority of the work will need
to take place independently in the library. Display final products
during a specified class period. Be prepared to discuss the
role of culture and history in speculating why the psychology
events occurred during your assigned year. Complete a
self-assessment on the quality of your work and give feedback
to other students based on the grading rubric.
Rubric for Newspaper Quality and Presentation:
Assessment of Product
Description skills: Observation.
___ Summarizes key ideas accurately
Conceptualization skills: Concept application.
___ Accurately identifies psychological events with assigned
year
Communication skills: Resource-gathering skills, conventional
expression.
___ Uses resources selectively and appropriately
___ Balances psychology events with other newsworthy
events
___ Expresses events in language appropriate for newspaper
audience
___ Produces aesthetically pleasing newspaper
Problem-solving skills: Bias detection and management.
___ Speculates about how cultural and historical influences
may have influenced psychology’s developments
Rubric for Collaboration: Assessment of Individual
Contribution to Group Process
Collaboration: Project completion skills, process management,
leadership, consensus-building skills.
___Completes complex project with others with minimal
direction
___ Contributes equitably to effectiveness of the group
through leadership, cooperation, and follow-through
on commitments
___ Evaluates quality of group process to improve work
___ Offers developmental feedback to group members
on quality of performance
Self-assessment: Self-regulation, self-reflection.
___ Accurately characterizes own contributions to
group process
___ Evaluates effectiveness of group plan in achieving
desired outcome
___ Describes lessons learned about working with
groups over time that may influence future collaboration
Grading Criteria
Three factors determine individual grades for this project:
the quality of newspaper page and its presentation during the
class, the quality of the collaborative effort, and the quality of
self-assessment. The relative weights of the available points
assigned for this project include 50% for the newspaper and
class presentation, 30% for collaboration, and 20% for the
self-assessment.
The newspaper pages themselves tend to be high-quality,
competitive productions. The presentation of the newspaper
page gives the group an opportunity to explain and defend
the choices made by the group.
(Halnonen et al., 2003, pp. 207-208)
