Implementation and Evaluation of Specifications Grading in a Video Production Course
...

by David Tamés, Assistant Teaching Professor, Department of Art + Design, Northeastern University

Presented at the 75th University Film and Video Association (UFVA) Conference: Empowering Spaces, July, 27–30, 2021

This paper describes the implementation of specifications grading (Nilson, 2015) in ARTD 2380 Video Basics at Northeastern University. The course engages students in an introductory exploration of the moving image as an art form while learning essential skills and techniques for creating video in various genres. Specifications grading is a method that empowers students and better situates them to manage their learning process by shifting the emphasis from subjective assessment to the achievement of learning outcomes. I compare student end-of-semester reflections from the Fall 2019 and Fall 2020 classes to evaluate the effectiveness of specifications grading from the students’ perspective.

Grading students, evaluating teachers, and assessing program outcomes has become synonymous with contemporary higher education. Students receive letter grades for practically every class they take while earning a college degree. However, the accuracy and consistency of grades as a measure of achieving learning outcomes have increasingly been called into question (Davidson, 2017). As a result, some educators are examining new approaches to evaluate student achievement in the classroom (Schinske & Tanner, 2014) to reflect individual strengths, abilities, and learning that are difficult to quantify in a single grade (Blum, 2017).

In light of these trends and the wide-ranging discussions among faculty and administrators across higher education during the spring semester of 2020 in response to the maelstrom of emergency remote teaching, I asked a question: Might this offer an opportunity to investigate, implement, and reflect on grading in my classroom? The upheaval of the pandemic and the urgent need to provide students with increased flexibility while maintaining rigor provided the catalyst I needed to re-center my grading practices around learning outcomes and empower my students to achieve those outcomes.

To implement specifications grading and establish better alignment between assessing student work and achievement of the learning outcomes would require that I flatten the assignment and project rubrics to a single dimension. Students either demonstrate mastery of the material, or the work is considered incomplete. By offering the students a mechanism to rework their projects, they can revise their work that does not meet the specifications of complete work. I hypothesized this would motivate students to master the fundamentals and better achieve the course learning outcomes. I made four changes to the course following best practices described in Nilson (2015) and Schinske and Tanner (2014): (1) create assignments and projects and evaluate them on a complete/incomplete basis; (2) provide clear criteria for achievement of learning outcomes (a.k.a. one-dimensional rubrics); (3) establish a mechanism for managing extension requests and resubmission of work that is not complete; and (4) scaffold projects with low-stakes, structured homework assignments with a reflection component to provide students with the scaffolding needed to complete their production projects. Hands-on workshops with well-defined deliverables, a core component of the course, remained intact.

While it was straightforward to implement specifications grading, I was unsure how to evaluate the effectiveness of this approach, particularly with regards to the student experience. I am sometimes puzzled by the difference between my perspective of teaching effectiveness and what students report in their TRACE end-of-semester course evaluations (see Note 1). Stephen Brookfield (2017) reminded me that “one of the hardest lessons to learn as a teacher is that the sincerity of your action has little or no correlation with students’ perception of your effectiveness... you can never be sure of the effect you’re having on students” (p. 2). This difference hit home. Significant differences between student and faculty expectations for a course are unavoidable (Dangel & Lindsay, 2014), but this no less alters the importance (and impact) of creating a learner-centered experience for our students. Brookfield’s notion of being a critically reflective teacher helped me connect my teaching and the reflective practice that is an integral part of my creative work as a documentary maker, so why not leverage the critically reflective approach to analyze the shift to specifications grading? As John Loughran (2002) points out, critical reflection “offers a window into the practice setting whereby the contradictory nature of the two views (students’ and teacher’s) creates a diversity of ways of seeing actions” (p. 40).

Because TRACE evaluations do not allow respondents to elaborate or provide a rich account of their experiences in the course, they do not afford the detailed descriptions needed to gain meaningful insights into the student experience. I imagined that it might be productive to take on an anthropologist’s stance in the classroom. What I needed was something akin to thick descriptions (Geertz, 1973) used in anthropology and qualitative research; however, it is time-consuming and challenging to achieve in the context of a single course. Is there a middle ground? Is there such a thing as “chunky data” that lies between the thick description of ethnographic practice and the data provided by TRACE? I turned to ethnographic reconnaissance to supply the insights I was seeking. Wolcott (2008) describes ethnographic reconnaissance as a lightweight version of the traditional ethnographic investigation; the reflexive methodology anthropologists use to express a subjective reality from the group’s point of view under study, derived from the ethnographer’s active participation in the group’s culture. The term reconnaissance implies a brief qualitative examination or survey over a short period with limited data, making “spot observations” instead of the more time-consuming participant observation practiced by ethnographers.

To establish a baseline of students’ perspectives, I began by reading the Fall 2019 and Spring 2020 final reflections along with the TRACE evaluations. One thing that stood out was that many students disliked doing the homework and vociferously complained that the assignments were not relevant to making videos. If I were going to implement specifications grading and require higher work standards to earn a pass on projects, I needed to provide the students with more scaffolding, and homework assignments had to be part of this. Still, given how much students dislike doing it, I realized I had to start from scratch during the summer and rework this class component before implementing specifications grading.

As a result of embracing a more critically reflective approach to my teaching, I was able to look at the homework, especially the readings, from my student’s perspective. I came to understand that I needed to provide a more structured path with immediate benefits. I accomplished this by tightly integrating the homework with the specific skills and knowledge students needed to make videos and discuss their work and their peers’ work in the context of critique sessions. I challenged my practitioner’s assumptions and blind spots and unpacked the process into a series of steps. My students were novices. There was only so much I could expect them to do with each project. I limited the viewing, readings, writing, and practice scope to the essential skills and knowledge required each week. Each of the new homework assignments starts with viewing assigned videos, then reading one or two chapters from the textbook (along with an occasional article). Students then reflect on the videos using terms and concepts from the readings, followed by producing two short video segments to reinforce their learning.

To help students manage their lives’ vicissitudes without a grade penalty, I implemented a coupon system. The system helped to emphasize learning outcomes while offering flexibility within limits (critical during the pandemic). Students were given three coupons that they could cash in for (1) transforming an unexcused absence to an excused absence; (2) obtaining a one-week extension on a project deadline as long as it was requested before the deadline; and (3) petitioning for the opportunity to rework a project that did not achieve complete status after it was handed in and critiqued in class. I believe this gives students the opportunity to make a choice for themselves between excellence and average performance.

How I’ve implemented specifications grading may seem rigid on the surface. However, it is quite flexible. While all projects and assignments are mandatory, students can be strategic and choose not to complete particular assignments. While this, in many circumstances, eliminates the possibility of an A as the final grade, it also places control in the student’s hands. The coupon system empowers students to take risks, and if those risks don’t pan out, they can rework the project. Looking over the work students completed and the self-assessment students shared in their final reflections, I’m convinced that the changes I implemented improved the student experience while creating better alignment between class activities and learning outcomes.

To compare the students’ perspectives before and after the course revision, I used thematic analysis (Braun & Clarke, 2006) to code and interpret the Fall 2019 and Fall 2020 final reflections. This approach provided a more thorough understanding of the reflections than merely reading through them. I chose this method because it is not constrained by the limitations inherent in participant responses, making it ideal for examining open-ended data like final reflections. This enabled me to identify patterns from the participants’ perspective. I coded 170 phrases from Fall 2019 reflections and 190 phrases from Fall 2020 reflections. I then grouped them into clusters, and from these, I created the themes. The following table shows the themes from six of the twelve questions I asked in the reflection prompt. Enrollment in the Fall 2019 class was 12 students and enrollment in Fall 2020 class was 15 students. The numbers in parenthesis indicate the number of responses in each theme.

tames-spec-grading-figure-1.png

While my study focused on the effectiveness of specifications grading in motivating students to learn more and produce better work, students had little to say about it directly. Still, they had a lot to say about the homework— an indirect implication of specifications grading. First, homework no longer appeared as a source of frustration or a hindrance to learning, and that was my goal. But I was surprised to see that some students included the homework among their proudest accomplishments. In retrospect, I attribute this to making the changes to the homework from a more student-centered stance. I also observed that students took significantly more pride in completing their project work. While it’s challenging to determine cause and effect relationships with thematic analysis, it does support a story consistent with shifting the focus to learning and achievement of learning outcomes. It is through the process of reworking a project that students learn how to produce better videos.

In this class of fifteen students, four students did not meet the specifications for a pass on their Final Project. One of these students chose not to rework the project, happy to pass the course but not concerned about earning an A. The other three students used their coupons for the opportunity to revise the final project. I held a learning conference via Zoom with two of them and exchanged emails with the third. I discussed the work’s strengths and weaknesses and the revisions required to meet the specifications with all three. All three made the revisions needed to earn a passing grade. I enjoyed how the dialogue was focused on completing the work and not engaging in a grade dispute. I believe this allows students to make a choice for themselves between excellence and average performance.

My interpretation is that implementing specifications grading teaches students how to achieve a measurable goal. With specifications grading, students are incentivized to evaluate the critique they receive from both their instructor and their peers and use that feedback to rework the project to meet the specifications. During prior semesters, I have observed that many students don’t read the detailed feedback I provide them once the work is complete. With specifications grading, if the student does not earn a complete grade, they must evaluate the critique from both their instructor and their peers and use that feedback to rework the project for it to meet the specifications. I believe this better aligns the work students do with the course’s learning objectives and reflects a more authentic evaluation of their work.

I wrote the specifications for each assignment and project in a relatively straightforward manner, and the critique from peers usually mirrored my critique. I entered into a dispute with only one student; however, the stakes were about iterating the work, not the final grade for the semester. This approach shifts the conversation to the quality of the work and learning outcomes, rather than arguing over subjective gradations between a B and an A. Ultimately, I see my role as a mentor helping students develop their video production skills, analytical abilities, and aesthetic sensibilities. When a project meets the specifications, I can focus my critique on what’s particularly compelling about the work.

A possible downside to the way I implemented specifications grading is increasing instructor workload. For every project that does not meet the specifications and a rework is requested and submitted, there will be additional time required to evaluate the revisions. What has helped in this course is that video project critiques are performed with the entire class, and therefore students are receiving feedback from their peers and myself. However, the time commitment to evaluating reworks is still an issue.

In a course implementing specifications grading, incomplete/complete becomes the grading dimension for assignments. Students learn that mediocre work represents unfinished work, and iteration is often required to achieve learning objectives fully. I discovered that specifications grading offers a viable alternative to traditional pedagogy and provides a framework for creating better alignment between coursework, assessment, and learning outcomes. A collateral benefit is that if students are completing courses, they are automatically meeting the course’s learning objectives, which, in turn, meets program objectives, vastly simplifying the process of program assessment.

Notes
...

  1. TRACE (Teacher Rating and Course Evaluation) is the faculty class evaluation instrument administered by the Office of the Registrar in collaboration with the Office of the Provost at Northeastern University. The survey includes both quantitative and qualitative responses including a student assigned “Overall Teaching Effectiveness” score on a five point Likert scale.

References
...

Blum, S. D. (2017). “I Love Learning; I Hate School”: An anthropology of college (reprint edition). Cornell University Press.

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101.

Brookfield, S. D. (2017). Becoming a critically reflective teacher (2nd ed.). Jossey-Bass.

Dangel, H., & Lindsay, P. (2014). What are our students (really) telling us. The Journal of Faculty Development, 28(2), 27-33.

Davidson, C. N. (2017). The new education: How to revolutionize the university to prepare students for a world in flux. Basic Books.

Geertz, C. (1973). The interpretation of cultures: Selected essays. Basic Books.

Loughran, J. J. (2002). Effective reflective practice: In search of meaning in learning about teaching. Journal of Teacher Education, 53, 33-43.

Nilson, L. B. (2015). Specifications grading: Restoring rigor, motivating students, and saving faculty time. Stylus Publishing.

Schinske, J., & Tanner, K. (2014). Teaching more by grading less (or differently). CBE Life Sciences Education, 13(2), 159-166.

Wolcott, H. F. (2008). Ethnography: A way of seeing (2nd ed.). AltaMira Press.

Acknowledgements
...

I would like to thank Nicholas C. Wilson, Gail Matthews-DeNatale, Bret Keeling, John Sangster, Anne L. van De Ven and Laurie Poklop for their insights, encouragement, and feedback. This work was completed during a Teaching and Learning Scholars Fellowship, 2019-2020, at the Center for Advancing Evidence Based Teaching, Northeastern University.