Automatic Essay Evaluation with the Virtual Writing Tutor

It’s the week before the beginning of the semester and you are finalizing your course plan. How many writing assignments will you assign this semester? Will you assign one midterm essay and one final essay like your colleagues, or will you assign more?

You say to yourself,

I’ve read the research, ‘Good pedagogy means that languages are best taught by maximizing the repeated exchange of meaningful messages with a focus on target structures and accuracy’ (Gatbonton & Segalowitz, 1988; 2005). I’ll assign 7essays this semester—one every 2 weeks for 14 weeks.

Good. It’s decided. “But wait! Correcting and scoring 120-150 essays 7 times will exhaust me,” you gasp. You do the calculation.

150 students x 10 minutes x 1 essay task = 25 hours of correction.

25 hours of correction and scoring x 7 essay tasks = 175 hours

You gulp, “I won’t last long in this job if I do that to myself. Forget good pedagogy! Two essays it is.”

Be Practical

Admit it. This little second-person narrative is not total fiction. It is a conversation that teachers, many teachers have with themselves, and it captures – we suspect – a similar reflection on good pedagogy that plays out all over Quebec at the beginning of every college ESL teacher’s career. We all know what good pedagogy entails, but good pedagogy can sometimes be a recipe for burnout. We soon realize that it is better to play it safe, optimizing second language pedagogy in other ways. After all, perfection is not the enemy of the good.

Crowdsource

One solution we have all tried is to crowdsource corrections and scoring through peer-evaluations. When you point a finger in judgment, 3 fingers point back at you, right? Indeed, judging the performance of our peers helps us reflect on our own performance. However, peer feedback does not produce reliable scores for the class. Half the students take the challenge of peer assessment seriously. The other half will be too focused on themselves to get the job done or try to game the system by trading high scores with a classmate for substandard work.

Another solution is self-assessment. However, it fails for the same reason: student feedback and scores are unreliable. In other words, students are often too inept to recognize their own ineptitude (Kruger & Dunning, 1999).

Blame class size

Perhaps, you are thinking that teachers would do a better job if we had smaller groups. That’s true. But discussing it is a waste of time for 3 reasons.

• First, the government is not suddenly going to reduce class sizes because of the additional costs that taxpayers would incur.
• Furthermore, there are not enough teachers out there to teach the new groups created by smaller caps on class-size.
• Finally, the effect on achievement from smaller class sizes is rather small in comparison to other interventions (Hattie, 2009).

Formative evaluation has almost 7 times the effect size of reducing class sizes (Hattie, 2009)

Automate essay evaluation

This is new. Until now, automatic essay evaluation was only available to large corporations. Companies like the Educational Testing Service have used a combination of automatic evaluation and expert essay ratings for college admissions exams and proficiency exams. Regular college teachers have had no access to automatic essay scoring systems until now.

Introducing the Virtual Writing Tutor’s automated essay evaluation system! You can now give students formative automatic essay evaluations for free and forever.

• IELTS Task 1 and Task 2 essay tests
• film-analysis essays
• and now argument essays about controversies

Automatically scored IELTS Task 1 and Task 2 essays

The British Council offers a standardized academic proficiency test, the IELTS. The Virtual Writing Tutor (VWT) provides automatic formative feedback and scoring of IELTS Task 1 and Task 2 essays for free. The goal is to give students the practice they need before taking the test. You can test this automatic evaluation system with the sample IELTS essay texts here.

The VWT’s Task 1 test for IELTS

What do students say?

Here are some comments the VWT has received from students who have used the system:

• “Beautiful tool! I appreciate it.”
• “The service you are providing on the Internet is the best in this context. IELTS seekers like me definitely appreciate your work heartily.”

Automatically scored film-analysis essays

In September 2019, Frank and I created a system to automatically score film analysis essays. Frank’s students participated in a film project with the following task from the newly published Actively Engaged in Academic Writing (Bonkowski, 2019):

1. Choose a film based on a novel.
2. Read the first 30 pages of the pages of the novel.
3. Watch a film adaptation of the novel.
4. Research and take notes on two professional film reviews.
5. Discuss your literary analysis using a jigsaw activity.
6. And finally, write a four-paragraph film analysis.

Students used the feedback from the VWT’s film analysis essay evaluation system to guide their revisions, and Frank scored their final revised version. By providing their formative evaluation in 2 seconds, the system cut Frank’s correction load significantly. You can test the system with a sample film-analysis essay here.

The VWT film-analysis essay evaluation system cut the teacher’s correction load in half

What do students say?

Here are the comments one of Frank’s students made when asked them about her revision strategies.

I improved my score through the Virtual Writing Tutor, and I revised it – at least – I think 7 times – until satisfied with my text. So, indeed, I improved it a lot. The comments were really good. The Virtual Writing Tutor gave really good comments and very specific comments. For example, when my topic sentence isn’t strong enough and I didn’t make a strong enough claim, it tells me my grade is 60/100. And so I can improve it. And that’s how I improved a lot of things. The vocabulary, the feedback is really good. It says when you don’t have enough words related to the literary analysis and to film terms. Ada, 103A student

From grammar checker to essay checker

After 7 years of development, the Virtual Writing Tutor is making a transition from a grammar check website to an artificially intelligent automatic tutoring system. It looks like it can help teachers maximize writing practice opportunities for students without impacting the teacher’s correction load. Early evidence from student interviews suggests that it can help students improve their writing with clear, detailed, formative feedback on multiple features of complex essay writing tasks.

Will you integrate the VirtualWritingTutor.com into your course next semester? What do you think?

Works Cited

• Bonkowski, F. (2019). Actively Engaged in Academic Writing. Montreal, Quebec: Bokomaru.
• Gatbonton, E., & Segalowitz, N. (1988). Creative automatization: Principles for promoting fluency within a communicative framework. TESOL Quarterly, 22, 473-492.
• Gatbonton, E., & Segalowitz, N. (2005). Rethinking communicative language teaching: A focus on access to fluency. The Canadian Modern Language Review , 61 , 325–353. doi:10.1353/cml.2005.0016
• Hattie, J. A. C. (2009). Visible learning: A synthesis of over 800 meta-analyses relating to achievement. London, UK: Routledge.
• Kruger, Justin; Dunning, David (1999). “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments”. Journal of Personality and Social Psychology. American Psychological Association. 77 (6): 1121–1134.