Providing feedback has several virtues that no teacher would deny. But what about automated feedback powered by artificial intelligence (AI)? Through a case study from an online programming course, I explore here the different aspects of automated feedback, and attempt to push our reflection beyond the boundaries.
The importance of feedback
It is said that to grade assignments fairly, it would require the intervention of at least 2 people. I myself put into practice this valuable pedagogical tip, by drawing from my teaching skills to provide a 1st intervention. For the 2nd intervention, I let the student carefully review their evaluation and report any mistakes or inconsistencies.
In the end, other than the long-awaited grade, what do we expect from evaluating learners, if not to guide them in developing knowledge or competencies?
After checking their grade, some learners will ask to receive feedback in order to do better next time. Others will be satisfied with the grade and move on to other things. Yet, feedback is essential to accomplish our teaching responsibilities. It involves knowing the students and offering constructive criticism on their learning process, to highlight their strengths, accomplishments, and avenues of improvement.
The time factor
When teaching large groups of students, giving feedback might pose certain problems, particularly regarding time management. We all know that time is a real blocking factor in education since we need time to get to know our students better. We need time to develop meaningful learning progress. We need time to reflect on the assessment process. We need time to provide appropriate and individual feedback to allow students to succeed.
The 4 levels of feedback
The time factor must be taken into account, especially when effective feedback operates on 4 different levels (Rodet, 2000 [in French]):
- cognitive feedback (errors correction, indications of whether the performance is correct or not, etc.)
- metacognitive feedback (proposal to finding potential solutions)
- methodological feedback (content organization)
- emotional feedback (validation of progression of learning, positive comments, etc.)
Automated feedback would allow educators to manage their time effectively, to consider all the levels of feedback, and to take more time to get to know the students.
Case study: Philips Pham, a student at Stanford University
This case study presents an example of the use of automated feedback in an online computer programming course and illustrates the various possibilities this technology offers. In the fall of 2021, Philip Pham, a 23-year-old Swedish student was taking a class called Code in Place. Run by Stanford University, the course taught basic skills in computer programming. After a few weeks of classes and successful formative assessments, Philips takes the final evaluation which consisted of developing an image generator. The student submits a program that could draw waves of tiny blue diamonds across a black-and-white grid.
A few days later, he received a detailed critique of his code. The system praised his work, and subtly pinpointed an error he made: “Seems like you have a small mistake. Perhaps you are running into the wall after drawing the 3rd wave.”
This is exactly the feedback Philips needed to improve his code. Indeed, the student enjoys being challenged, and the feedback allowed him to identify and solve the problem by himself.
This intervention came from a machine. During the online class, a new kind of artificial intelligence analyzed, evaluated, and offered feedback to thousands of other students who took the same test. Built by a team of Stanford researchers, this automated system suggests significant improvements when it comes to student monitoring. This is especially true for online education, which can involve thousands of students, but does not always provide the guidance many students need to find the motivation.
Stanford’s neural network
Chelsea Finn, PhD. and her team built this neural network, an algorithmic system able to learn skills from vast amounts of data. By diagnosing patterns in thousands of cat photos, a neural network can learn to identify a cat. By analyzing hundreds of old phone calls, it can learn to recognize the spoken words. By examining how teaching assistants evaluate the coding tests, it can learn to evaluate these tests by itself.
The Stanford system spent hours analyzing examples from previous midterm evaluations, taking advantage of a decade of possibilities. It was then ready to learn more. It provided 16000 feedback comments, and students approved the feedback 97,9% of the time, according to an evaluation of the system by the Stanford team, whereas they agreed with the feedback from teaching assistants 96,7% of the time.
AI to help teachers… not to replace them!
The technology was effective because its role was accurately defined. When taking the test, Philips Pham wrote a code with specific objectives, and there were only limited ways he and the other students could be wrong. Since the 1970s, researchers have been developing automated teaching tools, including robotutors and computerized essay graders.
Recently, researchers have built technology able to analyze natural language the same way the Stanford system analyses computer codes. We know this technology as ChatGPT, developed by OpenAI.
Although the Stanford system provides accurate feedback, it becomes useless if students have any questions about what went wrong. But for Chris Piech, the Stanford professor supervising the course, the goal is not to replace teachers when it comes to giving feedback. In fact, it is a way of reaching more students than teachers could ever reach on their own, but also to save time that can be spent on a more personalized approach. If the machine can clearly identify the mistakes made, by pinpointing the errors, it could help teachers to target the students in need. According to Chris Piech: “The future is symbiotic — teachers and AI working together”.
