March 21, 2023

Automated Feedback powered by Artificial Intelligence in an Online Programming Course

Providing feedback has several virtues that no teacher would deny. But what about automated feedback powered by artificial intelligence (AI)? Through a case study from an online programming course, I explore here the different aspects of automated feedback, and attempt to push our reflection beyond the boundaries.

The importance of feedback

It is said that to grade assignments fairly, it would require the intervention of at least 2 people. I myself put into practice this valuable pedagogical tip, by drawing from my teaching skills to provide a 1st intervention. For the 2nd intervention, I let the student carefully review their evaluation and report any mistakes or inconsistencies.

In the end, other than the long-awaited grade, what do we expect from evaluating learners, if not to guide them in developing knowledge or competencies?

After checking their grade, some learners will ask to receive feedback in order to do better next time. Others will be satisfied with the grade and move on to other things. Yet, feedback is essential to accomplish our teaching responsibilities. It involves knowing the students and offering constructive criticism on their learning process, to highlight their strengths, accomplishments, and avenues of improvement.

The time factor

When teaching large groups of students, giving feedback might pose certain problems, particularly regarding time management. We all know that time is a real blocking factor in education since we need time to get to know our students better. We need time to develop meaningful learning progress. We need time to reflect on the assessment process. We need time to provide appropriate and individual feedback to allow students to succeed.

The 4 levels of feedback

The time factor must be taken into account, especially when effective feedback operates on 4 different levels (Rodet, 2000 [in French]):

  • cognitive feedback (errors correction, indications of whether the performance is correct or not, etc.)
  • metacognitive feedback (proposal to finding potential solutions)
  • methodological feedback (content organization)
  • emotional feedback (validation of progression of learning, positive comments, etc.)

Automated feedback would allow educators to manage their time effectively, to consider all the levels of feedback, and to take more time to get to know the students.

Case study: Philips Pham, a student at Stanford University

This case study presents an example of the use of automated feedback in an online computer programming course and illustrates the various possibilities this technology offers. In the fall of 2021, Philip Pham, a 23-year-old Swedish student was taking a class called Code in Place. Run by Stanford University, the course taught basic skills in computer programming. After a few weeks of classes and successful formative assessments, Philips takes the final evaluation which consisted of developing an image generator. The student submits a program that could draw waves of tiny blue diamonds across a black-and-white grid.

A few days later, he received a detailed critique of his code. The system praised his work, and subtly pinpointed an error he made: “Seems like you have a small mistake. Perhaps you are running into the wall after drawing the 3rd wave.”

This is exactly the feedback Philips needed to improve his code. Indeed, the student enjoys being challenged, and the feedback allowed him to identify and solve the problem by himself.

This intervention came from a machine. During the online class, a new kind of artificial intelligence analyzed, evaluated, and offered feedback to thousands of other students who took the same test. Built by a team of Stanford researchers, this automated system suggests significant improvements when it comes to student monitoring. This is especially true for online education, which can involve thousands of students, but does not always provide the guidance many students need to find the motivation.

Stanford’s neural network

Chelsea Finn, PhD. and her team built this neural network, an algorithmic system able to learn skills from vast amounts of data. By diagnosing patterns in thousands of cat photos, a neural network can learn to identify a cat. By analyzing hundreds of old phone calls, it can learn to recognize the spoken words. By examining how teaching assistants evaluate the coding tests, it can learn to evaluate these tests by itself.

The Stanford system spent hours analyzing examples from previous midterm evaluations, taking advantage of a decade of possibilities. It was then ready to learn more. It provided 16000 feedback comments, and students approved the feedback 97,9% of the time, according to an evaluation of the system by the Stanford team, whereas they agreed with the feedback from teaching assistants 96,7% of the time.

AI to help teachers… not to replace them!

The technology was effective because its role was accurately defined. When taking the test, Philips Pham wrote a code with specific objectives, and there were only limited ways he and the other students could be wrong. Since the 1970s, researchers have been developing automated teaching tools, including robotutors and computerized essay graders.

Recently, researchers have built technology able to analyze natural language the same way the Stanford system analyses computer codes. We know this technology as ChatGPT, developed by OpenAI.

Although the Stanford system provides accurate feedback, it becomes useless if students have any questions about what went wrong. But for Chris Piech, the Stanford professor supervising the course, the goal is not to replace teachers when it comes to giving feedback. In fact, it is a way of reaching more students than teachers could ever reach on their own, but also to save time that can be spent on a more personalized approach. If the machine can clearly identify the mistakes made, by pinpointing the errors, it could help teachers to target the students in need. According to Chris Piech: “The future is symbiotic teachers and AI working together”.

About the author

Florence Sedaminou Muratet

Inspired by the conviction that education is the most powerful tool for changing the world, Florence Sedaminou Muratet has forged a solid academic career in anthropology and education. Cumulating over 20 years’ experience, she has not only taught French as a second language and Humanities, but also integrated interculturality and digital pedagogy into her courses at an early stage. Her major contributions include the redesign of educational programs, which led her to skillfully integrate multidisciplinary approaches and digital technologies, thus significantly enhancing the learning experience.

As a digital pedagogy consultant with Collecto, Florence stands out for her pioneering work in implementing emerging technologies in education. She is particularly committed to exploring the applications and implications of artificial intelligence in society, actively helping to shape the future of digital education.

Notify of

0 Commentaires
Inline Feedbacks
View all comments