Exploring Online Student Assessment Systems: Part 1 – Learning Catalytics
In search of a faster and more efficient way of assessing student comprehension of topics addressed in class, Anna Dera, a Biology Teacher at Champlain St.Lawrence College, stumbled upon Learning Catalytics. Diagrams, graphs, pictures and data analysis are all essential for assessing student understanding in biology, and many assessment systems are limited with respect to using this type of material. Anna’s research has taught her quickly that not all classroom response systems are made the same.
In biology, as in many other disciplines, there are fundamental concepts that students need to have a solid understanding of in order to be able to progress in the subject. Before trying online response systems, Anna’s assessment process for her students involved marking completed formative quizzes or assignments (on paper) and then analysing what required revision in class. The assignment was returned to students and the problematic topics were revisited with additional explanations. This process was long and it was often difficult to garner the same level of interest from students for a topic that had already been covered one or two classes prior. Anna wanted a tool that would allow her to immediately see which concepts were not understood while they were being taught, so that they could be addressed before going any further.
Getting to Know Learning Catalytics
Learning Catalytics, like other online response systems, allows for immediate feedback during class. It works with any web-enabled device, including smart phones (much to the delight of students). Once students have responded to the questions, results are instantly compiled and provided to the teacher. Not only can the teacher see which questions are causing the most difficulty, they can also see who is having difficulty with the subject matter, using a seating plan which is created by the teacher.
An example of a seating plan with student responses.
Teachers can provide immediate feedback to students providing them the help they need while the topic is still fresh in their minds. Registration for teachers is free and provides an interface for students to access. Student registration is not free however, and they are charged $12 for a 6-month subscription. Although it’s not free for students, once they have paid, they have access to all the modules created by any of their teachers who are using Learning Catalytics, regardless of department or program. When you consider the extent to which a Learning Catalytics account can be used, it is not an expensive investment.
For Anna, the most significant strengths of this system are the variety of questions that can be included in a module and that students can return to modules they have already completed for test preparation and further explanations from the teacher.
Here is a list of the different types of modules that can be created:
- Graphs – data can be plotted on a graph and predictions made about the results.
- Photos or diagrams – students can circle or draw the correct answer directly on the image (this can be done directly on the screen with tactile devices).
An example of how images can be integrated into the modules. On tactile devices (like tablets), the arrow to show the direction of synaptic transmission can be drawn using their finger.
- Data collection – data entered is automatically collated and the teacher can then decide how data is displayed to students.
- Texts –students can be asked to highlight parts of a text (words, key phrases, etc.), this can also be done directly on the screen with tactile devices.
- Typical quiz question types – long answer, short answer, multiple choice.
Photos are easy to upload and Learning Catalytics will allow modules to be answered a second time. In this example, after Round 1, Learning Catalytics put the students into teams to allow for discussion before answering a second time.
- Matching – 2 columns of information (great for studying definitions).
- Ranking – can be done in many ways: chronological, largest to smallest, etc..
- Word clouds – can be generated from student responses and discussions can follow
Examples of the variety of modules that can be created with Learning Catalytics.
Of course there is a substantial amount of upfront work to create the modules. However, Learning Catalytics will save the modules and it also maintains a bank of developed questions that teachers have chosen to share with the Learning Catalytics community. A simple search within a specific discipline will provide a whole host of available material. These resources can be used ‘as is,’ modified, or serve as an inspirational starting point for the creation of your own material.
The Learning Catalytics dashboard for one of Anna’s courses. Note that the proportion of correct answers is displayed for each question in each module.
Modules can be set up to be done in-class, or automated with a time frame for quizzes to be done at home. They can be self-paced with instant feedback, or teacher-paced with answers being provided in various ways. All of these choices provide a plethora of potential uses for this program.
There are 3 different ways the teacher can download the results after a module has been completed, as seen in the top left-hand corner of this screen shot: these are summary format, long format and a wide format. After a class, the teacher can review all of the responses and then provide feedback using the Comments feature. There is even a Message feature to allow students to send teachers a message while the module is active. This is a useful feature for shy students.
Another interesting feature is the collaboration element that can be programmed by the teacher, using parameters that are chosen in advance. Whether it is prior to answering questions or after the results have come in, students can be grouped together in order to promote peer learning. For example, if only a handful of students respond correctly to a question, the teacher can ask Learning Catalytics to assign groups based on parameters such as group size, location in the classroom, and correct responses. Once grouped, students are instructed to convince the other members of their group why they believe their response was correct. The exchange allows students to help each other identify unclear concepts. If many students still respond incorrectly, the teacher can intervene to determine misconceptions and clarify the correct responses. Students who still find the problem unclear can review the modules with detailed explanations after class.
Integration into Other Pedagogical Approaches
Using this tool as a part of her pedagogical practice has made time in class more productive, allowing for more one on one time with the students who need it the most. For example, a new topic could begin with a PowerPoint presentation. A few slides into the lesson, students would be asked to log in to their web-enabled devices and to complete a quiz. This allows Anna to see if the class is ready to move forward or if more time is required on the initial information. From the student perspective this tool is a huge success. Every year, students say this was the best part of the course! Results from student evaluation surveys have shown that students feel it clarifies the content, enhances their understanding and helps them learn on their own.
Drawbacks to Using Learning Catalytics
Aside from the fact that this software is not free, there are only a few drawbacks. A good Wi-Fi system needs to be in place before having an entire class use the software at the same time. Of course there can be one or two students who arrive in class with dead batteries in their smart phones or tablets. Luckily she has access to iPads as back up devices to ensure everyone can participate. Being in Quebec, it should also be noted that the software is from a US-based company and it is only available in English. Questions can be in created in French, but the software itself runs on an English-language platform. In addition, the modules cannot be saved indefinitely. At the end of a course, the teacher will receive a message asking them if they wish to save their modules, and for how long.
Looking for a Free Equivalent
Because this software is not free, Anna is only comfortable incorporating it into her pedagogy when it can be used on a regular basis. For example, in the winter 2016 semester she is teaching a course where it could be only used a few times throughout the semester. Therefore, she did not ask students to register. This is the issue that has led her to start researching again and evaluating other free online response systems. A fairly well-known one, Socrative, is highlighted in a Profweb article (in french). Anna will be analysing online response systems for their completeness, as she does not want to compromise some of the important aspects of Learning Catalytics, including the variety of modules, the collaboration options, and the review aspect for test preparation. Stay tuned for a follow-up article at the end of the semester to learn about which systems she will have experimented with and her analysis of their capabilities.