Abstract

COVID-19 has been a turning point for online learning across the world, but the remarkable story has been the meteoric growth of synchronous or “live” online education. As instructors around the world have turned to video conferencing platforms to teach, a lot of instructor feedback has recently emerged on a variety of topics. One of the key emerging issues for instructors and institutions has been extracting and consolidating data on student engagement and learning outcomes. This paper attempts to identify and classify the types of learning data required to understand student learning behavior in synchronous (or “live”) online classes based on a multitude of factors such as immediacy, consolidation, and availability to different stakeholders at different points in time. Further, an attempt is made to assess how popular video conferencing platforms address the issues of data generation, presentation, and collation. Lastly, the paper also proposes an alternative approach that can potentially help create data in an easier, more efficient manner.

Keywords: videoconferencing, online learning, learning analytics, students’ data, synchronous education

Introduction

Much has changed in higher-ed instruction in the months following the COVID-19 outbreak. As per a faculty survey conducted by Inside Higher Ed and Gallup (Jaschik & Lederman, 2019, p. 11), only 3% of all instructors in the US had ever taught a synchronous online course. As we now know, this number grew manifold due to the social distancing requirements necessitated by the COVID-19 pandemic. It may be said that learner engagement was expected to be a challenge, and universities intuitively expected that enrolment and retention would be a further challenge. Therefore, it may not surprise many that the demand for learning analytics went up significantly during the pandemic. Two-thirds of all US universities reported an increase in the demand for learning analytics, as per an EDUCAUSE survey (Arnold et al., 2020).

Specifically, the largest increase in demand for data points was in technology usage (videoconferencing, LMS, accessibility tools, etc.). The goals of acquiring learning data varies from institution to institution depending on their ongoing and new challenges, and the context in which they operate. For example, a college or university that has historically struggled with student dropout may seek the data to inform their student success and retention initiatives, whereas another may wish to glean insights about how students are coping with the new modality of instruction. Of all the factors, "Assessment of student activity in online courses" was the most popular as per the survey, with 56% of all the institutions who participated in the survey indicating that it is one of the topics/issues most relevant to student success analytics right now.

What is learning data and how is it used

Data in teaching and learning does not necessarily have to be digitized and stored in databases. In in-person instruction, much of the information regarding student engagement and course progress is never stored digitally. Often, this information is simply known to an instructor, who has the opportunity to meet students and observe them on a frequent basis. In other words, learning data is information about student engagement, whether it's stored via digital means or is simply known to a teacher.

However, there is tremendous value in digitizing, organizing and storing learning data for use by all stakeholders in an organization. While professors can use learning data for purposes such as improving the quality of instruction, receiving immediate feedback from students, and continuous assessment over the course of an academic term, universities can derive several benefits from the data as well. It can be used for student success and retention, institutional accreditation activities, and the disbursement of scholarships and financial aid in a systematic manner.

Figure 1
Utility of learning data in a higher education institution

Source: authors' own work.

Instructors' need for learning data

As teaching and learning moved to an online medium in March 2020, it was deemed necessary by the authors to identify the key challenges being faced by instructors. To do so, the authors devised a survey for higher education faculty. The survey was hosted online via Google Forms and the link was shared with verified faculty users of the Acadly app, a product developed and managed by the authors' company. All responses were collected in April 2020.

Of the survey respondents, 53% were from the US and Canada, 8% from Latin America, 15% from the EU region, 15% from India, and the others from the Middle East, Australia, and South Africa.

Figure 2
Instructors' response to the question "If you are teaching, or have taught, synchronous online classes, please select all the issues you faced"

Source: authors' own work.

As illustrated by the survey, the problem selected most by instructors was the difficulty of tracking students' overall engagement with the lecture, an area that is closely related to the domain of learning analytics.

Written feedback was also received from a number of instructors (10% of the survey respondents selected "Other," at which point they were prompted to elaborate), and some of the frequently occurring themes are summarized here.

Data points instructors wish to have about online meeting

Attendance, participation, discussion, preparation, and performance are some of the key metrics that shed light on students' progress in online courses. To break these down, these are some of the common questions that instructors would like answers to:

• Who joined the online class meeting?
• How much of the online class meeting did the students attend?
• Who responded to the polls?
• Who responded to the quizzes?
• Who was active in the chat?
• Who was paying attention to the instructor's screen during the meeting?

It is not enough for the data to just exist (and it does not always exist)

The availability of learning data is only one of the factors around the need for data in boosting learning effectiveness. However, for the data to be truly valuable, there are several other considerations as well:

• the data must be easy to generate without needing too many tools;
• data should be automatically generated by the video conferencing/virtual classroom platform, whenever possible;
• it should be easy to access for the instructor - during and after their online class meetings; real-time availability of the data is necessary but not sufficient; post-class availability is crucial as well;
• it should be easy to access for other stakeholders in the university, such as department-level heads, Information Technology personnel, the Academic Technology team, Instructional Designers, Student Success teams, and others;
• it should be easy to store in the gradebook;
• it should be easy to base day-to-day decisions on.

Answering key questions regarding the data generated in an online class

The following table is a closer look at all the data points that are (or can be) generated every time a class is taught online. The abbreviation "VC" in the table refers to video-conferencing platforms.

Table 1
Assessing the quality of learning data in an online class

Source: authors' own work.

As illustrated in Table 1, there are several gaps in how video conferencing platforms currently work, and what instructors and institutions need in terms of learning data. The key factor driving this is that video conferencing platforms have been created for the generic purposes of online meetings. Whether the context is a one-on-one meeting or a classroom, the current set of tools change very little and fall short on a number of factors, particularly with regards to how data is created, organized, and stored.

Proposed solution: A video-conferencing layer

As noted previously, the authors are creators of the Acadly product, which was launched in 2017 as a "Student Response System," which is the generic term used for products that help instructors boost engagement inside the classroom. Similar platforms include Kahoot, Socrative, Mentimeter, and Poll Everywhere, to name a few. While Acadly was developed as a solution for in-person teaching and learning, the pandemic necessitated a change in the approach to create a product more suitable to synchronous online learning. Currently, about 80% of Acadly's users are from universities in the US, Canada, and the EU.

Based on the survey referenced in Figure 2, Acadly's online version was developed and launched in August 2020, with support for video conferencing via integration with Zoom.

Figure 3
How Acadly tackles the issues of learning data generation and consolidation

Source: authors' own work.

While Acadly retains the video broadcasting portion of the Zoom platform, everything else is replaced by Acadly's own course components. For example, the Zoom chat and Zoom polls are no longer available when instructors use Acadly. Instead, these are replaced by Acadly's own chat and polling options, respectively. Instructors can use Acadly to share quizzes, polls, word clouds, and videos with students during online classes. It also includes an automatic attendance tracker. It integrates with LMSs like Canvas and Blackboard, and sends all learning data back to the LMS gradebook.

Creating a "skin", or frontend, for a video conferencing platform gives the platform control over several components of the lessons. Notably, all the data points mentioned in Table 1 are collected and consolidated by Acadly automatically. While some of them are available in real-time (during the meeting), others can be exported for analysis after the meeting.

How Acadly works for virtual teaching and learning

The Acadly platform aids virtual teaching and learning on both desktops/laptops as well as mobile platforms. Specifically, in the virtual learning scenario, Acadly delivers the video conferencing via its integration with Zoom, and adds teaching and learning elements of its own, as shown in the following figures and sections.

Figure 4
Acadly with Zoom integration - the "Class page"

Source: authors' own work.

As shown in Figure 4, the maximizable "Video" unit is powered by Acadly's integration with Zoom, but there are notable additions to the interface from a teaching and learning perspective:

Attendance

The attendance section on the class page in Acadly helps instructors keep track of absenteeism without putting in additional effort. The platform supports a number of attendance-tracking mechanisms out-of-the-box:

• Check-in-based attendance: Tracks students who log in to the class at least once;
• Watch time-based attendance: Tracks the duration for which a student attended the session;
• Action-based attendance: The instructor can launch an "Are you watching?" prompt that appears on students' screens silently. If a student is watching the lecture, they simply need to tap a button on the pop-up on their screens to confirm their presence;
• Manual: The instructor can mark students present, absent, late, or excused manually as well.

Figure 5
The attendance section in Acadly

Source: authors' own work.

Activities

Acadly includes the following activity options that instructors can use to engage their students:

• Polls: Include options such as anonymity, timers, and dedicated discussion threads;
• Quizzes: Include options such as automatic grading, and an on-screen timer;
• Word Clouds: The instructor can summarize open-ended textual feedback from students in the form of "Word Clouds" or a plot of words with sizes proportional to the frequency of their appearance in the discussion;
• Discussions: The instructor can ask students to respond to questions in an open-ended manner;
• Resources: Resources are files, videos, and links that an instructor may want to share with the class.

When a student or instructor navigates to an activity page or attendance page, Acadly minimizes the video unit to a picture-in-picture mode, as shown in Figures 5 and 6.

Figure 6
Activities in Acadly (shown in the image below - polls)

Source: authors' own work.

Discussions

The discussions section on the right is meant for communication between the course team members and students, and is built from the ground up without using any of the elements of the Zoom discussion feature. Comments can include mathematical equations and image/file attachments. More importantly, the discussions are not automatically erased after the meeting ends and can continue beyond the lecture hours as well.

Post-class availability

One of the emerging concerns with video conferencing platforms is that neither the activities nor the discussions stay active after an online class meeting. Acadly addresses this by automatically creating class archives, retaining all activities, discussions, and meeting recordings in an organized manner. All the activities created on Acadly can be copied and reused by instructors in the in-person teaching and learning as well. Therefore, any effort spent on the platform can serve to aid post-COVID eventualities too.

Transferring data to the LMS using the LTI standard

The Learning Tools Interoperability (LTI) standard was developed by IMS, a non-profit alliance of education technology tools providers for institutions of higher education. The LTI standard helps education technology platforms connect to the LMS to provide a variety of administration tasks in an automated manner. Tools like Acadly can leverage the LTI standard to link deeply with LMSs and transfer data to the LMS gradebook. Notable LMS providers that support the LTI standard include Moodle, Blackboard, Canvas, Brightspace (formerly known as Desire2Learn), and Sakai.

This has notable ramifications for online learning analytics. All data generated during an online meeting can be "synced" or automatically transferred to the LMS by LTI-compatible tools. Leveraging the standard increases the value of the data for all stakeholders in the system.

The Acadly platform uses the LTI standard to make all data regarding attendance, discussions, poll participation, and quiz performance transferable to the LMS with little effort. This can serve as a blueprint for how video conferencing platforms can evolve to better address the online learning needs of a higher education institution.

Figure 7
Acadly's LMS plugin

Conclusions

Given the nascent state of synchronous online teaching and learning, deficiencies in video conferencing platforms from the education perspective are to be expected. However, as synchronous learning grows, providers must adapt to the new needs, which can be expected to continue for not only the foreseeable future, but also well beyond the pandemic.

Learning Analytics must be a focal point of conversations on online education, and video conferencing platforms can leverage existing standards such the LTI standard to integrate closely with Learning Management Systems, which are used by several universities.

For video conferencing platforms, an important step in their evolution will be to help universities and instructors acquire learning data that is crucial for the improvement of the quality of education and student success. Tools such as Acadly provide a blueprint for such growth, but the work on this front can still be considered formative and introductory.

References

• Jaschik S., & Lederman D. (Eds). (2019). 2019 Survey of faculty attitudes on technology: a study by Inside Higher Ed and Gallup. https://www.insidehighered.com/booklet/2019-survey-faculty-attitudes-technology
• Arnold, K., Feng, L., Ham, M., & Miller, A. (2020, May 28). EDUCAUSE COVID-19 QuickPoll Results: Student Success Analytics. Educause Review. https://er.educause.edu/blogs/2020/5/educause-covid-19-quickpoll-results-student-success-analytics