Abstract

Retrieval practice is described as a pedagogy that includes ways that students have a chance to recall information during their study process. With the introduction of online learning and course management systems which permit secure testing online, a way is provided to generate many sets of small quizzes from large question pools which can serve as an efficient digital method for practicing recall of the information to be learned. Course management systems also provide a way to capture and store the results of the small quizzes used for retrieval practice. This study looks at data captured by taking numerous small quizzes as the retrieval learning process and compares the captured data sets to the grade distribution in the major exams for which the small quizzes are being used as a way to learn the material being tested. The overall results demonstrate that the use of retrieval practice does enhance student learning and also was found to lower the drop rate for the micro and macro anatomy courses studied.

Keywords: retrieval practice, online courses, small quizzes, higher grades, drop rate

Introduction

Retrieval practice describes a method of learning that includes strategies designed so that students have a chance to recall information during their study process (Agarwal, 2019; Brunyé et al., 2020; Karpicke & Roediger, 2008). One familiar method to practice information recall is to study by using flash cards. But, the introduction of online learning and course management systems which permit testing online now provides a method to generate many sets of small exams from large question pools which can serve as an efficient digital method for practicing recall of the information in a way similar to the use of hand held flash cards. Course management systems also provide a method to capture and store the results of the small quizzes used for retrieval practice. Earlier studies (Kolitsky, 2008) using multiple test taking as a method to learn in exams taken in non-proctored conditions supported the view that exams taken in short time conditions gave similar grade distribution data when compared to exams that were given in a face-to-face classroom. The introduction in recent years of video proctoring technology such as Respondus Lockdown Browser (Respondus, n.d.) now provides a method to ascertain with greater certainty that cheating is not happening when testing is done online. This study looks at data captured from students taking numerous small quizzes for credit as the retrieval learning process and compares the captured small quiz data to the grade distribution in the major exams in several online courses offered in previous years when small practice quizzes were available but not for credit. The results show that offering students a small amount of credit for taking practice quizzes as a form of retrieval practice results in higher grade attainment when compared to the same courses offered in the fall semester for the past three years.

Retrieval practice data

Three courses were used as the focus for studying the impact of retrieval practice or multiple testing on learning. Histology has been offered online since 2000 and is currently offered three times a year in the fall, spring and summer semesters with enrollment now ranging from 180-200 students per semester. The Histology course has both a lecture as well as a lab component with lab images digitized from a collection of high-quality photos of both light and electron microscopic views of human tissues. The online Anatomy and Physiology lecture and the online Anatomy and Physiology lab are taught as separate courses and for this study, only the fall 2019 semester will be studied with enrollments of 182 for the lecture and 131 in the lab. Each Anatomy and Physiology lecture and lab course utilize Anatomy.tv etext sold by Primal Pictures (Anatomy.tv, n.d.). The lab course also utilized the PhysioEx 9.1 lab manual (PhysioEx, 2020) with four online simulated experiments assigned for the fall semester. Other labs in the same course use, with permission, high quality human anatomy images from the Bassett Collection at the Stanford University Medical History Center (Bassett Collection, n.d.). Originally, they were designed to be used as 3D images when viewed with ViewMaster, but a similar experience could be achieved when using inexpensive red-blue glasses. Many of the right-left eye views used for ViewMaster can be made into anaglyph images for a more realistic experience (Kolitsky, 2013). Students in this lab have access to 3D red-blue glasses for viewing anatomical dissection in 3D. Other images for this lab came from the National Library of Medicine Visible Human project (National Library of Medicine, 1995) which has available images from the cross sectioning of frozen male and female cadavers so that transverse sections can be made into learning objects for the study of all body systems.

The basic plan for testing the effect of retrieval practice involves the use of small quizzes taken from a larger question pool and presenting the option to take these small quizzes numerous times. Incentive to do the quizzes was provided by giving a small amount of credit for doing the quizzes a number of times. How many times was dependent on an estimate made from an earlier study (Kolitsky, 2008) showing the number of quizzes students took who received high grades in online courses in which the number of quizzes done was determined. The number of quizzes done ranged from 20 to 40 for students in the A or B category with no credit assigned for doing the quizzes. For this study, it was decided to have 10 times be the point from which more quizzes would earn credit. So, there was a minimum of 10, and then the next five counted for credit, but the students usually did many more than 10 and then the last five tests taken counted for credit, and if high scores on quizzes 11-15 were not attained, then more would be done until the final five in a row earned a high credit score.

The credit for doing the quizzes varied in each course. In the Histology course, the smaller quizzes were each worth just 0.5 points out of the possible 100 that were generated for a grade for the course. The total points for the Histology lecture and lab small quizzes was 11 points out of the possible 100 for the entire course. For the Anatomy and Physiology lecture course, each small quiz was worth one point with 10 small quizzes counting for 10 points of the total 100 that can be earned. For the Anatomy and Physiology lab course, three small quizzes worth two points each for a total of six points were employed for the retrieval practice study. And because the small quizzes were providing some percentage of the overall points that determine the grade a student receives for a course, some of the exam data appearing in the following sections are listed as the actual points earned in the exam so those figures could then be compared to the number of small quizzes taken or the total points earned from doing the quizzes.

Histology course

Figure 1 shows the results of looking at the grade distribution of 182 students completing an online Histology course in the fall semester of 2019 compared with the grade distribution for students in the same online course offered in 2018, 2017 and 2016 with a total of 172 for fall 2018, 128 for fall 2017 and 137 students for the fall 2016 semester. Note that the data profile for the 2019 students in the dark gray columns visibly shifts to the higher grade category when compared to the light gray bars for the combined 2018, 2017 and 2016 grade distribution, with the highest percentage of students in the B grade category, and shows the positive effect of providing small quizzes for credit as an incentive for students to do them.

Figure 2 uses the Blackboard Evaluation option to generate course reports which focus on the Course Activity Overview that permits analysis of the total amount and type of activity for each student taking the exam. From this type of analysis, it is possible to gather data for how many times a small quiz or group of small quizzes was used in the study of a particular lab activity. For the data appearing in Figure 2, the number of times students completed a small quiz was captured and shows that students who received As in the course used the small quizzes the most with a decreasing number shown for B, less for C, even less for D and a large drop in small quiz usage in the F category.

Anatomy and Physiology lecture course

Figure 3 shows the grade distribution for an online Anatomy and Physiology I lecture course offered in the fall 2019 semester with 182 students enrolled and compared to a combined grade distribution for students in an Anatomy and Physiology I fall semester lecture course offered in 2018 with 153 students, 2017 with 115 students and in 2016 with 122 enrolled, for a combined number of 390 students for all three courses shown in Figure 3 as light gray bars. Note, as in Figure 1 for Histology, there is a similar shift to the higher grade categories as indicated by the dark gray bars for A and B when students were encouraged to do the small quizzes by attaching a small amount of credit to them compared to the combined data for 2018, 2017 and 2016 when small quizzes were also available but with no credit assigned for doing them.

In Figure 4, the total points accumulated in the fall online Anatomy and Physiology lecture course for four exams (three lecture and one final exam) are shown in the darkest gray bars compared with the total points accumulated for doing the small quizzes (medium gray bar) and also for the total number of small quizzes done (lightest grey bar). In this graph, no points for other assignments such as submitting a brief bio (one point), doing a scavenger hunt (one point) and writing clinical studies (5 points) were included in the total exam points columns (darkest gray bars). The removal of points for other assignments leaves a more direct comparison of the number of points accumulated in the three lecture exams and one final exam with the total quiz points and number of small quizzes done. These numbers support the view that doing the small quizzes has a direct positive impact on student performance in the lecture exams which determine their grade. The total points earned from doing quizzes and the number of times a quiz was taken were obtained by using the Course Activity Overview tool in Blackboard which captures and displays every attempt a student makes and its result. Other course management systems also offer similar methods to capture and store usage data, providing a way to study the impact of retrieval practice on a broader scale.

Anatomy and Physiology lab course

Figure 5 shows the grade distribution for 131 students in the fall, 2019 offering of the online Anatomy and Physiology (AP) lab course compared to the grade distribution for a combined number of 131 students participating in the 2018 offering, 114 students in the 2017 and 115 students in the 2016 offerings of the same course. There is a small but noticeable shift of AP lab grades (dark gray bars) to the higher grade categories when compared to the combined grades (light gray bars) in the fall 2018, 2017 and 2016 semesters similar to what was observed in Figures 1 and 3 for the fall 2019 online offerings of the Histology, and Anatomy and Physiology lecture courses.

Figure 6 shows a strong correlation of the total points earned from doing the small quizzes (light grey bars) with the total exam points earned from taking the online Anatomy and Physiology lab exams (dark gray bars). The total points for the small quizzes were obtained by using the Course Activity Overview tool in Blackboard which, like many course management systems, provides a method to capture and display the total number of times that an activity such as taking a small quiz is recorded and also keeps a record of the points earned from taking the quizzes.

Impact of retrieval practice on the drop rate

Figure 7 shows an unexpected result from reviewing the percentage of students who dropped the online Histology, Anatomy and Physiology (AP) I Lecture and the Anatomy and Physiology (AP) I lab courses over the fall 2019 semester compared to the same online courses offered in the 2016, 2017 and 2018 fall semesters. The data in dark gray bars represent the courses in which the retrieval practice strategy of 'learning through testing' was encouraged through the offering of a small amount of credit for doing small quizzes. The light gray bars represent the percentage of students who dropped from the same courses offered in 2016, 2017 and 2018 with only encouragement given but no credit assigned for doing the small quizzes.

Number and types of questions in small quizzes

To implement a retrieval practice method of learning, two important parameters must be determined for optimal deployment. First, the number of questions in the larger question pool from which questions for each retrieval practice attempt are selected needs to be determined. And second, the types of questions which can reinforce the learning of terms (in this study, terms related to micro- and macro-anatomic structure, for example). Questions that require the learner to use the terms to explain the function of both normal and abnormal conditions must also be taken into consideration. The study of anatomy and physiology covers the language that must be understood so that the terms for the structures and their locations in the body can be used in sentence form to explain how they work in good health and also disease conditions. In this sense, the types of questions to include in the larger pool from which the small quizzes are made must be designed so that learning the answers to those questions can over time lead to the ability to recall the information in the testing that counts for credit. Images with labels for identifying structural elements must also be part of the large question pool.

The small quiz pool for the lecture portion for the Histology course contained 523 items to learn when questions were chosen for retrieval practice whereas for the Histology lab portion, 850 answers were available in question format when questions were chosen for the small quizzes used for retrieval practice. For the Anatomy and Physiology I lecture course in this study, the small quiz pool contained 817 questions with answers available for retrieval practice and the Anatomy and Physiology I lab course included 243 questions with answers to be used for that activity.

Future use of retrieval practice

Online courses which utilize course management systems such as Blackboard offer the opportunity to gather meaningful data to test the impact of retrieval practice learning methods in courses such as those studied in this report. The courses in this study require learning the definitions of terms which are linked to how they explain normal and abnormal conditions. The learning method provided by the doing of small quizzes many times requires a large set of questions to draw from for each small quiz taken. How many times a student must do a small quiz, how many chapters should be covered in a set of small quizzes and how many small quizzes should be employed for optimal learning are still testable areas of study. However, the observations thus far show that pools of questions ranging from 50-100 seem to work well in providing enough coverage for learning the important information found in one or two text chapters. Courses from other areas of Biology and even other Science as well as non-science disciplines could have different portions of the material that needs to be learned in order to become literate in that field. Therefore, the extent to which using the retrieval practice methods may be beneficial for a particular discipline may vary.

The impact that retrieval practice has on student learning is linked to the capability of being able to analyze the periodic production of large amounts of data generated from taking small quizzes following completion of each major exam in the course being studied. In this study, the Environmental Science and Engineering Ph.D. program at The University of Texas at El Paso (UTEP) awarded Anand Raj, Ifeanyi Nwigboji and Kamal Nyaupane graduate teaching assistantships for the 2019-2020 academic year and they assisted in the calculation of the data gathered from the use of small quizzes in the online Histology, Anatomy and Physiology lecture and lab courses for the 2019 semester. From the data gathered in this study, retrieval practice does show an increase in student performance. Future use of retrieval practice may be more beneficial to students if data compilation was also employed to show students soon after completing an exam how their performance on doing the small quizzes used for study was important to their success on the major exam that was taken. This strategy, however, does require a quick turn-around of taking the major exam and feeding back to students the results from the calculations of the small quizzes done during study for the exam. It would also offer the opportunity to study the nature of how students learn, as student success in a particular discipline may be linked to their natural tendency to study based on a particular method that may or may not include a retrieval practice strategy. And lastly, there is also no reason that retrieval practice should only be used in online teaching. The use of small quizzes in the learning phase for a subject does not have to be only thought of as happening in an online class experience and could be employed as a learning tool in a traditional face-to-face teaching environment.

Effect of pandemic on retrieval practice study

The COVID-19 pandemic hit the El Paso area in the middle of the 2020 spring semester just prior to spring break. One of the responses of the administration was to postpone by one week the student return to campus following spring break to give the faculty extra time to transition all face-to-face classes into online courses. Students who were living on campus also had to go home or off campus and travel was also reduced for students who attended UTEP from Juarez, the city in Mexico just across the Rio Grande River from El Paso. Students in all courses were also given the opportunity to change their grading option from traditional letter grading of A-F to Satisfactory/Unsatisfactory. Many projects such as this one, which was continuing the effort started at the beginning of the 2019-2020 school year to demonstrate how best to utilize retrieval practice to improve online learning, were interrupted. What was learned, however, from the fall semester study was of benefit to students by providing them with a strong foundation for online learning through the continued use of retrieval practice methods for the spring and also now in summer online courses.

References

• Agarwal, P. K. (2019). Retrieval practice & Bloom's taxonomy: Do students need fact knowledge before higher order learning? Journal of Educational Psychology, 111(2), 189-209. https://doi.org/10.1037/edu0000282
• Anatomy.tv. (n.d). Retrieved May 5, 2020 from https://www.anatomy.tv/welcomer
• Bassett Collection of Stereoscopic Images of Human Anatomy. (n.d.). Retrieved March 5, 2020 from http://lane.stanford.edu/biomed-resources/bassett/index.html
• Brunyé, T. T., Smith, A. M., Hendel, D., Gardony, A. L., Martis, S. B., & Taylor, H. A. (2020). Retrieval practice enhances near but not far transfer of spatial memory. Journal of Experimental Psychology: Learning, Memory, and Cognition, 46(1), 24-45. https://doi.org/10.1037/xlm0000710
• Karpicke, J. D., & Roediger III, H. L. (2008). The critical importance of retrieval for learning. Science, 319, 966-968. https://doi.org/10.1126/science.1152408
• Kolitsky, M. A. (2008). Analysis of non-proctored anti-cheating and formative assessment strategies. e-mentor, 4(26), 84-88.
• Kolitsky, M. A. (2013). 3D stereo images enhance online learning. http://www.nextgenemedia.com/3Danatomy/index.html
• National Library of Medicine. (1995). https://www.nlm.nih.gov/research/visible/visible_human.html
• PhysioEx. (n.d.). Retrieved May 5, 2020 from https://www.physioex.com/order_info_pex9.html
• Respondus Lockdown Browser. (n.d.). Retrieved May 5, 2020 from https://web.respondus.com/he/lockdownbrowser/