Abstract

Open Badges have become a metadata standard that has been used to transform digital credentialing practices in learning contexts including Massive Open Online Courses (MOOCs). Badges may be designed and used in diverse ways to enhance learner motivation in MOOCs. However, empirical research about the motivational effects of the badging process has been limited to anecdotal evidence. Objective: The goal of this study was to explore the motivational effects of open badges in MOOCs. The primary research question was whether and how open badges motivate participants to: a) learn in MOOCs, b) to complete MOOCs, and c) to use Open Badges outside of the MOOC environment. Method: An exploratory learner perception study was conducted in two platforms: OpenVM and MiriadaX. Both platforms issue Open Badges to recognize learning outcomes in MOOCs. The research sample included 1788 participants with diverse backgrounds, who participated in OpenVM (n = 1412) and in MiriadaX (n = 376) courses. Learner perceptions were self-reported via two online surveys, which included an aligned set of items in English (OpenVM) and Spanish (MiriadaX). Results: The findings show that regardless of the MOOC platform, the topic of the course and the language of instruction, MOOC participants were motivated by open badges, especially by an adequate description of the skills developed in the MOOCs. A regression analysis indicated that an adequate skill description in the open badges had the greatest predictive value for motivation to learn in MOOCs and to complete the course (r2 = 56.3%). The instrument with five items showed high internal consistency (α = 0.92).

Keywords: MOOC, Open Badges, micro-credentials, motivation, competencies, skills

Introduction

The concept of the Massive Open Online Course (MOOC) is based on an instructional model that allows access to a large number of participants (McAuley et al., 2010). Since its inceptions around 2008, the MOOC format has been established as a means of open and global education (Ortega-Arranz et al., 2019). This movement has been seen as the evolution of the OpenCourseWare (OCW) initiative from MIT OCW (Ma, 2015), which revolutionized higher education with Open Educational Resources (OER) embedded into course structures (Tovar et al., 2013). MOOCs resemble online courses in their structure and elements (Peco & Lujan-Mora, 2013), because they have a beginning and an end, a syllabus, and a set of resources, assessments and communication tools (Dillenbourg et al, 2014; Martin, 2012). MOOCs differ from more traditional online courses through open access for learners from any background, design tailored for a large number of participants, voluntary and self-paced participation, and mechanisms which allow learners to decide and navigate freely through the learning activities and content (Haug et al., 2014).

Despite recent developments and innovations in the field of MOOCs, including the use of learning analytics and performance prediction (Duru et al., 2016), as well as the use of reward-based gamification strategies (Ortega-Arranz et al., 2019), motivation to learn and complete the MOOCs remains the key challenge; low completion rates still remain an issue (Reich & Ruipérez-Valiente, 2019). The factors influencing the successful completion of MOOCs have been categorized and mapped into various dimensions, such as learners' characteristics, pedagogical approaches and instructional design, and MOOC quality (Albelbisi et al., 2018). Critical MOOC characteristics include free and flexible modes of participation that allow learners to enroll and leave the course at any time. Students declare a lack of intention to complete the course, lack of time, lack of digital skills, and lack of self-regulated learning strategies (Reparaz et al., 2020; Shukor & Abdullah, 2019) among others as difficulties in this context.

Recent studies have also considered (open) badges as possible factors enhancing motivation for learning. This article builds on the research on open badges in MOOCs and explores learners' perspectives on the role of open badges for motivation to learn in and to complete a MOOC. The article is based on the results of an empirical study with approx. 1,800 MOOC participants from diverse backgrounds, who participated in MOOCs on two different platforms: OpenVM and MiriadaX. The OpenVM Learning Hub , founded under the Erasmus+ program, is an online learning environment for the achievement and recognition of virtual mobility skills in higher education. OpenVM hosts eight MOOCs which were developed in the project by nine partner organizations. The MiriadaX platform, driven by Telefónica Digital Education, is the first Ibero-American MOOC platform with over 6 million users in over 100 institutions, primarily universities, that publish open courses on a wide range of topics (MiriadaX, n.d.).

This article is structured as follows. The following section outlines past research on motivation in MOOCs including studies focusing on (open) badges. The subsequent section describes the design of the MOOCs and Open Badges in both MOOC platforms: OpenVM and MiriadaX. The fourth section describes the methodology applied to explore learners' perceptions about the role of Open Badges for motivation in MOOCs. This section is followed by the analysis of data and presentation of results from the online survey. The paper ends with conclusions and recommendations for future research.

Learner motivation in MOOCs

Research on learner motivation in MOOCs

Motivation plays a vital role in successful participation in MOOCs (Albelbisi et al., 2018). In the sections below, we outline some relevant research results related to learner motivation in MOOCs in general and to the use of (open) badges to enhance learner motivation in MOOCs in particular. Motivation in MOOCs has been studied using a range of methods, such as surveys, interviews and learning analytics. Past studies investigated a number of factors related to learner motivation, including content-specific interaction with the instructor (Hone & El Said, 2016), the use of video lectures and assessments (Kizilcec et al., 2013), course forums (Balakrishnan & Coetzee, 2013), videos and assignments (Coffrin et al., 2014), course materials, and monitoring participant progress (Hew, 2015), innovative pedagogic strategies and personalization (Bakki et al., 2015), instructor accessibility and peer interaction (Hew, 2016), content, accessibility and interactivity (Deshpande & Chukhlomin, 2017), and interaction and socio-demographic characteristics (Reparaz et al., 2020).

Researchers have explored the role of self-regulated MOOC learning strategies for retention and found that the major difference between course-completers and non-completers were differences in self-regulating learning (Reparaz et al., 2020). Studies have also investigated the role of instruction language and social engagement on motivation to learn. The findings from a study by Miri Barak et al. (2016) indicated that regardless of the instruction, learning in MOOCs was predominately related to intrinsic motivation and self-determination, while there was a positive relationship between social engagement (e.g. number of messages posted in online forums and number of members in online study groups) and motivation to learn.

Learning and retention in MOOCs has been explored in the context of reward-based gamification strategies, as an approach to promote engagement and prevent dropout. Research in this area demonstrated the effects of badges as a type of gamified reward on retention and engagement. The results showed that reward strategies failed to have a significant effect on retention and engagement, but learners who were able to earn badges participated more in tasks than learners in the control group (Ortega-Arranz et al., 2019). In the section below, we outline research results on (open) badges in MOOCs, taking a different perspective: instead of considering badges as gamification elements, we consider them as digital micro-credentials which recognize learning outcomes in MOOCs.

Research on motivation effects of credentials and badges in MOOCs

Open Badges are web-enabled digital tokens of accomplishment (Grant, 2014). Typically, these credentials in MOOCs have been used to recognize accomplishments such as participation, completion of activities and the entire course, as well as learning outcomes such as competencies or skills developed. Open badges differ from conventional credentials because they can contain verifiable metadata about the issuer, the earner, the accomplishment recognized by the badge, the criteria used to award the badge, date of issue, links to the evidence of learning, and verification of the endorser. Furthermore, open badges can be shared easily on the web including social networks where they can gain additional meaning and add value, especially through social validation of the evidence displayed in the open badges. In this way, they facilitate a broader shift in credentialing towards capturing, validating, and recognizing learning, away from measuring and accrediting achievements (Hickey & Chartrand, 2019).

In past years, certificates and credentials such as those enabled by open badges have been studied as possible factors for motivation in MOOCs (Buchem, Carlino et al., 2020; Cross et al., 2014; Jovanovic & Devedzic, 2015; Ortega-Arranz et al., 2019). Research has shown that most platforms use some form of an accreditation system, usually issuing statements or certificates of accomplishment and/or participation. While some MOOCs offer credits toward professional development and verified certificates for a fee, others offer a free statement of participation (Liyanagunawardena & Williams, 2014; Witthaus et al., 2016). Usually the final recognition is obtained after passing some kind of assessment or evaluation. Customarily this recognition has the form of digital certifications, in the form of open badges and/or blockchain-based certificates (Downes, 2019; Ma, 2015; Tovar et al., 2013). Beyond recognition practices related to certification of participation and/or completion of a MOOC, open badges have been used to recognize specific learning outcomes, in the form of micro-credentials, that offer an opportunity to make learning in MOOCs more visible (Lemoine & Richardson, 2015; Pickard et al., 2018; Witthaus et al., 2016). Some MOOCs use third-party badging platforms to increase the flexibility and versatility of their digital recognition, allowing for managing and issuing a greater number of badges, e.g. for specific skills, and for the creation of a badge ecosystems and/or badge pathways (Buchem & Carlino, 2018; Higashi et al., 2012).

Motivation to participate has been attributed to the possibility of obtaining certification, especially that which can be authenticated by the link provided in the certificate, eliminating the possibility of faking the certificate (Rai & Chunrao, 2016). There is also evidence showing that learners who attempt to earn credentials for career advancement or personal interest find MOOCs attractive not only due to the flexibility of learning at their own pace, time and place, but also because of the digital certificate/credential issued upon successful completion of the course (Chaw & Tang, 2019). In the context of a study of 30 new open badge systems, Hickey and Schenke (2019), indicated that open badges may be more meaningful than other forms of credentials especially due to the possibility of sharing and circulation on web-based networks.

Nevertheless, concerns about corrupting motivation due to the use of badges as extrinsic incentives for learning have been raised in the past (Hickey & Schenke, 2019). Research has shown however, that learners see badges less as extrinsic motivators and more as enjoyable and meaningful relevant learning targets that help visualize achievements (Rughiniş & Matei, 2013). Badges can be used to highlight desirable performance in a visual modality and in this way help learners focus their attention, map the curriculum and stimulate the exploration of the learning environment (Rughiniş & Matei, 2013). Furthermore, it has been argued that potentially corrosive effects of badges on motivation, such as undermining intrinsically motivated learning and free-choice engagement, may be avoided if badges are not used as evidence-free incentives, but as tokens of accomplishment with detailed evidence supporting those claims and documentation about how the evidence was gathered (Hickey & Schenke, 2019). A study by Abramovich et al. (2013) concluded that badges may have both positive and negative effects on motivation depending on the design of the badge system and the learner characteristics.

Context and participants

To explore the motivational effects of open badges on learner motivation in MOOCs, we conducted a learners' perception study using an online survey as a tool. The study took place on two different MOOC platforms and with two different groups of participants in 2019 and 2020. The sections below give an overview of the context of the two case studies, including the characteristics of the MOOC participants.

OpenVM Learning Hub

The first study took place in the context of the MOOCs developed in the Open Virtual Mobility project (abbreviated: OpenVM), a three year (2017-2020) strategic partnership for innovation and the exchange of good practices founded under the Erasmus+ program of the European Commission. The initiative's purpose is to enhance the uptake of virtual mobility in higher education by enabling virtual mobility and supporting teachers and students in higher education in developing the skills required for successful participation in and/or preparation of virtual mobility.

One of the key outcomes of the OpenVM project is the Learning Hub¹, an online learning environment for the development, assessment and recognition of virtual mobility skills in higher education. The Hub is a multilingual online learning environment that promotes collaborative learning as an instructional method and provides digital credentials based on the Open Badge 2.0 standard for recognition and validation of competencies. The application hosts a set of eight mini-MOOCs, in eight competency areas. Each one is dedicated to a specific competency cluster needed for successful engagement in virtual mobility. Learners can study at one of three levels: Foundations, Intermediate and Advanced. All modules are offered in English and available under Creative Commons Licences.

The target groups of the OpenVM project include educators and students in higher education. During this phase of the study, most participants in the OpenVM MOOCs were university students from different countries in Europe and other locations. In some cases, participation was part of university curricula, as described by Buchem, Amenduni et al. (2020). At the beginning of 2020, there were 7,300 visitors and 1,500 MOOC participants.

MiriadaX

The second study took place in the third edition of the MOOC "Empower yourself with social networks" (Borrás-Gené et al., 2019) from Universidad Rey Juan Carlos (Spain), within the MiriadaX MOOC platform. The target groups are Spanish and Portuguese speakers, exhibiting a wide range of interests, ages and educational levels. In general, there is a predominance of users with some affiliation with higher education, including educators, staff, graduates and students. Within the course catalogue, there is a wide variety of options from very specific competencies including skills such as personal and professional development. There are two types of completion certificates: a free certificate of participation in the form of an open badge that is obtained automatically when 75% of the course is completed, and a digital certificate of completion, in PDF format, that is obtained upon successful completion of 100% of the course. The second certificate has a fee.

The MOOC was offered in 2019 for five weeks (October 29th to 4th December) and was organized in one module as a learning guide, and four modules in which participants were offered video formats, content to work on personal branding and diffusion through the use of social networks and web 2.0 tools such as blogs and video production. The course helps participants to analyze their digital identity and to be reflective about their digital footprint.

Design of MOOCs and Open Badges

Design of MOOCs and Open Badges in OpenVM

The MOOCs and Open Badges in both studies were designed in different ways. The sections below outline some of the key design features, which are discussed in relation to research results in further part of this paper. OpenVM MOOCs available in the Learning Hub were designed to support learners (both students and educators) in developing the competencies required for successful engagement in virtual mobility. Each MOOC is dedicated to one of the eight virtual mobility competency areas identified in the project: (1) Media and digital literacy, (2) Active self-regulated learning skills, (3) Autonomy-driven learning, (4) Networked learning, (5) Intercultural skills and attitude, (6) Interactive and collaborative learning in an authentic international environment, (7) Open-mindedness, and (8) Open virtual mobility knowledge. OpenVM MOOCs were designed as mini-MOOCs following micro-learning principles related to new forms of acquiring, producing and distributing knowledge in networks (Buchem & Hamelmann, 2010). The design of mini-MOOCs focusses on providing a shorter learning experience and covering less content and fewer skills than traditional MOOCs (Pilli & Admiraal, 2016). OpenVM MOOCs have a modularized structure, loosely coupled short learning activities with integrated granular pieces of content and different forms of e-assessments (such as automated self-assessments and e-portfolios with peer-reviews), as well as a total of 24 digital micro-credentials (Open Badges) which recognize the specific competencies that the learners developed in each MOOC level (Buchem et al., 2019). The design of mini-MOOCs allows learners to accomplish learning in a shorter time, which has been considered as a promising approach for increased engagement, higher completion rates and better learning results (Challen & Seltzer, 2014). The design of the OpenVM mini-MOOCs follows the principles of Constructive Alignment as the learning objectives, e-assessments and digital micro-credentials are aligned with each other through specific sets of skills (Buchem et al., 2019). Upon successful completion of each course level, the learner can claim a micro-credential for the given competency area and level. a1 shows the structure of the OpenVM MOOCs with three levels as self-contained courses.

Figure 1
Design of OpenVM badges with focus on metadata

Source: from "Open Virtual Mobility Brochure. Erasmus+ OpenVM Project", by I. Buchem, J. Konert, A. Poce, C. Carlino, G. Tur and G. Ubachs, 2018 (https://www.openvirtualmobility.eu/wp-content/uploads/2020/04/OpenVM-Erasmus-brochure.pdf).

OpenVM badges are conceptually designed as micro-credentials recognizing the specific sub-set of skills in a given competency area. Technically, OpenVM badges are based on the Open Badges 2.0 standard. As shown in Figure 2, the use of the Open Badges 2.0 standard allows for an embedded metadata field, in which the sub-set of skills in a given competency area is listed. Additionally, OpenVM badges link to the semantic Competency Directory (CD), in which all OpenVM competency descriptions are available in human- and machine-readable format (Konert et al., 2019). OpenVM badges are issued via the Bestr platform leveraging the Badgr infrastructure. The Bestr issuing platform is integrated in the OpenVM Learning Hub via xAPI. All OpenVM badges are listed and displayed on the dedicated Bestr OpenVM project website².

Figure 2
Design of OpenVM badges with focus on metadata

Source: Open mindedness - Foundation Level, Badgr, 2019 (https://badgr.com/public/assertions/UX1vUDv6TfWILN46Kasn4A)

The granular form of OpenVM badges, each recognizing a particular skill at a particular level, creates a unique system of micro-credentials. The design of the badges corresponds to the competency framework for open virtual mobility with the main eight competency areas, each characterized and operationalized by a specific skill set (Buchem, 2018). In order to obtain a badge, the learner must complete all of the required activities and pass all of the e-assessments. For automated e-assessments, a minimum of 50% must be achieved in order to pass. Advanced Level mini-MOOC credentials are issued not only as Open Badges, but also as Blockcerts³, as a way of differentiating the more advanced achievement. Figure 3 shows two examples OpenVM badge-types (three levels per type).

Figure 3
Example of OpenVM badges in three MOOC levels

Design of MOOCs and Open Badges in MiriadaX

MiriadaX offers the traditional tools for a MOOC, however with limitations for adding new elements and adaptations, such as collaboration and motivation requiring third party platforms. The design of the "Empower yourself with social networks" course is based on a hybrid MOOC approach with a high collaborative component requiring the participants to generate conversations and new content (García-Penalvo et al., 2015; Fidalgo-Blanco et al., 2016). In order to enable this feature, it was necessary to create a virtual learning community, using an open group within the social network Facebook. This design stimulates participation, e.g. through volunteering activities in each module, and engagement in the community by sharing learning results.

An open badges model was proposed to reward participants' work for doing additional MOOC activities and sharing them through the learning community. The result was the design of a badge pathway, displayed in Figure 4, consisting of a set of badges that recognize the different activities performed by the participants and could be obtained during the MOOC. This complemented the completion badge and certificate offered as standard by MiriadaX.

Figure 4
MiriadaX badge pathways

Source: authors' own work.

The MiriadaX platform automatically issues an open badge to participants who interact with all elements of the MOOC and properly complete the proposed tests. However, this badge does not reflect the effort of those participants who completed some or all of the voluntary activities. To recognize this work, an additional system of 8 open badges was designed and implemented through the Badgr platform, because of the inability of MiriadaX to configure additional badges in their MOOC. Figure 5 summarizes the designed badge system.

Figure 5
MiriadaX badges

Source: authors' own work.

This badge system was designed using the Badgr⁴ platform, which enables the creation of pathways composed of different badges. This platform also allows the addition of milestone badges that are automatically generated each time a user earns a predefined required badge and allows the creation of a comprehensive pathway with badges added to others. The design of MiriadaX badges combines milestone and single badges to obtain a final badge.

On the Badgr platform, badges can only be issued by the university staff. In order to facilitate participants claiming badges, one Google Form per badge was created. For each activity associated with a badge, the participants had to complete the form with personal information and add the evidence required to obtain the badge. Most of the evidence comprized screenshots of the publication in the virtual learning community performed as an activity. In order to integrate this design into the MOOC at MiriadaX, all of the information was included. Within the MOOC, each volunteer activity rewarded with a badge included a link to its Google Form with instructions on how to obtain it. The participant could accumulate a total of eight badges, five of which were related to volunteer activities in the MiriadaX MOOC, i.e. Twitter badge, LinkedIn badge, Twitter event badge, Facebook Group badge and Digital Identity Plan badge, and one that included the final MiriadaX badge. The earners of the final MiriadaX badge had to upload it to Badgr in order to validate it (MiriadaX validation badge). The MOOC team reviewed each application and, if it satisfied the criteria, the badge was awarded via Badgr. This platform automatically notified the participant that they were awarded the badge and sent it to them by mail. Two additional milestone badges were obtained automatically when earning the previous ones. The two milestone badges that Badgr generated automatically were the Communication Badge, which was issued as a result of participating in live events during the MOOC edition, and the Professional Digital Identity Management Badge, which was issued as a result of obtaining all of the previous badges. This badge includes the final MiriadaX badge, which must be uploaded to Badgr. Figure 6 shows an example of a volunteer activity which was rewarded with a badge in a MiriadaX MOOC.

Figure 6
MiriadaX: A volunteer activity rewarded with a badge

Source: MiriadaX.

Research methods

The research was conducted in a total of nine MOOCs, i.e. eight MOOCs offered in the OpenVM Learning Hub and one MOOC offered in MiriadaX, using two online surveys with an aligned set of items. The sections below give an overview of the procedure, the sample and the survey items applied in the study.

Procedure

Participants in all MOOCs were invited to participate in the online survey designed for the specific audience in each of the two MOOC platforms. Participants in mini-MOOCs offered by the OpenVM Learning Hub were invited to participate in an aggregated online evaluation survey in English, which was used to evaluate various aspects of OpenVM MOOCs including Open Badges. The OpenVM survey was organized in eight sections and the participants were required to express their level of agreement with a set of statements related to OpenVM badges including the five survey items used for the joint study presented in this paper. Out of the 1,412 OpenVM MOOC participants who completed the survey, approx. 20% participated in the Active Self-Regulated Learning, 17% in Media and Digital Literacy, 15% in Collaborative Learning, 14% in Open-mindedness, and 12% in Intercultural Skills MOOCs. Participants in the MOOC offered by MiriadaX used the OpenVM project online survey translated into Spanish. The survey was sent to all MiriadaX MOOC participants once the edition of the MOOC was over.

Study sample

The study sample consisted of a total of 1,788 participants, i.e. 1,412 from OpenVM Learning Hub, and 376 from MiriadaX. The participant profile both in the OpenVM Learning Hub and in MiriadaX was diverse. Table 1 below gives an overview of the distribution of the demographics in both groups.

Table 1
Study sample

Source: data from OpenVM and MiriadaX (2020).

The OpenVM participants were mostly students from 88 different universities in Europe, predominantly from the countries represented by the project partners (Romania, Italy, Germany, Spain, the Netherlands) and other European countries (UK, Lithuania, Sweden, Norway, Slovakia, Bulgaria, France, Switzerland). Participants also included students and teachers from outside of Europe, including the USA, Colombia, Ecuador, Chile, Mexico, Argentina, Turkey, Burkina Faso and New Zealand. Approximately 1,500 participants were enrolled in all eight OpenVM MOOCs at the beginning of 2020. 1,967 badges in total were issued upon completion of the MOOC courses at three levels (altogether 24: 8 MOOCs x 3 levels).

MiriadaX participants included both students and teachers, mostly from Latin America with 54.7% (Peru, Mexico, Argentina, Colombia and Ecuador) and Spain with 37.5%. A total of 2,815 participants were enrolled in MiriadaX, of which 1,547 started (54.8%) and 503 finished the MOOC (32.5% of students who finished the MOOC and 17.8% of enrolled students). In total, 711 badges were issued, 503 from MiriadaX and 208 from the Badge Pathway in Badgr (65 Twitter badges, 45 LinkedIn badges, 24 MiriadaX validation badges, 23 twitter event badges, 22 Facebook Group badges, 14 Communication badges, 9 Digital Identity Plan badges, and 6 Professional Digital Identity Management badges.

Survey items

The online survey was designed with the objective of attracting a large number of respondents through its compact format. The survey participants in OpenVM and MiriadaX evaluated the five statements on a scale from 1 (totally disagree) to 5 (totally agree). The statements were related to learner perception (2 statements), learner motivation (2 statements) and further use of the Open Badges (1 statement). The statements used in the online surveys on both platforms were aligned with each other but semantically adjusted to the given context. The internal consistency of the scale with five items was measured with Cronbach's alpha. The results showed high internal consistency (α = 0.92). The survey items are listed in Table 2.

Table 2
Survey items

Source: OpenVM and MiriadaX.

Research questions

The two research questions guiding the study were: (1) Q1: How did the participants perceive the open badges in terms of their design and conveying the skills developed in the MOOCs?; (2) Q2: How did the participants perceive the motivational effects of the open badges: Did the badges motivate the MOOC participants to a) learn in MOOCs, b) complete the MOOCs, and c) use of open badges outside of the MOOC platforms?

The hypothesis behind the research study was that learners who liked the design of the open badges and thought that the open badges described the skills they developed in an adequate way would be both more motivated to learn and to complete the MOOCs. They would also be more inclined to use the badges outside of the MOOC platforms in which they were earned, e.g. websites, social media. Table 3 depicts the tentative research model used for statistical analysis.

Table 3
Research model applied in the study

Source: authors' own work.

Research results

The sections below outline research results in relation to the two research questions.

Q1: How did the participants perceive the open badges in the MOOCs?

The first research question was concerned with the perception of the open badges by the MOOC participants. Descriptive statistics summarized in Table 4 show high average values, gravitating towards "strongly agree" (value 4 on the scale), for all five survey items, including the first two related to participants' perceptions of the open badges: a) Badge design, and b) Skill description. The results indicate that the MOOC participants in both OpenVM and MiriadaX liked the design of the badges (badge design) and found that they accurately described the skills they developed in the MOOCs in an adequate way (skill description).

Table 4
Descriptive statistics. Scale 1 (totally disagree) to 5 (totally agree)

Source: authors' own work.

To compare the cumulative distributions of the two data sets from OpenVM and MiriadaX, the two-sample Kolmogorov-Smirnov test was conducted. Since p < 0.05 values were achieved for all five variables, it can be concluded that the two groups in OpenVM and MiriadaX were sampled from populations with different distributions. To analyze the mean differences between the samples from OpenVM and MiriadaX, an independent sample t-test was conducted. Levene's test for equality of variances reached p < 0.05 for all variables except for "badge design" and so equal variances could be assumed only for "badge design". Additionally, effect size was calculated using Cohen's d for the comparison between the means of the two groups. The results show that p-values less than 0.05 were reached for "badge design" t (1786) = 2.68, d = 0.12, p < 0.05, and for "further use" t (542.83) = 3.94 p < 0.05, d = 0.33, which means that the means for these variables are statistically different (cf. Table 5).

Table 5
Independent sample t-tests (2-tailed)*

*Equal variances could be assumed only for "badge design".
Source: authors' own work.

The results of the t-test are noteworthy because they show that the MOOC participants in OpenVM and MiradaX, regardless of their different perceptions of badge design and further use, did not statistically differ in their perception of how open badges influenced their motivation to learn, t(513.50) = 1.60, p = 0.11, d = 0.14, and motivation to complete the MOOCs, t(517.02) = 0.47, p = 0.64, d = 0.04, which indicates that in both cases, open badges were perceived by learners as motivating factors. The OpenVM and MiriadaX participants also did not statistically differ in their perception of how well the badges described the skills they developed in the MOOCs t(515.35) = 1.62, p = 0.10, d = 0.14, which indicates that in both cases, the badges properly described the learning outcomes. The values of d < 0.5 in all five variables indicate that the effect size is low. This means that the two groups of learners - OpenVM and MiriadaX - did not substantially differ in their perception of the open badges.

Q2: How did the MOOC participants perceive the motivational effects of the open badges?

To test the hypothesis that learners who responded in a positive way to the design of open badges and thought that they accurately described the skills they developed in an adequate way would be both more motivated to learn and to complete the MOOCs, and would be more inclined to use the badges in further platforms, correlation and regression analyses were conducted using the survey data.

The results of the correlation analysis, summarized in Table 6, show that all of the relationships assumed by the research model are significant at the level of p < 0.01 (2-tailed).

Table 6
Correlation analysis

Note: ** p < 0.01
Source: OpenVM and MiriadaX.

As Table 6 shows, higher Spearman's rho values were found for the variable "skill description". This result of the correlation analysis is very interesting as it shows that what motivated the MOOC participants in both OpenVM and MiradaX were adequate skill descriptions in the open badges. Furthermore, the correlation tests show that positive and significant correlations exist between both the badge design and skill description scores and the motivation for further use of the badges in other environments/contexts. Again here, higher Spearman's rho values were found for the variable "skill description", which means that the badge earners are more inclined to use open badges in other environments if the skills in the open badges are described in an adequate way.

Conclusions and further work

The results of the joint research study presented in this paper show that open badges motivated MOOC participants on both OpenVM and MiriadaX platforms to learn in MOOCs and to complete the MOOCs. Given a large sample of almost 1800 participants, the results presented in this paper are an important contribution to the field of study, indicating that adequate recognition of skills with digital credentials, such as micro-credentials based on Open Badges, may be an important contributor to motivation to learn in MOOCs and for motivation to complete MOOCs. These findings correspond with research showing that low completion rates and losing interest in MOOCs may be a result of certification and accreditation practices which allow students to earn a certificate without any proper engagement in the MOOC (Rai & Chunrao, 2016) and include digital badges or certificates that offer little value to MOOC participants (Jiang et al., 2014).

The results of the study presented in this paper should be interpreted in context of the self-reported perspectives of learners via an online survey. Further studies might combine survey-based methodology with learning analytics. The analysis of data in MOOC platforms could be used to validate the perceptions of learners, especially by focusing on the learning patterns and completion rates. Further studies could also use mixed-method approaches to explore the relationships between open badges, different types of learners' motivations and diverse modes of participants' engagement in MOOCs.

References

• Abramovich, S., Schunn, C., & Higashi, R. M. (2013). Are badges useful in education?: it depends upon the type of badge and expertise of learner. Education Technology Research and Development, 61, 217-232. https://doi.org/10.1007/s11423-013-9289-2
• Albelbisi, N., Yusop, F. D., & Salleh, U. K. M. (2018). Mapping the factors influencing success of massive open online courses (MOOC) in higher education. Eurasia Journal of Mathematics, Science and Technology Education, 14(7), 2995-3012. https://doi.org/10.29333/ejmste/91486
• Bakki, A., Oubahssi, L., Cherkaoui, C., & George, S. (2015). Motivation and engagement in MOOCs: How to increase learning motivation by adapting pedagogical scenarios? In G. Conole, T. Klobučar, C. Rensing, J. Konert, & E. Lavoué (Eds.), Design for teaching and learning in a networked world (pp. 556-559). Springer.
• Balakrishnan, G., & Coetzee D. (2013). Predicting student retention in massive open online courses using hidden Markov models. University of California at Berkeley. http://www2.eecs.berkeley.edu/Pubs/TechRpts/2013/EECS-2013-109.pdf
• Barak, M., Watted, A., & Haick, H. (2016). Motivation to learn in massive open online courses: Examining aspects of language and social engagement. Computer & Education, 94, 49-60. https://doi.org/10.1016/j.compedu.2015.11.010
• Borrás-Gené, O., Martínez-Núnez, M., & Martín-Fernández, L. (2019). Enhancing fun through gamification to improve engagement in MOOC. Informatics, 6(3), 28. https://doi.org/10.3390/informatics6030028
• Buchem, I. (2018). Design patterns for digital competency credentials based on open badges in the context of virtual mobility. Development of generative design patterns for competency credentials. In D. Schiffner (Ed.), Proceedings of DeLFI Workshops 2018, 2018 Lecture Notes in Informatics (LNI). Gesellschaft für Informatik.
• Buchem, I., & Hamelmann, H. (2010). Microlearning: a strategy for ongoing professional development. eLearning Papers, 21, 1-15.
• Buchem, I., & Carlino, C. (2018). Update of milestone 4: Implementation and user-testing of open credentials to recognise virtual mobility skills in the OpenVM Learning Hub. Open Virtual Mobility Erasmus+ (2017-2020). https://www.openvirtualmobility.eu/topics/outputs
• Buchem, I., Konert, J., Poce, A., Carlino, C., Tur, G., & Ubachs, G. (2018). Open Virtual Mobility Brochure. Erasmus+ OpenVM Project. https://www.openvirtualmobility.eu/wp-content/uploads/2020/04/OpenVM-Erasmus-brochure.pdf
• Buchem, I., Poce, A. & Tur, G. (2019). Microlearning in MOOCs. A case study on designing engaging micro-learning experiences in the Media and Digital Literacy MOOC. Comunicación y Pedagogía, 315-316, 7-12.
• Buchem, I., Amenduni, F., Poce, A., Andone, D. Mihaescu, V., Tur, G., Urbina Ramírez, S., & Smitek, B. (2020). Integrating mini-MOOCs into study programs in higher education during COVID-19. Five pilot case studies in context of the open virtual mobility project. In S. K. Softic, D. Andone, & A. Szucs (Eds.), Human and artificial intelligence for the society of the future (pp. 299-310). https://proceedings.eden-online.org/wp-content/uploads/2020/09/Annual_2020_Timisoara_Proceedings_ISSN.pdf
• Buchem, I., Carlino, C., Amenduni, F. & Poce, A. (2020). Meaningful gamification in MOOCs. Designing and examining learner engagement in the Open Virtual Mobility Learning Hub. In Proceedings of the 14th International Technology, Education and Development Conference (9529-9534). IATED. https://doi.org/10.21125/inted.2020.1661
• Challen, G., & Seltzer, M. (2014). Enabling MOOC collaborations through modularity. In Proceedings of the 2014 Learning with MOOCs Practitioner's Workshop.
• Chaw, L. Y., & Tang, C. M. (2019). Driving high inclination to complete massive open online courses (MOOCs): motivation and engagement factors for learners. The Electronic Journal of e-Learning, 17(2), 118-130. https://doi.org/10.34190/JEL.17.2.05
• Coffrin, C., Corrin, L., de Barba, P., & Kennedy, G. (2014, March). Visualizing patterns of student engagement and performance in MOOCs. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (pp. 83-92). https://doi.org/10.1145/2567574.2567586
• Cross, S., Whitelock, D., & Galley, R. (2014). The use, role and reception of open badges as a method for formative and summative reward in two Massive Open Online Courses. International Journal of e-Assessment, 4(1). https://ijea.org.uk/index.php/journal/article/view/72
• Deshpande, A., & Chukhlomin, V. (2017). What makes a good MOOC: A field study of factors impacting student motivation to learn. American Journal of Distance Education, 31, 275-293. https://doi.org/10.1080/08923647.2017.1377513
• Dillenbourg, P., Fox, A., Kirchner, C., Mitchell, J., & Wirsing, M. (2014). Massive open online courses: current state and perspectives (Dagstuhl perspectives workshop 14112). Dagstuhl Manifestos, 4(1), 1-27. https://doi.org/10.4230/DagMan.4.1.1
• Downes, S. (2019). Recognising achievement with badges and blockchain in a Connectivist MOOC. Journal of Learning for Development, 6(3), 273-286.
• Duru, I., Dogan G., & Diri, B. (2016). An overview of studies about students' performance analysis and learning analytics in MOOCs. In 2016 IEEE International Conference on Big Data (Big Data) (pp. 1719-1723). https://doi.org/10.1109/BigData.2016.7840786
• Fidalgo-Blanco, Á., Sein-Echaluce, M. L., & García-Penalvo, F. J. (2016). From massive access to cooperation: lessons learned and proven results of a hybrid xMOOC/cMOOC pedagogical approach to MOOCs. International Journal of Educational Technology in Higher Education, 13(1), 24. https://doi.org/10.1186/s41239-016-0024-z
• García-Penalvo, F. J., Cruz-Benito, J., Borrás-Gené, O., & Blanco, Á. F. (2015, August). Evolution of the conversation and knowledge acquisition in social networks related to a MOOC course. In P. Zaphiris, & A. Ioannou (Eds.), Learning and collaboration technologies (pp. 470-481). Springer. https://doi.org/10.1007/978-3-319-20609-7_44
• Grant, S. (2014). What counts as learning: Open digital badges for new opportunities. Digital Media and Learning Research Hub. https://dmlhub.net/publications/what-counts-learning/index.html
• Haug, S., Wodzicki, K., Cress, U., & Moskaliuk, J. (2014). Self-regulated learning in MOOCs: Do open badges and certificates of attendance motivate learners to invest more? In U. Cress, & C. D. Kloos (Eds.), EMOOCs 2014 - European MOOC stakeholder summit (pp. 66-72). P.A.U. Education.
• Hew, K. F. (2015). Towards a model of engaging online students: Lessons from MOOCs and four policy documents. International Journal of Information and Education Technology, 5(6), 425-431. https://doi.org/10.7763/IJIET.2015.V5.543
• Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCs. British Journal of Educational Technology, 47(2), 320-341. https://doi.org/10.1111/bjet.12235
• Hickey, D. T., & Chartrand, G. T. (2019). Recognizing competencies vs. completion vs. participation: Ideal roles for web-enabled digital badges. Education and Information Technologies, 25(2), 943-956. https://doi.org/10.1007/s10639-019-10000-w
• Hickey, D. T., & Schenke, K. (2019). Open digital badges and reward structures. In K. A. Renninger, & S. E. Hidi (Eds.), The Cambridge handbook on motivation and learning (pp. 209-237). Cambridge University Press.
• Higashi, R., Abramovich, S., Shoop, R., & Schunn, C. (2012). The roles of badges in the computer science student network. In C. Martin, A. Ochsner, & K. Squire (Eds.), Conference Proceedings (pp. 423-429). Carnegie Mellon University: ETC Press.
• Hone, K. S., & El Said, G. R. (2016). Exploring the factors affecting MOOC retention: A survey study. Computers & Education, 98, 157-168. https://doi.org/10.1016/j.compedu.2016.03.016
• Jiang, S., Williams, A. E., Schenke, K., Warschauer, M., & O'Dowd, D. K. (2014). Predicting MOOC performance with Week 1 Behavior. EDM.
• Jovanovic, J., & Devedzic, V. (2015). Open badges: Novel means to motivate, scaffold and recognize learning. Technology, Knowledge and Learning, 20(1), 115-122. https://doi.org/10.1007/s10758-014-9232-6
• Kizilcec, R. F., Piech, C., & Schneider, E. (2013, April). Deconstructing disengagement: analyzing learner subpopulations in massive open online courses. In Proceedings of the third international conference on learning analytics and knowledge (pp. 170-179). https://doi.org/10.1145/2460296.2460330
• Konert, J., Buchem, I., & Stoye, J. (2019). Digitales Kompetenzverzeichnis mit Technologien des Semantic Web [Digital competency directory using semantic web technologies]. In S. Schulz (Ed.), Proceedings of DELFI Workshops 2019 (pp. 61-70). Gesellschaft für Informatik. https://doi.org/10.18420/delfi2019-ws-108
• Lemoine, P. A., & Richardson, M. D. (2015). Micro-credentials, nano degrees, and digital badges: New credentials for global higher education. International Journal of Technology and Educational Marketing (IJTEM), 5(1), 36-49. https://doi.org/10.4018/ijtem.2015010104
• Liyanagunawardena, T. R., & Williams, S. A. (2014). Massive open online courses on health and medicine: review. Journal of Medical Internet Research, 16(8):e191. https://doi.org/10.2196/jmir.3439
• Ma, X. (2015). Evaluating the implication of open badges in an open learning environment to higher education. In 2nd International Conference on Education Reform and Modern Management (pp. 105-108). Atlantis Press.
• Martin, F. G. (2012). Will massive open online courses change how we teach? Communications of the ACM, 55(8), 26-28. https://doi.org/10.1145/2240236.2240246
• McAuley, A., Stewart, B., Siemens, G., & Cormier, D. (2010). The MOOC model for digital practice. University of Prince Edward Island.
• MiriadaX. (n.d.). Our numbers. Retrieved June 2, 2020 from https://miriadax.net/en/nuestros-numeros
• Ortega-Arranz, A., Bote-Lorenzo, M. L., Asensio-Pérez, J. I., Martínez-Monés, A., Gómez-Sánchez, E., & Dimitriadis, Y. A. (2019). To reward and beyond: Analyzing the effect of reward-based strategies in a MOOC. Computers & Education, 142. https://doi.org/10.1016/j.compedu.2019.103639
• Peco, P. P., & Lujan-Mora, S. (2013). Architecture of a MOOC based on CourseBuilder. In 12th International Conference on Information Technology Based Higher Education and Training (ITHET) (pp. 1-8). IEEE. https://doi.org/10.1109/ITHET.2013.6671045
• Pickard, L., Shah, D., & Simone, J. J. (2018). Mapping microcredentials across MOOC platforms. In 2018 Learning with MOOCS (LWMOOCS) (pp. 17-21). https://doi.org/10.1109/LWMOOCS.2018.8534617
• Pilli, O., & Admiraal, W. (2016). A taxonomy of massive open online courses. Contemporary Educational Technology, 7(3), 223-240. https://doi.org/10.30935/cedtech/6174
• Rai, L., & Chunrao, D. (2016). Influencing factors of success and failure in MOOC and general analysis of learner behavior. International Journal of Information and Education Technology, 6(4), 262-268. https://doi.org/10.7763/IJIET.2016.V6.697
• Reich, J., & Ruipérez-Valiente, J. A. (2019). The MOOC pivot. Science, 363(6423), 130-131. https://doi.org/10.1126/science.aav7958
• Reparaz, C., Aznárez-Sanado, M., & Mendoza, G. (2020). Self-regulation of learning and MOOC retention. Computers in Human Behavior, 111. https://doi.org/10.1016/j.chb.2020.106423
• Rughiniş, R., & Matei, S. (2013). Digital badges: signposts and claims of achievement. In C. Stephanidis (Ed.), HCI International 2013 - Posters' Extended Abstracts (pp. 84-88). Springer. https://doi.org/10.1007/978-3-642-39476-8_18
• Shukor, N. A., & Abdullah, Z. (2019). Using learning analytics to improve MOOC instructional design. International Journal of Emerging Technologies in Learning, 14(24), 6-17. https://doi.org/10.3991/ijet.v14i24.12185
• Tovar, E., Dimovska, A., Piedra, N., & Chicaiza, J. (2013). OCW-S: Enablers for building sustainable open education evolving OCW and MOOC. In 2013 IEEE Global Engineering Education Conference (EDUCON) (pp. 1262-1271). https://doi.org/10.1109/EduCon.2013.6530269
• Witthaus, G., Inamorato dos Santos. A., Childs, M., Tannhäuser, A., Conole, G., Nkuyubwatsi, B., & Punie, Y. (2016). Validation of non-formal MOOC-based learning: An analysis of assessment and recognition practices in Europe (OpenCred). EUR 27660 EN. https://doi.org/10.2791/809371

Acknowledgements

This paper is based on the joint work and research conducted by partner organizations in the Erasmus+ Project Open Virtual Mobility, Cooperation for Innovation and the Exchange of Good Practices, Strategic Partnerships for higher education, (partially) funded by the European Union, Project Number 2017-1-DE01-KA203-003494. Disclaimer: The creation of these resources has been (partially) funded by the ERASMUS+ grant program of the European Union under grant no. 2017-1-DE01-KA203-003494. Neither the European Commission nor the project's national funding agency, DAAD, are responsible for the content or liable for any losses or damage resulting from the use of these resources.

The MOOC in MiriadaX platform was funded by the Madrid Regional Government, through the project e-Madrid-CM (P2018/TCS-4307). The e-Madrid-CM co-project is also co-financed by Structural Funds (FSE and FEDER).