Lessons Learned from Vanderbilt’s First MOOCs

Posted by Derek Bruff on Monday, August 19, 2013 in Commentary.

by Derek Bruff, CFT Director

When Vanderbilt announced its massive open online course (MOOC) initiative almost twelve months ago, one of the stated goals of the project was to explore ways that digital technologies might enhance Vanderbilt’s teaching mission. As faculty, staff, and students worked over the last year to launch Vanderbilt’s first three MOOCs, we grappled with the following question: How do you create effective learning environments online for thousands and thousands of students? We haven’t fully answered that question, but we have learned much about the design and implementation of MOOCs. As Vanderbilt prepares to launch its next two MOOCs in September and as the instructors of those first three MOOCs look ahead to leveraging their online experiences in their on-campus courses, it’s a good time to reflect on some lessons learned over the last year.

But first, as they say on Marketplace, let’s do the numbers.

Participation Data

Vanderbilt’s first MOOC, launched on March 4, 2013, was Pattern-Oriented Software Architectures for Concurrent and Networked Software (POSA) by Doug Schmidt. The course ran for ten weeks and enrolled about 31,000 students. Of course, “enrolled” doesn’t mean much in the world of MOOCs. It’s a bit like clicking “like” on a Facebook page; it’s no guarantee of active engagement. A more meaningful statistic is the number of what Coursera calls “active” students. These are students who did something beyond enroll–watch a video, take a quiz, visit the discussion forum. There were 23,313 active POSA students.

Of those 23,313 active students, 20,933 of them (90%) watched at least one lecture video, 5,702 (24%) took at least one quiz, 2,072 (9%) submitted at least one assignment for peer grading, and 942 (4%) posted at least once in the discussion forums. Earning a standard statement of accomplishment in POSA required only quiz completion, but earning a statement of accomplishment “with distinction” required quiz completion and submission of programming assignments for peer grading. Neither statement was based on discussion forum participation. Thus, the lower participation rates for peer-graded assignments and forum posts.

How many students completed the course? Of the 23,313 active students, 1,051 (4.5%) earned a standard statement of accomplishment and 592 (2.5%) earned a statement of accomplishment “with distinction” for a total of 1,643 (7%) students earning some form of statement. Of course, one can’t compare MOOC completion rates with those of traditional online or on-campus courses. Since MOOC students neither pay tuition nor earn credit, the motivation for completing a course is largely intrinsic. The statements of accomplishment earned might have value for some students, but they are not equivalent to course credit. That said, it would take maybe 15 years for Professor Schmidt to teach 1,643 students in his on-campus courses.

What about participation and completion rates in Vanderbilt’s other MOOCs? Our second MOOC, Leading Strategic Innovation in Organizations (LSIO) by David Owens, launched the day after POSA, and our third MOOC, Nutrition, Healthy, and Lifestyle: Issues and Insights by Jamie Pope, started up in May 2013. Let’s compare the data across all three courses.

Some context for these numbers:

• The nutrition course had almost twice as many active students as the other two courses. This makes sense, given the course’s broad appeal and lack of prerequisites.
• As with POSA, the peer-graded activities in LSIO were only required for those wishing to earn a “with distinction” statement, which is perhaps why relatively few students submitted work for peer grading in these two courses. The nutrition course had no peer-graded activities.
• Why so much participation in the LSIO forums compared with the other two courses? David Owens, LSIO instructor, encouraged forum participation, building it into the completion criteria and seeding the forums each week with a question that permitted multiple perspectives.
• Earning a statement “with distinction” in LSIO required students to completed a final project as part of a team. This raised the bar considerably, thus the relatively low “with distinction” rate in LSIO. Of course, we’re still talking about over 700 MOOC students who completed team projects!
• In the nutrition course, the difference between a standard statement of accomplishment and one “with distinction” was a matter of the student’s average grade in the course. The “with distinction” track did not require different kinds of assignments, only higher scores on the quizzes.

Participation and completion data are, ultimately, proxies for student learning. So are grades in more traditional contexts, of course. When you see that a student has an A on her transcript for a particular course, you don’t really know what she learned in that course. You would have a better idea if you knew the grading criteria for the course, and an even better idea if you looked at her exams and papers in that course.

However, we’ve seen enough As on transcripts that we can make some safe bets about such grades and what they mean. We haven’t yet seen enough completion rate data for MOOCs to really understand what they mean. Does a 10% completion rate mean that only 10% of the students learned something? Should a rate that low reflect poorly on the course? Or on the students? Or is that rate actually pretty high, given the motivations students have for participating in MOOCs?

For now, a MOOC completion rate is perhaps best compared with other MOOC completion rates–and used as a signal for further investigation. Sixteen percent of LSIO students earned a statement of accomplishment, but only 7% of POSA students. What aspects of the course design or characteristics of the students might have led to that difference? These are the questions our MOOC teams are asking each other as they look at these data.

Lessons Learned

What have the Vanderbilt faculty, staff, and students involved in our MOOC initiative learned about teaching online over the last year? Lots. Since I can’t speak for the dozens of people involved in these first three MOOCs, I’ll focus on a few observations that occurred to me as I helped our MOOC teaching staff think through their design and implementation choices.

I should note that the Center for Teaching plays a supporting role in Vanderbilt’s MOOC initiative. The project was led this past year by associate provost and professor of musicology Cynthia Cyrus, whose herculean efforts in support of this work constantly amazed and impressed me. With the launch of Vanderbilt’s new Institute for Digital Learning, leadership of our MOOC initiative will transition to VIDL’s director, computer science professor Doug Fisher.

And so, some lessons learned:

Teaching online is a team effort. This isn’t news to anyone who has taught online, but it’s still worth putting at the top of my list. Launching these MOOCs required the efforts of faculty, teaching assistants, videographers, instructional designers (like me), educational technologists, librarians, administrators, Coursera support staff, and more. How many person-hours were needed for each MOOC? I don’t know, although I think the Provost’s Office has some estimates. My point here is that whereas an individual professor might feel comfortable putting together an on-campus course with minimal assistance, that model doesn’t work for MOOCs.

There’s more to MOOCs than lecture videos. I don’t think this lesson was fully understood by many here until March 3rd and 4th, when the courses were turned on and the students showed up. Tens of thousands of students, all interested in a well-designed learning experience. Sure, lecture videos are part of that experience, but students wanted meaningful, tractable assignments and both informal and formal feedback on their learning. Our teams had paid attention to such things before launch, of course, but student requests and feedback in those first days made clear that producing high-quality lecture videos was only part of the whole picture.

Open content is our friend. Although there’s more to MOOC than lecture videos, those lecture videos take significant time and effort to produce. One challenge: Given the open nature of videos posted to Coursera, no copyrighted material could be used in the lecture videos without permission from the copyright holder. Our lawyers told us that the educational clause under fair use doesn’t apply here. Every single image in every single slide deck used in our MOOCs had to be checked for copyright status, and many of those images had to be replaced with public domain, Creative Commons, or locally created images. Open content helped in other ways, too: David Owens reported watching a number of free online instructional videos as a way to learn video recording and editing, as he filmed many of his lecture videos himself!

The cognitive diversity seen in MOOCs is far greater than in closed courses. We’re used to teaching courses full of students who are fairly homogeneous in terms of their academic backgrounds and knowledge of the discipline. There are variations, of course, but they pale in comparison to the diversity found within a MOOC. You have students who have never set foot in a post-secondary classroom and other students with advanced degrees. You have students who have very little experience with the course topic and other students who have spent decades in the field. You might tell students that your MOOC is designed for students with certain backgrounds, but there’s no way to enforce that.

How will you handle this diversity? To what extent will you accommodate students with very different backgrounds and expectations? Those are key design questions. Doug Schmidt cast a wide net in his POSA course, providing optional lecture videos on background material and encouraging more experienced programmers in his course to contribute their expertise to a course wiki. Next year, we have a couple of education faculty, Barb Stengel and Marcy Singer-Gabella, offering a MOOC intended for K12 teachers. That’s a fairly narrow audience, as MOOCs go.

MOOC students are well-motivated students. Remember, there’s no credit being offered here, just a statement of accomplishment “signed” by the instructor that arrives in a student’s inbox as a PDF. People take MOOCs largely because they want to learn something of interest or value to them. Contrast this to, say, the statistics course I teach here at Vanderbilt. I might have 50 engineering students in the course, most of them taking the course as a requirement for their majors. They may have generally positive attitudes toward technical courses, but I still have to work to get them interested in statistics. Their motivations for taking the course are largely external, and cognitive science research indicates that such motivations typically lead to strategic, not deep learning. Imagine teaching a course where all (okay, most) of the students are there for the sake of learning, out of interest in the subject. That’s an entirely different kind of learning environment.

(This is also one of the reasons I’m skeptical that MOOCs can be used as course replacements. They work, to the extent that they do, for self-motivated learners. We don’t know how well they’ll work for students taking required courses. Replacing traditionally taught courses with MOOCs seems to me like a risky gamble at this point.)

Cognitive Diversity + Intrinsic Motivations = Crowdsourcing Success. In his book The Wisdom of Crowds, James Surowiecki analyzes a number of crowdsourcing initiatives to determine what separates the successful ones from the rest. He points to the importance of diversity: If everyone in the crowd has the same perspective, there’s no reason to poll the crowd–just ask one member of the crowd. The more cognitive diversity the crowd has, the more they can offer towards solving problems and accomplishing hard tasks. Of course, the members of the crowd need to be willing to participate, to pool their ideas and efforts. In his book Cognitive Surplus, Clay Shirky describes a number of factors that motivate people to participate in crowdsourcing activities, such as feeling part of a community, having one’s contributions acknowledged, and contributing to something greater than oneself.

MOOCs would seem to be natural venues for crowdsourcing, given the cognitive diversity and intrinsic motivations of participants. I mentioned above the wiki that Doug Schmidt set up for his course, in which students contributed a variety of resources and how-tos. As Jamie Pope designed her nutrition course, she was aware that her expertise lay in nutrition in the US context but her students would come from around the world. One way she met this challenge was through crowdsourcing: During the week on food labeling, she asked her students to snap photos of food labels and share them in a thread in the discussion forums. Hundreds of students participated from dozens of countries, creating a resource available to all students in the course–a resource that Jamie’s on-campus students could not have created. Looking to the literature on crowdsourcing has proved to be a great help in brainstorming MOOC learning activities.

MOOC students can be producers as well as consumers of information. MOOCs are about one-way transmission of information, right? Professors film lecture videos, and students watch those videos and take quizzes on the information covered, right? That’s so 2012. Many of today’s MOOC encourage students to take what they’ve learned through lecture videos and readings and apply it to new situations. I’m taking a MOOC called “Maps and the Geospatial Revolution” from Penn State right now, and the final assignment is to use the map-making skills we’ve developed during the MOOC to create a map of our own that tells an interesting story. And, as I mentioned above, the final project in David Owens’ course on strategic innovation was a team project, one that asked students to come up with a creative solution to a problem of common interest. You can see a few of his students’ projects in my blog post from May, “Not Your Father’s MOOC.”

Can MOOC students produce the same kind of interesting, original work possible in more traditional settings with more direct instructor feedback? Possibly not, although smart, experienced, well-motivated students have the potential to make things of value, even in the MOOC context.

Accommodating students on different time tables can be challenging. Some students want to move through the course week by week, others want the freedom to explore material during the entire run of the course, and others want all the lecture videos and assignments right up front. I think all our MOOC faculty had the week-by-week model in mind when they designed their courses, since that’s the model we use for on-campus courses. Who lets their students turn in all the work in a course on the very last day of the course? Okay, maybe a few faculty here and there, but that’s rare. Not so rare in the world of MOOCs. For instance, one could procrastinate to the very last day of the POSA course and still earn a statement of accomplishment, since all the quiz deadlines were set to the end of the course.

Our instructors wanted to accommodate different student pacing as much as they could, although there were some limitations. For instance, peer-graded assignments require students to participate with some synchronicity. All student work must be submitted before any student work can be distributed for peer grading, and all peer assessments must be finished before the results of those assessments can be shared back with students. This is one of the reasons that peer-graded assignments were required only for the “with distinction” statements in POSA and LSIO. One could move through the material at any pace and still receive a standard statement, but achieving the “with distinction” statement required more of a week-by-week pacing. Also challenging given different student pacing: questions of the week. Forum management helped with this, but we still saw students answering Week 2’s question in Week 5 and vice versa.

Instructor presence is important, even in a MOOC. Last fall, as our MOOC project team was considering options for producing lecture videos, several people floated the idea of shooting some classroom footage in the fall for use in our spring MOOCs. I remember our Coursera course ops contact recommending against this. Not only is it usually challenging in a classroom setting to obtain the kind of audio and video quality necessary for a good lecture video, but also reports from earlier Coursera courses indicated that students who watched such videos often felt they were secondary to the instructor’s on-campus students. They didn’t want reheated leftovers, they wanted a meal cooked just for them. The connection that a student feels to the teacher still matters when learning moves online, and it still matters when there are tens of thousands of students involved.

How can a MOOC instructor establish some form of presence in such a course? Virtual office hours were a popular strategy among the Vanderbilt MOOCs. We used Google Hangouts on Air to broadcast live office hours, with video archives available after the fact on YouTube. Students submitted questions via Google Hangouts text chat, and teaching staff responded to those questions on the video. An assistant, typically CFT graduate assistant Katie McEwen, monitored the text chat, fielding questions for the teaching staff. See this video from Jamie Pope’s nutrition course for an example.

Doug Schmidt used virtual office hours, but he also was a very active presence in his course’s discussion forums. As noted above, POSA didn’t have quite the same volume of discussion forum posts as the other courses, which made it possible for Doug to read (what I believe to have been) every single post. And he responded to something like a third of the posts. That’s above-and-beyond effort, but it seemed to help his students see him as someone genuinely interested in their learning–not just someone they watched on pre-recorded lecture videos.

Looking Ahead

This fall, I expect we’ll learn additional lessons, about “flipping” one’s course with one’s own MOOC (all three of our MOOC faculty are planning to do this to some extent) and about offering a MOOC for the second time. Will all these lecture videos allow our faculty to focus class time away from lecture and toward greater student engagement? Will Vanderbilt’s on-campus students benefit from the cognitive diversity among our MOOC students? Will running a MOOC for a second time take much less time and attention–or about the same, given the importance of instructor presence? Will successful students from our initial offerings appointed as “community TAs” enhance the quality of discussion forum conversations? I’m sure we’ll have some answers to these and other questions in the coming months.

We’ll also see how well we can apply the lessons learned from our first three MOOCs to our next three, two of which start next month: “Online Games: Literature, New Media, and Narrative” taught by English professor Jay Clayton and “Data Management for Clinical Research” taught by bioinformatics faculty Paul Harris, Stephany Duda, and Firas Wehbe. I’m expecting the latter course to work something like POSA, since it’s technical in nature and will likely appeal to a relatively small audience. The online games course, however, will likely be a wild ride, with an incredibly diverse group of students and a set of creative, multimedia assignments. After those two courses, we get a short break before our third upcoming MOOC, “Student Thinking at the Core” taught by education professors Barb Stengel and Marcy Singer-Gabella. In addition to targeting K12 teachers, Barb and Marcy are asking participants in their four-week MOOC to work in pairs. I’m excited to see how all three of these courses play out.

For more detail on Doug Schmidt’s POSA course, see this paper he recently co-authored with his TA, Zach McCormick.

Tags: Coursera, MOOCs