What my 11 year old's Stanford course taught me about online education

[This post was originally published at Forbes.com on 7th May 2012]

My 11 year old son just took a course at Stanford. That has a nice ring to it but it is actually meaningless because these days anyone can take a course at Stanford. You don’t even have to pay. All you need is access to a computer and a reasonable Internet connection. So what we can say is my 11 year old son just watched a bunch of videos on the Internet. 

That doesn’t make for an interesting post except that this ‘bunch of videos’ is currently being heralded as the future of higher education. In the New York Times, David Brooks saw courses like the one my son took as a tsunami about to hit campuses all over the world. And he isn’t alone. Harvard’s Clay Christensen sees it as a transformative technology that will change education forever. And along with Stanford many other institutions, most notably Harvard and MIT, are leaping into the online mix. This is attracting attention and investment dollars. It has people nervous and excited. So I wondered, what happens when someone who has grown up online encountered one of these new ventures?

The course my son just completed was ‘Game Theory’ taught by Matthew Jackson and Yoav Shoham. That wasn’t his first pick (he favored Computer Science 101) but that course was delayed and so he jumped into Game Theory a couple of weeks late

The logo for Game Theory userbox (Photo credit: Wikipedia)

to see how it was. I was more than a little curious, as I’m a game theorist myself. But he agreed to take the course on his own; although I would be on hand to help with some specific questions and occasionally shake my head at some answers he wanted to submit for problem sets. Nonetheless, to a large degree, I could observe when thrown into the deep end of a real University level course, how would a middle school student do? And more importantly, what would he learn?

At the outset, I had modest expectations for this exercise. I expected my son would enjoy game theory. He enjoys thinking about games and strategies. He devoured both volumes of a Cartoon Guide to Economics. And he was interested to learn more about what I did everyday.

Could he handle game theory? Well, at its essence, game theory isn’t too difficult. It is just a way of structuring common sense. I personally had taught some gifted students aged 10 to 16 two days of game theory and knew that there was little ‘life experience’ was going to add to the learning exercise. But that said, the Stanford version wasn’t a course full of fun along with the games. Instead, it was very quantitative. So while game theory can be filled with stories of Cortez and vicious competition between Microsoft and Apple, this one was very, very dry. It focussed on the formal mathematical concepts and shied away from applications. Certainly, none of my MBA students would tolerate it for long and I know from experience that few of them would feel comfortable with the quantitative approach. This was a course that would appeal to engineering types.

What this meant is that you needed a good solid base in algebra and probability in order to keep up with Game Theory. For my son that meant learning some of these on the fly. So he spent more time on the Wikipedia page for geometric series than would a typical student taking this at Stanford. And franky, by the time he got to Perfect Bayesian Equilibrium, he was struggling to understand it deeply. Of course, he should join the club. Few economists have mastered that one. I had worried that some of the maths might prove a barrier to him but it wasn’t enough to stop him from ploughing on.

The lectures

So what of the lectures themselves? Professors Jackson and Shoham had done exactly what you would expect. They had taken the Powerpoints slides from their current course and lectured to a camera with their own disembodied heads popping up from time to time. They had divided the course up into chunks so you didn’t have to block out two hours at a time to absorb a whole lecture. But that was pretty much it.

And so how did my son react to those? He commented, “I think the concepts are interesting but the presentation is dull. Couldn’t they have done animations and things to make it better?” He had a point. Compared to educational videos he had watched these were extraordinarily dull. As I took a look through his eyes, I could see what the problem was: they were exactly like most university lectures. They slowly and methodically take you through the material. Sometimes in a classroom a professor might react to a question with an interesting story or clarifying anecdote. But for these online versions, they were all business. My son asked whether University was really like this and I was sad to say that, in the main, yes it was. More to the point, if you were to ask a current University student what they thought of these videos they would likely think of them as wonderful. After all, this is what they were getting in class but could now pick and choose where and when to attend. They would see the liberation. But for an 11 year old, there was a more demanding standard and, on many levels, a standard worth respecting.

This is when I started to learn more about this medium. The issue is one of design. University lectures are designed to bring everyone along. They have to because you need to build up knowledge and it can’t be easily chunked. This is tolerated when people are in a lecture hall but online for even the average student it is all going to seem somewhat slow.

The most important button for video lectures is not ‘play’ but ‘pause.’ Students can always choose to pause at a point and, say, absorb a slide. What that means is that when you are creating an online lecture, you can build this option in and go fast. This is something that the Khan Academy have already worked out. But that Stanford course was still in the old style. To be sure, you can double the speed (and the pitch) of the lectures but I suspect design when students have the power to pause will look very different. (Click here for more of my thoughts on speed learning).

The significance here has implications for online education. What it tells us is that it is not simply the case that the elite institutions will be able to take their existing courses and pop them online. More will need to be done and lectures will have to be rethought; something I will have more to say about below when we through assessment into the mix. Now the Stanford course did do more than just lectures. It offered some online experiments where you could play games but my son found these disjointed from what was really going on in the lectures. They also offered Google Hangout office hours but as they were during the day they interfered with school so we didn’t see what happened there. But my point is that we have only just begun to learn how online subject matter differs from its traditional counterpart.

The assessment

The Stanford course had assessment although, as with all these things, they were at pains to tell us that this did not constitute official accreditation. Want that from Stanford and you have to get in and show up. For online courses, no one has cracked how to verify whether an identified student is the same person as the one doing the assessment.

Nonetheless, being game theorists, Jackson and Shoham seemed to want to ensure that students had the right push. So they put in place four problem sets comprising multiple choice questions and also a final exam (which was much the same but with a time constraint). As this was a quantitative course, the assessment matched the lectures.

But it was here that my son, taken out of middle school and into a University environment, started to struggle. He had started the course late and so rushed the first few lectures and problem set. That didn’t bode well and we sat down together to see what went wrong. Basically, he was not used to a format that required very careful reading of instructions. For these problems, you had to pay attention to every detail. Miss something small and you would get the answer wrong. He needed to learn something new: how to check your work and read things carefully.

And this worked for the next couple of problem sets until he made a more drastic mistake; one only really possible online. Due to a confusion about precisely what week it was, he opened the final problem set, saw that there was something amiss and rather than just close it, he clicked submit (and also the ‘are you sure’ confirmation). Another bad outcome. The reason was that only your first attempt at the assessment counted. You could try again to improve your understanding but that was all.

Again, I knew why our game theory professors did this. If you want to assess, allowing people to keep trying wasn’t going to get you a good signal of their ability. But then if you think about it for two seconds you have to wonder why we want a good signal of these students’ ability. This is not assessment for accreditation so who cares about getting such incentives right? What one surely wants are problem sets that signal to the student whether they had mastered the material or not. By not breaking out of the assessment mould, the course designers missed the opportunity to focus on learning rather than signalling.

In contrast, this course was very well placed for a Khan Academy type model. In that model, students engage in online assessment and are not permitted to go to the next stage until they have mastered the current one. In game theory, that would mean trying a variety of problems again and again until you got them right.

After the accidently submitted problem set incident, my son was devastated and took a little persuading to push ahead with the course and the final exam. He did that and scored almost 90 per cent for that part. But alas, with two out of the four problem sets a disaster and problem sets receiving the bulk of the weighting, he was a percentage point or two shy of a pass mark of 70 per cent. That said, he was only frustrated by that for a couple of minutes. Another lesson that rules are rules and that sometimes taking care and taking your time is important.

The learning

Assessment and grades aside, let’s focus on the learning outcomes. Did my son actually learn anything from this? The answer is a resounding yes. As I noted the lecturers were rather dry and divorced from the real world, especially his world, but once he started he began to see applications of game theory everywhere. What is more he believed what he was learning could help him understand the world.

Extensive form game 2. An example of finding perfect Bayesian equilibrium. (Photo credit: Wikipedia)

Now in some cases this was very useful to me as it saved some parenting time. On one occasion he decided he did not want to eat a particular vegetable for dinner. We said that he wouldn’t be leaving the table until he did so. I said, “now use your game theory to forecast what the outcome of this is and let me know what you decide.” He laid out his options, our reactions and everyone’s payoffs and worked out that he would do well to eat his vegetables sooner rather than later.

But he took his lessons to more unexpected, and depending upon how you look at it, somewhat disturbing places. At school, students were asked to form teams and go out onto the streets and raise money for kids with cancer. His team had to choose a location and that is where he informed me that he used his game theory. On their street they saw a homeless man (a comparative rarity in Toronto). He realised that the homeless man had already worked out what the best location for charitable contributions was. In this case, it was at a point next to a subway entrance and a Starbucks. He convinced his team that they could set up their own stand right there. Two things could happen, he explained to me. One is that the homeless man, moved away to come back another day. The other was that the homeless man stayed in which case he believed that his team would have edge in their claim as their cause was for other people and his was just for himself. In the end, the homeless man abandoned the post.

Now it is hard to know precisely what to think about that but it is hard to fault his raw logic in terms of the competitive charitable market. The game theory was impeccable assuming that the homeless man was rational. Alas, the Stanford course hadn’t touched on the possible alternative but fortunately some of the more dire scenarios there did not happen. I’m also not sure about relatively well to do kids competing with the homeless but my son had some good arguments in his own defence.

All that aside, this course did actually manage to teach my son some game theory. For that we can (probably) all be grateful. But if you were to ask my whether it is worth getting middle students to take these courses, I would say that there is no need to rush. I’m sure they will get better and better in quite short order so it is best to wait.

The future

In the end, from this exercise I learned that online learning will require considerable investment in time and energy of academics before it really hits the mark. While an 11 year old is far from the right test subject, the notion of taking someone out of their element to see what a learning experience is really like is insightful. Lectures need to be redesigned with pause in mind. The purpose of assessment has to be completely rethought. But when it comes down to it, the subject matter is broadly accessible and one can imagine that younger and younger students will delve into what we previously thought of as University-only course material.

There is much hype about online learning and its potential to disrupt higher education. But all we have really learned by the tens of thousands signing up is that there is tremendous demand. That itself, however, does not spell doom for higher education but should be a sign of its continuing value.

My guess is that online education is more of a complement than a substitute for offline experiences. To be sure, watch something like Michael Sandel’s online course, Justice, and there is no need to attend Harvard for the same thing. But that is because a huge amount of effort went into its design and production. And it is also one where the lecture experience translates online. For courses like Game Theory, the lecture experience is limiting and bringing it online only demonstrates that more starkly.