Monday, September 28, 2009

Teaching linear algebra

In a recent Hacker News post I made reference to an interesting teaching experience I had in the mid-90s. This is a longer explanation of the same.

I was a graduate student in math at Dartmouth College. I wound up teaching an introduction to linear algebra course that was also the first course where students were asked to do proofs. The class was somewhere in the range of 15-20 students. If I remember correctly, this was in the fall of 1996.

In preparation for the class I set myself goals around how well the students would learn the material taught. After some thought I settled on four ideas that I would use:
  1. Homework not present at the start of class would not be accepted. However students were only graded on the best 20 out of 27 possible homework sets.
  2. All homework sets were cumulative. Generally 1/3 was the current day's material, 1/3 from the last week, and 1/3 from anywhere in the course. Those thirds were in increasing order of difficulty.
  3. Every class would start with a question and answer session to last no less than 10 minutes.
  4. Every student could expect to be asked at least one question every other class.

These ideas may seem odd, but there was a method to my madness. Here is each idea explained.
  1. Homework not present at the start of class would not be accepted. However students were only graded on the best 20 out of 27 possible homework sets.

    The point was to make sure that class started on time, with everyone ready to pay attention for question and answer time. I also didn't want to deal with people doing homework during lecture, evaluating sick excuses, etc. The leniency of not having to turn in 7 homework sets compensated for the rigidness of the policy. And cumulative homework sets meant that I didn't have to worry about students not practicing any given day's material.

    This worked even better than I hoped. The downside was that I had an argument on the second day when someone came in 2 minutes late and was not allowed to turn in his homework. But the first complaint was the last, and the students liked the freedom to decide when something else took precedence over doing homework.

  2. All homework sets were cumulative. Generally 1/3 was the current day's material, 1/3 from the last week, and 1/3 from anywhere in the course. Those thirds were in increasing order of difficulty.

    This was the most important idea I wanted to try. I had long been aware that research on memory had demonstrated that when you're reminded of something as you're forgetting it, it goes into much longer term memory. As a result periodic review at lengthening intervals is very effective in increasing long term recall. A typical effective study schedule being to review after half an hour, the next day, the next week, then the next month.

    Now of course you can tell students this until you're blue in the face - but they won't do it. However when the study schedule is disguised as homework, they don't have a choice.

    This really seemed to work. What I noticed on tests is that students were noticeably shaky on material they had learned in the previous week, occasionally didn't remember stuff for a half-month before that, but absolutely nailed every concept that they'd first learned at least 3 weeks earlier. I credit the forced review schedule from cumulative homework sets for much of that.

  3. Every class would start with a question and answer session to last no less than 10 minutes.

    For me this was the most important part of the class. The questions that came up in this session were my opportunity to refresh people on what they were forgetting, and were how I kept track of what topics should come in for more review on future homework sessions. Given my knowledge of how critical review is to learning, I honestly felt that time spent answering questions was more valuable than lecture. As long as there were questions, there was no maximum on how much time I was willing to spend on this.

    Of course the challenge is getting students to ask questions. My strategy was simple: I told them that someone will ask questions and someone will answer them, but they don't want me to be the one asking questions. On the second day nobody asked me any questions and I had to demonstrate. I picked a random person and asked her to explain a key point from the first day's lecture. She couldn't. I asked another student the same question. Again difficulty. I asked if everyone was sure that they had no questions. Someone asked me the question that I had been asking everyone else. I answered the question, answered the follow-up, and the point was made. I never again had to ask a question during question and answer period. :-)

  4. Every student could expect to be asked at least one question every other class.

    My goal here was to be sure that every student was awake and following the lecture. It was never my goal to embarrass anyone or put them on the spot. To that end I developed a rhythm. Every few minutes I'd stop, say, "Let's make that a question," ask the question, pause so everyone could think through the answer, then ask a random person the question. I made sure to rotate people around so that everyone got their turn fairly.

    The questions I'd ask were always straightforward. They were things like, "What is the result of this calculation?" Or, "Why is this step OK?"

    I treated failure to get the answer as my failures, not theirs. If they couldn't get the answers then they weren't following the lecture, and I needed to slow it down, figure out the rough spots, etc. It might seem that the constant interruptions were slow. But I found that having everyone pay attention more than made up for it. The class as a whole moved as fast as any other class - but with far greater comprehension. And the interactivity made the class become very open about asking questions.

    As a bonus I managed to convince the entire class that taking notes was not worthwhile. I learned this lesson about math in first year undergrad. What you do is read ahead in the textbook. If you really want a set of notes, you can make them from the textbook before class. Then show up at class having read the day's material and ready to pay attention. Then if anything that the professor says doesn't make sense to you when you're paying attention and have already read the day's lesson, then ask the question then and there. If you don't understand it, then probably nobody else does either. Add to that periodic reviews, and you'll have a huge edge in any math courses.

    Nobody ever believes that that works. But this class had no choice because there is simply no way to take notes and pay attention at the same time. Which meant that the note takers couldn't answer questions. But within a few days they learned to not take notes, and I believe did much better for it.

So how well did this package work? As far as my goals were concerned, much better than I had dreamed possible. What really brought this home was the final exam. Based on class performance I drew up a test that I though was a fair test of what I thought they understood. I showed it to some fellow graduate students. They thought I was crazy. They thought the class would bomb, and were willing to bet me on whether anyone would get the bonus question.

The class aced the test. That bonus question? 70% of the class got it. I don't remember what the bonus question was, but I do remember another one that I thought was cute. It went like this. Let V be the vector space of all polynomials of degree at most 2. a) Prove that d/dx is a linear operator on V. b) You can put a coordinate system on V by mapping p(x) to (p(0), p(1), p(2)). (Please imagine that flipped 90 degrees so it is a column.) Find the matrix that represents d/dx in this coordinate system. My fellow grad students got me worried that this might be too advanced for an introductory linear algebra courses. But I needn't have worried - the only significant errors were minor arithmetic mistakes in the calculation. And I think I dinged someone for not having enough detail in the proof.

Furthermore I was lucky enough to talk to some of my students about the experience a few months later. The general consensus was that the material really stuck. Furthermore nobody studied for the final. No joke. As one girl said, "I tried studying because I thought I should, but I gave up after a half-hour because I already knew it all." That is how I think it should be - if you study properly through the course, then you won't need to study for the final. Because you've already learned it. And you'll have a leg up on the next course because you still remember the material that everyone else has forgotten.

So were there any downsides? Unfortunately there were some big ones. I had set goals around learning. I failed to set any around happiness. Having to pay attention during class was hard on the class. Also it motivated them to work hard. Since everyone worked hard and they thought that I was going to grade them on a curve, there was a lot frustration that they wouldn't properly be recognized for their work. (In fact I gave half of them A's in the end.) This frustration showed up the teacher evaluations at the end of the course. :-(

Therefore if I had to do it over I'd ask somewhat fewer questions, hand out a lot more compliments, make it clear that I would not grade on a curve, and if they performed anything like that first class, I'd be even more liberal with good grades. Of course the point is moot since I've found myself profitably displaced from math to software development. But if anyone decides to replicate my experience, I'd recommend paying more attention than I did to those issues.


Unknown said...

>>and 1/3 from anywhere in the course

Was this material already covered? Or any material in the course?



Wang said...

@John: I would presume from the material already covered.

I like the idea of taking notes before class and then just paying attention during class. In general, students learned to take notes any time the professor writes something down and that didn't seem the best way to learn to me even though I was also doing the same thing.

Ben Tilly said...

@John: Homework was only on material that had been covered. But questions on the first week's material could show up on the last homework set. And if they did, you could be sure that they would be the hardest questions available.

rjanezic said...

I'd imagine that the students resisted your methods early on, but it appears that they came to accept, if not appreciate them over time. I like your recipe. You had obviously given much more consideration than most regarding how to effectively convey the material. Have you ever gone back to your students (or have any reconnected with you) to get a sense for 1)how they've done in their career, particularly as it relates to the material you taught, and 2) retrospectively, how they felt about your methods. I'd ask that you make your "experiment" more broadly published, as those methods which are truly and thoughtfully designed to help students "absorb" complex material (rather that simply talk and expose them to it) are those which can help shape a future classes of kids who understand math and other complex subject materials. We, as a country, need more kids who get math, not fight it.
Many thanks.

Peter Goodman said...

Do you still have the problem sets available? I am currently doing an intro to linear algebra course with proofs and I would love some material other than class assignments to work with.

Max said...

I don't think we need more teachers treating students like kids and not letting them turn in HW when they are only 2 minutes late. You deserved to be resisted on that one.

Ben Tilly said...

@rick: Beyond some conversations a few months later, I did not follow up to see how they did later. And it can't be published. All I have now are my memories and no real data. Someone would have to set it up as a proper experiment, and properly measure how the students did against other classes taught with more traditional means.

@Peter: Sorry, I don't have the homework sets. I can't even remember what text I used, though I'd probably recognize it if I started to read it. But all I did for homework sets was pick problems out of the textbook. The trick was that a homework set would be drawn from different sections of the book.

@Max: Being an adult means taking responsibility for your own actions, and accepting the consequences of the same. Believe it or not, adults have to follow lots of rules that they don't like. Frequently those rules revolve around being on time for things. Because, like it or not, wasting just a few minutes of a lot of people's time means you've wasted a lot of time.

Besides which I didn't get significant resistance on that policy. Yes, one kid was unhappy at the start of one class. But on another day 2/3 of the class was happy because they were free to skip their homework to study for a midterm in another course. In the end I got far more thank yous than complaints.

Parag said...

I like your idea of reading the text before a lecture and not taking notes in the lecture.

I also like the thought of building in recall at different points in the course (especially promoting recall when students are just about to forget the material).

Thanks for posting.

Basel said...

Was there a certain reaction from the course coordinators or the dean regarding the general high performance of the class? Sometimes supervisors have certain expectations with regards to class performance, and alarms go off if it goes significantly up or down.

Ben Tilly said...

@Basel: After the final exam I went to the course coordinator and had the following amusement conversation.

Me: "I have a problem with the class that I'd like to discuss."

Her: "What is it?"

Me: "I think it would be easiest if you read the final exam first."

(she read it)

Her: "Your problem is that they all bombed?"

Me: "No, my problem is that they all aced it and I want to give half of them A's in the course."

Her (laughing): "If they passed that test you can give them any grade you want."

Basel said...

That's impressive, imposing discipline is and will always be the best technique. Thanks, you crystallized many thoughts in my mind.

toposphere said...

I have a few questions:

1) To fit in 27 assignments into a semester it seems you would have to give out a new assignment each lecture which was due in the following lecture. Is that correct?

2) What percentage of the total mark was made up by the assignments?

3) How much material was on each assignment? Did you aim for a certain number of questions or a certain estimated time until completion?

4) You talk about both concretely in this situation 'starting the question session with questions' and abstractly 'the advantage of reading ahead and turning up with questions'. Did these two overlap in practice: ie did students turn up with questions on the material to be covered that day? Or was it mainly from the previous sessions?

5) As a follow on, did you enforce or encouraging reading ahead? If so, how? Was it just a matter of saying "Next lecture I want to look at .... ."?

6) In the conclusion you say you'd ask "somewhat fewer questions". Is this in reference to the questions you'd ask students whilst in the 'blackboard teaching' part of each lecture?

Thanks for taking the time to write this up. :)

Ben Tilly said...

Answers for @toposphere.

1. Yes, homework was assigned every lecture and due the next.

2. I forget exactly what portion homework was of the final grade. However I do remember that it was small, but you had to pass the homework to pass the course. Of course that turned out to be a non-issue since everyone did well on the homework. :-)

3. Each day I would come up with a lecture plan, and then I would assign questions based on what I expected to cover. If there was material that I was unsure I would reach I wouldn't ask questions on it - after all I could always ask those questions next time. :-)

4. Questions asked during question and answer time were almost entirely from previous material. However I also got asked questions during lecture from people who had clearly read ahead.

5. I certainly mentioned that I thought it was a good idea to read ahead, but I didn't push the idea. I certainly didn't try to enforce it in any way.

6. Yes, I would have reduced questions asked during regular lecture from the blackboard. I still think asking those questions is a valuable thing to do, but it also stresses the students out, and I don't think I found the right balance.

Unknown said...

I can't find an email address for you so I'll just post here.

I think you have a big opportunity here and you're blowing it by fiddling around in your blog comments instead of writing a whole series of articles on this topic.

Clearly there's a lot of interest in teaching using spaced repetition. I couldn't care less about linear algebra and I'm not a teacher... yet the comment feed of this one blog post in my RSS reader (and it's the only comment feed. the others are regular blog feeds).
Why? Because I'm really interested in spaced repetition learning and you're one of the very few people who's used it successfully. (Maybe some not-on-the-internet people are doing it... I don't care. I live on the internet)

Why not write a whole series on the different elements here? Just off the top of my head I can see you've got
+ planning out the course
+ choosing how to split the lessons up
+ selecting homework assignments
+ running the daily Q&A session
+ specific gotchas to plan for and how to deal with them
+ dealing with topics the students just don't "get"
+ but it wasn't FUN.... (how to make it more fun)
+ ideas for how to tune this idea for teaching non-math topics
+ etc etc etc

Do you realize that you're an authority on this now? Just by posting your success story and answering some questions about how you did it? Who cares if you only did it once... and that was a long time ago. It's still relevant and interesting and you're one of the very few talking about it.

You're making people beg for more information in the comments... why? It's like that TED video with the kid that made his own windmill to power his family's house. When he gets up to talk about it he basically says "yeah, I built it out of parts I found. It was pretty cool"...

We want to make something like what you made... teach us how.



Ben Tilly said...

For @Scott.

I don't see things the way that you do, for reasons I explained in my latest post.

To illustrate why I don't feel I have more to say of interest, let me answer the questions you pose. The pattern is that I generally did things in the most obvious and generic way possible, and don't think I have any particularly valuable insight.

I planned the course by using the textbook used the last time it was taught, starting at the beginning, and going as far as I got. I did this because I was satisfied with the textbook and knew it covered the curriculum for the course.

The textbook was conveniently split into sections which took approximately one lecture each to get through. Therefore I'd approach each lecture by taking the next section, reading it, coming up with an outline for what it would take to present it, and adding in appropriate examples, deciding what to stress, etc. Sometimes I'd do a little more or less than a section, but a section per day was my usual pace.

Each section had a list of problems to choose from which ranged from easy to hard. So I'd grab 3 of the easiest questions from the current section, 0-2 from each section from the previous week, and then I'd pick 3 random topics from the whole course to pick a hard question from. As I selected I marked off the questions in my copy of the text so I didn't accidentally ask the same question twice. My rule for picking hard topics was to prioritize things by whether people showed confusion on it in question/answer, and then by how long it had been since a question was asked on the topic. Honestly it was haphazard, and I probably spent about 5 minutes preparing homework.

After the first day the daily Q&A section ran itself. Because of the homework policy, class started on time. People came with questions, and I never again had a shortage.

I didn't encounter any unanticipated gotchas. The anticipated ones were already covered in my post. For instance I anticipated that question and answer would be weakened by disruptions from people coming in late. Hence a homework policy set up to get class to start on time.

The general principle in math when students don't "get" something is to try to break it down into pieces. One of two things happens. Either breaking it down clarified things, or a particular piece and be identified as trouble. In the latter case you repeat until you identify the real problem. Then reverse the process step by step to get back to the original question, and find whether there are any other misconceptions.

I didn't succeed in making the class fun, so I don't feel qualified to offer advice on how to do so...

As you see, I don't think I have anything particularly surprising to say on any of those topics. But if you wish to continue this conversation by email, my email is

gwern said...

> That's impressive, imposing discipline is and will always be the best technique. Thanks, you crystallized many thoughts in my mind.

But Basel, note that the discipline is just a small part of it. The important thing is the questions and spaced repetition.

Suppose Ben had run the course as a mediocre one, except you failed if you missed a class. Strict discipline, surely, but would his student have learned so impressively? I think not.

Miranda said...

Wow! You seem like such a great teacher! I took linear algebra in college and totally was that kid writing down what was on the blackboard 2-3 minutes behind the prof, doing homework during class, and eventually just skipping class altogether. I got lucky to get a B, but I definitely don't remember anything from that class. Wish I'd had you as a teacher!

Unknown said...

you have rediscovered
what made education work fifty years ago.

we lost it.
even those of us who went through it.

there was so much nonsense around pandering to the students, that we lost even the memory of what worked for us.

your post is the most
wonderful thing, i have read in ages.

Freebird said...

Well done!

I don't know why spaced repetition is not used universally in education at all levels.

Spaced repetition during medical school would create less-stressed medical students who panic less before the big board exams, and would train better doctors.

A couple of fellow medical school teaching fellows and I implemented a series of quizzes and cumulative review sessions for a dense course. For us, it was an obvious choice. These aids got rave reviews from the students. There was a bit of opposition to the quizzes, but this died down as they realized that quiz contribution to their final grade was tiny while the value of consolidating and reviewing information was high.

I still cannot believe that most courses do NOT use these techniques. Actually, I can -- it takes time and effort to implement, and it is seldom required of busy teachers/professors.

ianrhile said...

I used spaced repetition somewhat like this for organic chemistry 2 last spring. Much of the knowledge for organic requires just short problems, so I was able to cycle through problems more than a few times in the course. In general, the student evaluations were great--they loved it. And the students on average did better than I expected.

Unknown said...


When you were giving your lessons, did you write every definition, theorem and proof on the blackboard, or did you use a powerpoint/beamer presentation, or a mix of both?

FrozenWinters said...

I would not have gotten through an undergraduate degree if every course instructor followed this structure, especially re lateness. That policy is so egotistical, inconsiderate, and plainly disrespectful to students. You being a flipping PhD student does not give you the right to demand students adopt your organizational expectations on a daily basis; this has zero relevance to the mathematical content of the course.

e.g. I would probably see what you saw as a 30 min task as something that could be done in 10 min, however organising my time to put it at a specific place would result in this taking 4h.

Unknown said...

This blog post and the comments are very useful. I will be teaching Calculus 1 this semester and am looking forward to using spaced repetition and more Q&A during class. Thank you for your post ... way back in 2009! ;-)

Unknown said...

I am an undergraduate linguistics student looking to do my thesis on spaced repetition in the classroom, and I would love to interview you to learn more about your experiences. Please reach out to me at (I can provide an institutional email, but I don't want to post it publicly) if you are available/interested.