Apparently the most popular of my old posts is one I wrote almost two years ago about university exams.

I’ve edited the post a little, and if you didn’t read it yet, you might want to check it out:

A Tirade Against Exams

[…]

So why are exams a bad idea when you want to check whether a bunch of science undergrads understood what you taught them? Well, one part of the problem should be obvious to anyone with even a rudimentary understanding of science:exams are not very good experiments. There is no way to control for interference of irrelevant, extraneous factors. When scientists conduct a study, in any field and with any methodology, they seek to control for irrelevant interferences. For example, when psychologists test hand-eye coordination, they’ll do something like only taking right-handed people with healthy hands and eyes, in order to make sure that the results aren’t skewed by irrelevant differences between individuals.

You can’t do anything like that in exams.

Continue reading »

I’ve also changed the blogs settings so that comments are now open on old posts, too (they used to close automatically after two months). Feel free to rekindle the discussion on the Tirade, or on any other old post.

2. An abdication of responsibility

Grading encourages you to abdicate all responsibility for evaluating your own learning. That’s somebody else’s job. Other people know better about not only what you should be learning but how well you are learning it. This is their game, and it’s your job to play it. Why you would want to learn something, what you would apply it to, what meaning or importance it has for you, what enjoyment you get from it—these are completely irrelevant to your grade. So why even pay attention to these considerations? They are a waste of time. They’re not going to help you pass that next test.

Read the rest over at Lenz on Learning.

• “The Many Benefits, for Kids, of Playing Video Games”
• “Video Game Addiction: Does It Occur? If So, Why?“

One more link: While we’re on the topic of Peter Gray, here’s an interview he gave, very much worth watching:

Blog status comment: Over the past 10 days, I’ve had 4 exams (which, remember, I really don’t like having). Real posting will resume soon; comments are sporadically open, depending on how thick the spam is coming in – but I hope to fix that issue soon, too.

For the BA degree in linguistics, me and my classmates are required to choose some courses from outside of the core linguistics curriculum. This is, in theory, a good thing – it gives undergraduate students a chance to see what’s going on in other departments, and particularly gets us acquainted with some fields related to our own. However, these semi-electives are simply the introductory modules that students in other programs take in their first semesters; this can cause a lot of frustration.

Over the past days, I spent several frustrating hours doing homework in such a course. I remember seeing what must have been the same frustration in students from outside of linguistics in the introductory courses I’ve taken and the one in which I tutored. I think this frustration is an indirect result of the Bologna Process, which creates a basis on which courses from different departments, universities, and countries, across Europe, are evaluated for accreditation. The problem, I think, is that it’s very hard to evaluate a course and the effort that goes into it outside of context.

Understandably, when designing courses, faculty is focussed mostly on training the next generation of scholars in their field. A certain number of students are accepted for each course from outside the field (let’s call them “outsiders”), but they are almost always evaluated in the same way as students from within the field (“insiders”) and, as a result, are supposed to do the same coursework. A part of preparing a future generation of scholars – at least as the Institute for Linguistics and some others seem to view things – is to present beginners with a large amount of hard work so that they can either quickly jump in, or figure out that they chose the wrong field and switch (or leave altogether). However, the motivations, abilities, and interests of outsiders are very different from those of insiders.

In my first semesters, I was in the process of falling in love with linguistics, and this meant I was eager to understand course material and to acquire any new skills helpful for coursework, even when this was difficult. As such, it didn’t terribly bother me that the linguistics modules were tough, or that they required a lot of homework and self-study. I was trying to enter this new world of thoughts, terminology, and ideas, so I wasn’t irked by the fact that I was required to do so. The module that’s frustrating me right now is supposedly a very small one, composed of just one course, in a field I’ve always had some familiarity with and which I find interesting, but which I’ve never been deeply into, nor have any intention of making my professional home. The homework is gruelling, even though I only have to do it every other week, and every single time I find myself kind of furious about it. Yes, I chose this module, but out of a rather narrow set of alternatives, and I have to complete it in order to earn my degree. It may be cast as a choice, but it’s really a requirement.

As I hinted above, I think the problem is a mismatch between the goals and motivations involved in creating the course and those of (some) individuals taking them. When I take an introductory module in linguistics, I am doing so as part of a bigger commitment I’ve made to the field as a whole. I know that if I find the field isn’t right for me after all, I can start an entirely different degree, but I’m willing to accept some parts along the way that I’m not crazy about, since I’m committed to the whole. It also helps that I’m surrounded by a group of people in the same situation. Now, when I’m taking an introductory module outside my field, I naturally approach it in a very different way. The little part is the whole. I’m probably interested in some aspects of the material, but I’m there basically because I need the ECTS points. I’m looking for the interesting things, but the nature of introductory courses dictates that much of what you learn is merely scaffolding for later courses, where the real fun comes. That scaffolding, which could be exciting if I planned to build on it, becomes a terrible chore when I have no reason to expect to ever use it again.¹ As a result, the whole experience becomes one of jumping through hoops, often taking shortcuts, if for no other reason then because there are so many other things I am more interested in doing with my time. And to make things worse, I have no idea who of the many people taking the module is in the same situation, and who is there for the long haul.

All of this would be okay if, say, I merely had to attend the course, with the option of doing homework and taking the exam if I want to get feedback. But module credits are awarded for completing tests, usually written exams.² And in this case, the lecturer only lets you take the exam if you got at least 50% of the points for homework assignments throughout the semester. But the course is not designed for us outsiders – it’s designed for the insiders, who have made a long-term commitment to the field, have a reason to try hard to get good at it, and have a peer group to help them out. The difficulty of assignments and exams is calibrated for them, not for us. As a result, the semi-elective often becomes the most taxing and frustrating module of the semester, even though you “merely” have to pass.

I’m not really sure what could be done about this. I don’t think it would make sense to ask lecturers to go well out of their way to accommodate the small group of outsiders. I do think it’s good that undergraduates get a peek into other disciplines, but I’m not sure that it should be a degree requirement. And as long as it’s a degree requirement, it is understandable that the university wants to make sure people actually take the courses, hence the exams etc. It’s not clear that there’s any real way out of this situation.³

If anyone has any perspective to add, please leave a comment.

Footnotes

1. This expectation may be wrong – you never know where things could come in handy – but it seems, at the very least, highly unlikely that I’ll ever need it again; I can’t help but see it as a chore, rather than a means to an end.
2. I’ve written before about why exams are bad.
3. This kind of problem is, of course, only a problem in institutions which do not fundamentally trust students to take responsibility for their own education. I believe, as with school-level education, that this is not a good design feature for an educational institution. But it would be a mistake to think that universities are mainly educational institutions. Their primary social function is rather accreditation – giving people a stamp of approval so others will allow them into some prestigious jobs and social functions. They educate only as much as they can get away with, unfortunately. And so we are left with the clash of the wish to create some inter-disciplinary cross-pollination, the need to rigorously introduce newbies into your field, and the need of the system not to give away accreditation too easily.

Lately I’ve been having a very hard time accepting the structure of the university program I am in. It’s hard to put my finger on it, but I certainly have not been happy with the requirements this semester.

Over the four semesters I completed so far, I mostly took courses in linguistics. They were not all exactly my cup of tea (only about half of them) and I don’t have to repeat what I think about being tested at the end of every semester, but for the most part I was happy to jump through the hoops, knowing it was progressing my understanding of the discipline and domain of research that I had chosen. My fascination with linguistics and language grew over time as I learned more, understood more, and appreciated new ways of approaching the subject matter. I could accept the expectation that all of us learn a little of all of it, even the approaches we are not interested in pursuing.

However, the way this BA program is designed is a little strange. After the 4th semester, there are no linguistics courses anymore. For the last year of our studies — the year in which we are expected to write our BA thesis in linguistics, mind you — the plan is to take courses from the three different lists of more-or-less elective courses. In total, the program requires 180 ECTS credits throughout the six semesters of study, corresponding to the unrealistic total of 900 hours of class and self-study time per semester. (Hardly anyone at the university, student or instructor, takes this requirement seriously. It seems like something the Bologna process dictates and the universities do their best to fulfill, mostly on paper.)

90 credits — half of the program — are to be obtained in linguistics courses, including 10 credits for the thesis. The other half is composed of:

• 30 credits: courses you get by lottery (from your first few choices) from other departments, university-wide, usually limited to introductory offerings
• 30 credits worth of courses from an “obligatory electives” list, which lets you choose from exactly 70 credits worth of courses from other departments — introductory computer science (20), inter-cultural communication for Russian (10), philosophy of language (10), the languages of Africa (10), the system and history of German (10), or basic Hausa (10)
• 30 credits worth of “key qualifications” courses, being a strange mixed bag of courses offered by different parts of the university on a basis which is not quite interdisciplinary as much as it is simply unrelated to any of the disciplines of those who might take the courses. Luckily, 10 of these credits have to be taken in a language course and the other 20 can be semi-officially replaced by language courses.

I have a feeling this is a case of good intentions gone amiss. There is apparently a social norm of going straight from highschool into university if you were in the academic “gymnasium” highschool system — which you are selected for at the age of 10. As a result, most beginning students have no clue what they’re getting into. So it’s probably doing many students a favor to force them to get a taste of other disciplines before giving them a degree, and indeed the majority changes to another program, or quits, by the second year of studies. But perhaps it’s just cruel, seeing as those of us without wealthy parents have two semesters of grace in which to switch majors, after which financial aid is no longer available.

But I digress. The point is that the structure of this program — not the content — is crushing my interest and desire to complete it. I can’t emphasize enough how this is not a matter of content. I feel like those 80 credits worth of linguistics courses both gave me an excellent, broad understanding of the discipline (and sub-disciplines) of linguistics, as well as giving me a chance to develop real interest in research.

The problem is that the structure of the program makes it entirely impractical to continue pursuing that interest. It’s not just that I have to take some other courses. It’s that a student like me, who is engaged in extra-curricular activity and dependent on financial support, can’t realistically do much besides the required work.

Right now I feel trapped. I am working as a tutor in the introduction to linguistics, and as a research assistant in a language documentation project. I decided to take these jobs both in order to stay involved in linguistics and to work towards more financial independence. I’m very glad I made that choice, and I think it is entirely in line with what engaged and serious students are supposed to do (faculty seems to agree entirely). Yet with all of the time and effort my work requires, I’m struggling to keep up with the computer science coursework, and just desperate to devote more time to reading linguistics literature and perhaps work on some research of my own. (With theoretical grammar as my primary interest, research is thankfully something I can do without any special equipment.)

It makes me furious that in order to receive my BA in linguistics, I am expected to now more or less put my interest in linguistics aside and focus on hoop-jumping.

A note

As some of you may know, as of last Thursday I’m taking time off from my work on EUDEC Council, until the end of 2010. This makes some much-needed room in my schedule for dealing with these requirements. Hopefully it will also give me more time to blog.

I do not expect to make a habit of personal, emotional posts like this one, lacking a clear and general point. It’s just something I had to write about today. At any rate, comments are open and I’d love to hear some of your thoughts on all of this.

Forget What You Know About Good Study Habits

By Benedit Carey

Full article on NYTimes.com

As you can imagine, I read the whole thing on the spot. I fully recommend the entire article, and it’s not long.

I’d like to comment on a few things in the article. I’ll quote them in the order they appear:

Science and the school system

“We have known these principles for some time, and it’s intriguing that schools don’t pick them up, or that people don’t learn them by trial and error,” said Robert A. Bjork, a psychologist at the University of California, Los Angeles. “Instead, we walk around with all sorts of unexamined beliefs about what works that are mistaken.”

Take the notion that children have specific learning styles, […] In a recent review of the relevant research, published in the journal Psychological Science in the Public Interest, a team of psychologists found almost zero support for such ideas. “The contrast between the enormous popularity of the learning-styles approach within education and the lack of credible evidence for its utility is, in our opinion, striking and disturbing,” the researchers concluded.

[…]

[…] many study skills courses insist that students find a specific place, a study room or a quiet corner of the library, to take their work. The research finds just the opposite. In one classic 1978 experiment, psychologists found that college students who studied a list of 40 vocabulary words in two different rooms — one windowless and cluttered, the other modern, with a view on a courtyard — did far better on a test than students who studied the words twice, in the same room. Later studies have confirmed the finding, for a variety of topics.

These paragraphs (emphasis mine) show a recurring theme of the article: the school system has not been learning from science. This is, indeed, “striking and disturbing”. But I can’t say I’m surprised. In my encounters with “education sciences” in Germany, I have to say I did not get the impression that they are very scientific. As my friend Sören Kirchner of tologo often remarks, they seem to be more in the business of reinforcing a philosophy than that of empirical science. Because “education sciences” are wedded to the traditional school system (in Germany at least, the educational sciences faculties are where accredited teacher training takes place) they typically seem rather unmotivated to produce true criticism of the beliefs that drive the traditional system. The truly critical — and I am glad to know a few such people in the faculties of a few German universities — are the exception, not the rule.

The traditional school system has stuck to the same basic paradigm since it was conceived during the Industrial revolution. Society is deeply invested in that paradigm, since the vast majority of us have been through that system, and rejecting the validity of their assumptions about learning means rejecting the validity of how we spent many (unpleasant) hours in childhood. Making that kind of concession is not easy. At this point, improving education is a matter of revolution, not evolution.

Context, relevance, context!

“What we think is happening here is that, when the outside context is varied, the information is enriched, and this slows down forgetting,” said Dr. Bjork, the senior author of the two-room experiment.

Varying the type of material studied in a single sitting — alternating, for example, among vocabulary, reading and speaking in a new language — seems to leave a deeper impression on the brain than does concentrating on just one skill at a time.

I will have to remember this next time someone tells me that students in a democratic school won’t learn anything properly because they aren’t forced to stick to a topic for 45 minutes in a static context. This research strongly suggests that the constantly changing, dynamic atmosphere of democratic schools is a terrific boon for learning. This seems right in line with the thought that having a relevant context is crucial for learning: when you learn something because it is interesting and relevant to you at that moment, you learn it better. Classrooms have a hard time providing that kind of context. A school where students explore things freely allows that relevance to happen all the time.

Exams, revisited?

[…] cognitive scientists see testing itself — or practice tests and quizzes — as a powerful tool of learning, rather than merely assessment. The process of retrieving an idea is not like pulling a book from a shelf; it seems to fundamentally alter the way the information is subsequently stored, making it far more accessible in the future.

Tests are a learning tool? I guess you really do learn something new every day! I wrote a little about testing in July, and indeed as most people do, I treated testing as mere assessment. I stand happily corrected.

In one of his own experiments, Dr. Roediger and Jeffrey Karpicke, also of Washington University, had college students study science passages from a reading comprehension test, in short study periods. When students studied the same material twice, in back-to-back sessions, they did very well on a test given immediately afterward, then began to forget the material.

But if they studied the passage just once and did a practice test in the second session, they did very well on one test two days later, and another given a week later.

It’s good to know that testing can actually help you learn things, but if this is their usefulness, this is not reflected by the way they are treated in schools and universities. I can only emphasize what I’ve said before: the importance of exam grades must be abolished. Then perhaps tests can be useful. Making test grades important only encourages the kind of learning that gets forgotten.

Like this post?

Flattr it!

I

The term “democratic school” has always seemed problematic to me. It’s problematic because democracy isn’t really the point. Democracy is a tool for creating something else: a community where free learning is possible, as much as such a community is possible. All democratic schools should be run by a democracy, but not every school that is run democratically is automatically a democratic school.

A democratic school is a place where students are responsible for how they use their own time. It is a school which does not try to encourage students, explicitly or implicitly, to take classes and tests. It is a place where people are treated with respect, and know they can expect justice to be served when someone disrespects the community or an individual.

II

It just so happens that certain styles of democracy serve as excellent tools for upholding freedom and respect. However, it’s very easy to get it wrong, which is why Sudbury schools are very insistent on getting it right. These schools set up very well-defined democracies, because democracy is only good so long as it does not overreach — it has to be there to protect students’ freedom in the present, without presuming to know what choices are better for their future, or infringing on the privacy of their feelings.

III

Incidentally, the word “republic” comes from the Latin res publica, meaning “public matter”. This hints at a very important idea: the polity (the state, the city, the school) is a public institution, and is something you keep separate from private things.

Sudbury schools use a Judicial Committee which focusses on whether school laws were broken (not on why, or what the individual is going through personally). Some in the free school movement express uneasiness about this seemingly severe approach to justice. However, anyone who has spent some time in such a school knows it is a good thing. Judicial Committee deals with the public aspect of disputes — disrespect of community decisions in such a way that bothered someone enough that they fill out a complaint. This process ignores the personal aspects completely and intentionally.

However, it leaves plenty of room for individuals to address these aspects on a truly personal level. And these are things that come across better when they’re truly and sincerely personal (like talking about problems at home, or about issues one is having with the school or with people there). The judicial process may not directly address the problems that lead people to break community decisions, but it does help others see the problem, which allows them to deal with it. And on the upside, it respects people’s privacy — sometimes you don’t feel like telling just anyone about how you feel.

IV

There are other benefits to separation of the public and the personal. When the community has accustomed itself to this habit, democratic meetings work better — being warned by the Chair is a technical issue, not a personal thing you have to get annoyed about; you can argue strongly against a friend’s motion without them taking it as an insult; every member of the community can apply their thinking to the process as much as they’d like without constantly worrying about the conclusions being taken the wrong way.

V

When a democracy protects the community’s interests and the individuals’ interests while keeping them separate, that democracy can create a democratic school. It can create a place where students develop freely and learn to direct their own learning and gauge their own success. It empowers students to determine their own direction and participate vigorously in community life.

None of these things are automatic, and protecting them is half the secret of success for those democratic schools that have succeeded.

Like this post? Flattr it!

Summer break has just begun. I managed to get away without any exams this semester, for the first time. In the past weeks, like every end of semester, I find myself thinking what an awful, ridiculous system these exams really are, especially in university. I’d like to try and articulate why.

I can imagine a university where exams are hardly even relevant because people only study things they find interesting, and only so long as they are interested. Such places exist (take Tokyo Shure for example).

However, most officially-recognized undegrad programs are still based on instructors providing students with pre-packaged chunks of information, and then judging whether each student has properly digested the information. This post will be about exams in that context; my point of reference will the linguistics BA program at the University of Leipzig. As far as I know, it’s as good an example as any of a normal undergrad program in science.

Exams are bad experiments

So why are exams a bad idea when you want to check whether a bunch of science undergrads understood what you taught them? Well, one part of the problem should be obvious to anyone with even a rudimentary understanding of science: exams are not very good experiments. There is no way to control for interference of irrelevant, extraneous factors. When scientists conduct a study, in any field and with any methodology, they seek to control for irrelevant interferences. For example, when psychologists test hand-eye coordination, they’ll do something like only taking right-handed people with healthy hands and eyes, in order to make sure that the results aren’t skewed by irrelevant differences between individuals.

You can’t do anything like that in exams. For example, one of my exams once took place at a time when I was infatuated with someone. I spent about a quarter of the exam staring into blank space and thinking about things quite unrelated to linguistics. As you might expect, my grades for that semester were not spectacular. This was not a reflection of how well I understood the material in question, but rather a reflection of how capable I was of concentration at the time of the exam.

Exam stress: an antidote for learning

Not only can’t exams control for interference, they create a strongly interfering, totally irrelevant factor: stress.

Exams cause those who take them to get stressed out, usually for weeks in advance.

Google the words “stress” and “learning” together. The first result I got (of some 25 million) was this site, which says “Stress can disrupt learning and memory development”. Huh. That sounds like a great way to lower people’s performance on a test.

One obvious remedy is to train people so they’re used to taking tests and don’t get so stressed out. This is what traditional schools do, and perhaps why they do it.

For some reason, that really doesn’t work for most people. I’m guessing that the way schools make a big deal out of exams rather trains people to think exams are a big deal and worry about whether they’ll pass. What also doesn’t help much is that the resulting grades are relevant to one’s progress in a degree program as well as one’s chances of getting accepted for further studies or a job.

Exams are bad science

But even if we accept that it’s schools’ job to prepare students for the stress of university exams, what are those exams preparing them for? Surely, it can’t be their future work as scientists. Exams are good preparation for bad science.

A scientist’s job is essentially the opposite of exam-taking.

Exam-taking is swallowing a more experienced person’s presentation of information (course material), then regurgitating small bits of it as closely as possible to the original (“the right answers”). Science is carefully considering information (raw data) and other people’s presentations of information (prior work), carefully deciding whether or not to swallow it, then, optionally, producing a novel presentation of the information (research), which is considered useless if it’s in small bits that are exactly like they were when you got them.

The whole idea of one person telling the beginners how it is and expecting them to accept it is bad science. Obviously, my instructors are far more experienced and knowledgeable than me in their respective fields. Still, it would not be very good if I accepted everything they taught me unquestioningly.

If I take my role as a budding scientist seriously, I should critically examine everything I am taught and decide for myself whether I agree or disagree (and why). Exams tell me the opposite, and it takes real effort to continue thinking critically while I am expected to soon be able to reproduce the instructor’s view.

Worse still, in some introductory courses, the theory being taught is not perfect: instructors use simplified or “toy” versions of the theories being taught, or perhaps just a rather recent theory which is more a work in progress than the final word about anything. Either way, attentive students might notice inconsistencies or incoherences. This is good for undergrads; they can be inspired and take the theories further. That value is diminished by needing to swallow theories whole for a test.

Some suggestions

I could probably think of another point or two against exams, but instead I will dedicate the end of this post to pointing out a few things that might make the situation better:

Abolish the importance of exam grades.

This is the most important thing, but also likely the most difficult. Exam grades should not be available to anyone but the student and instructor. It might make sense to indicate on a degree whether the holder’s grades were consistently above average — this might potentially be an indication of extraordinary ability. But knowing that even the best exams are inaccurate and susceptible to extraneous variables, it does not make sense to prefer B students to C students.

Make feedback the goal of all exams.

Finding out I got a C on an exam doesn’t help me improve. Telling me what the weak and strong points of my exam were, could. I learned a lot on the few occasions where I’ve asked an instructor to go over the exam and tell me what my mistakes were. This value as a learning tool is wasted by not presenting all exam takers with feedback. (Some instructors do this, but all should.)

Make some or all exams optional.

If the goal of exams is to give feedback, then save it for those who want it. Mandatory exams create unnecessary stress. There are plenty of other ways to run a system like the modular European Credit Transfer and Accumulation System, in which it is essential to judge whether a student really took part in their courses.

Replace some exams with real work.

Writing a term paper takes more effort than writing an exam, but you learn new things from it and experience something akin to actual academic work. Some disciplines have other “simulations” of real work which could be graded as tests. Sure, this requires more effort per test from the staff, but grading something other than an exam may be a welcome change. And perhaps a system could be created where more advanced students grade beginners’ work and get graded for their grading work (being real academic work practice itself).

Filter students in conversation, not testing.

I get the impression that one of the main reasons I had to take so many exams in the first year of my studies was to filter out students who are not really interested in the program they chose. (I’ve mentioned before that some people choose their major at random here, and if that’s as common as I think, filtering is a good idea.)

I imagine a ten-minute conversation with each student after their first semester could replace some or all of that testing. If the courses didn’t do the trick, simply asking the students if they want to continue with this major, and if yes then why, will get them thinking about those questions themselves. With all of the second chances people are given after failing, it’s their choice anyway; a short conversation could save a lot of exam creation, administration and grading. And of course, this could be done by advanced students as well as by faculty.

Clearly, all change in the university system is slow. Certainly, there are many different changes that can be made. I hope I have provided a few good points of critique and a few good ideas on how to improve the system. Further ideas, comments, and criticism are most welcome in comments.