Evidence-informed Teaching: John Hattie’s Visible Learning for Teachers: Maximizing Impact on Learning

Teachers are professionals. I believe that but I suspect a lot of people don’t because I often hear people talk about teachers in a way they would never talk about dentists, doctors, lawyers, or chefs. But you know what, I think it is largely the fault of teachers themselves. Do you need proof? OK, look at the list below and put the teaching factors into one of two categories: very effective,  and not really effective. Then take a look at the bottom of this post to see how you did.

  • whole language
  • ability grouping
  • homework
  • using television
  • feedback
  • teaching  strategies
  • peer tutoring
  • repeated reading programs

As it turns out, the top four have been found to be not very effective at all and are actually among the least effective of 150 interventions/approaches that have been examined in the now-up-to-900 -plus-study meta-analysis that John Hattie has been engaged in for the last few years. The bottom four are among the most effective. I think teachers and non-teachers alike have difficulty with this task and even teachers who managed to complete the task successfully probably didn’t do so with confidence.  That is a problem. Data on successful and unsuccessful approaches and techniques are not widely known and/or accepted it seems. John Hattie is trying to change that. In his wonderful new book, Visible Learning for Teachers, he not only ranks 150 interventions/approaches in terms of effect size, but he interprets the key results, showing how they affect the business of learning.

In this book, he talks about the implications of these research results on teaching. He is not suggesting some new technique for teaching. Instead it is a new mindset that he is recommending, actually a series of mindsets that have the power to make teachers and teaching much more effective and much more professional. It is in places  buckets of icy water aimed at the faces of teachers themselves. Yet he carefully stresses the importance of the role of teachers in improving education and in doing so empowers teachers–or rather encourages teachers to empower themselves–in a way no book or program I’ve ever experienced has. The main part of the book looks at all aspects of teaching, from preparing and starting the lesson, to giving feedback and ending the lesson. The emphasis is on learning, not teaching, however. And that is one of the mindset changes he advocates. Everything a teacher does must come with questions: Did this help my students learn? Did it help everyone? How much? Teachers need to shift their thinking away from the day-to-day tasks of getting the job done, toward a belief that their fundamental overriding task is to evaluate the effect of their teaching on students’ learning and achievement. Teachers also need to believe that they are the change agents above all others. Success or failure in student learning is the result of what teachers do (or do not do). Other environmental/situational factors are what they are. It is up to each teacher to work with what they have and maximize the effect they have. This is an important shift.

In my job as a teacher trainer I often get sucked into a trap Mr. Hattie warns about. Teachers always want to talk about teaching rather than learning. It is important to keep the eyes focused on learning and see everything through that lens. Everything at the school–the classroom, the staff room, the administration–must all be focused on maximizing learning. Assessment is not just for the students; assessment is how teachers know what their impact is, how administrators know what is happening, how parents and the public can see what is happening. Knowing the effects of education/interventions/policies/approaches, discussing the effects, and displaying the effects are how to improve the current situation. This will require mindset changes for learners, their parents, teachers and administrators. But the beauty of this system is that there is a clear purpose and it is informed by data. That is something to rally around.

The research in the book provides merely a rough guide, however. The studies are from a wide range of subjects and schools and countries and you will likely find yourself wishing that it was more specifically aimed at your particular discipline. There is plenty of room to tweak things for your own specific situation. But the data are a good challenge to anyone wedded to a certain approach for no other reason than habit. Some approaches have been clearly shown to be better and need to be given more prominence in any program. Everything is up for discussion, however,  and through carefully monitoring learning and having an open dialogue better classes, better courses, better schools and better, more professional teachers can be realized.

I highly recommend this book. It is an important book. It will make you think and re-assess your position on the way you approach your craft. It is a book to be read by teachers, discussed by teachers, and referred to when making decisions on teaching and policy. This book has the power to make us teachers stop whining and making excuses. We are the agents of change, data-informed, positive change.

 

According to Dr. Hattie’s calculations, anything intervention with an effect size of 0.40 or higher is very positive and has thus demonstrated a benefit to learning. Here are the effect sizes for the approaches/interventions listed in the beginning of this post.

  1. feedback (0.75)
  2. repeated reading programs (0.67)
  3. teaching  strategies (0.62)
  4. peer tutoring (0.55)
  5. homework (0.29)
  6. ability grouping (0.12)
  7. whole language (0.06)
  8. using television (-0.18)

 

 

 

 

A Month and a Half of MOOCs

I enrolled myself in two MOOCs through Coursera.org., one on gamification and one on statistics. I wanted to experience a  MOOC. I wanted to see how it compared to the self-directed exploration and research I do regularly; specifically, I wanted to see if following a structured approach with 70,000 other people could be beneficial.

These two courses ran at roughly the same time, with Gamification starting a week earlier. Gamification demanded about three hours of time each week, more later when some of the assignments got longer. Statistics demanded more than double (triple?)  that from the first week. The lectures were longer, the concepts more difficult, and the assignments required working with some rather non-intuitive open-source software called R. I found it impossible to continue both courses at the same time. I stopped trying to stay current with the syllabus and just downloaded the Statistics videos and assignments, planning to return to them when Gamification is finished. And that’s what I’m doing. It feels strange, like starting a marathon a few hours late, but it still works.

Which brings me to my first point: MOOCs are as course-like as you make them. If you want the experience of an online course, there are boards for discussion, meet-up groups, wikis, and Facebook groups you can get really active in. You can also just follow along with the videos and the assignments at your own pace. There is actually a lot of flexibility. Professor Werbach who taught the Gamification course  mentioned that only about 12% of the enrolled students were actually handing in the assignments. There are a lot of lurkers. There are a lot of participation options. And that’s not a bad thing, I think.

Next, I really didn’t have high expectations for the courses as e-learning experiences, as courses. But the ones I took were much more interactive and responsive to students than I thought they would be. I thought they would be more canned. I thought they would be finished products produced months before the actual delivery. But instead I realized that they we being constructed–and adjusted–as the course went on. There were very real changes made in response to technical problems (system outages, for example) and content challenges (extra instructional support added on the fly when learners found some things especially difficult). There were e-mail updates and previews, messages of encouragement, bits of behind-the-scenes information, and  Facebook and Twitter participation by instructors.

But were they worthwhile? I have to say yes. They were more challenging than I expected. The lectures were video-based and re-created the feel of a professor and a powerpoint (some room for improvement there, I think, but not a problem), and the assignments were well designed in that they really required application of course content. Gamification relied on quizzes and multiple peer assessments of assignments–a logistically ambitious and mostly successful idea. Statistics relied on quizzes, and assignments that generated a lot of engagement and interaction as learners scrambled to complete them and help each other. But there were things that I knew and couldn’t skip, and things that I wanted to look at more in depth but had to move on from. Course forums and wikis helped a little with this, but an extended reading/resource list would have been helpful. Gamification provided one but it was at little shallow. the biggest “problem” was perhaps the focus of the course. There was a small difference in focus between the courses and my own interests. I wasn’t in Gamification to learn about business, for example, and I would have preferred a more educational focus. But that’s a minor point.

Compared with online courses I’ve taken before, I felt more like I was going through it alone because there weren’t as many regular discussions/assignments with interaction among a group of people you get to know virtually. That was obviously missing. But as I said earlier, there were forums and Facebook, etc. interactions that I chose to not get active in. The MOOCs were more limited in the range of interaction, but they did not compare that unfavorably.

Bottom line: these MOOCs were a great resource and I plan to take more. I’ve been recommending them to everyone. I think they represent something potentially very exciting. A recent presentation by John Daniels (via Tony Bates) lists up some of the potential effects MOOCs could have. I recommend the report and trying a MOOC yourself.

Confidence No Confidence

The chart above comes from a recent MEXT report on emotional states of mind of high school students. They asked high school students in four countries about their opinions of themselves regarding their confidence and value as a person. The results are truly scary–if you are Japanese or have an interest in Japan. I earlier wrote about an Adobe survey that showed everyone in the world thought the Japanese were creative, except the Japanese themselves! Well, this survey, which I found out about through a Rocket News blog article that outlines the major findings, is even more dramatic. You really need to read it. Japanese students are well behind the norm for the region. The graph above shows the answers to the question of whether the kids think they have value as a person.  Do you have value as a person? 89.1% of American kids feel they do 57.2% feel strongly that they do. The Chinese kids were not far behind at 87.7%, and 42.2% believing strongly in their value. The Korean kids were more reserved at 75.1% overall but only 20.2% believing strongly in themselves. Only 36.1% of Japanese kids thought they had value, and only a small number (7.5%) thought so strongly. We can also view this from the opposite side: more than 60% of students think they have little or no value as people! That is not a small number. It mind-numbing! It sends shivers up my spine.

 

MOOCs

I would very much like to do my doctorate. I have several ideas and have spent  time scrounging around on the web for some place where my budget of time and cash and my schedule can all be accommodated in a blissful combination, and the fetters of my everyday responsibilities will fall away and I will be able to concentrate on learning and writing with a chorus of heavenly voices as BGM. Yup, it’s a fantasy. I know it. It just ain’t gonna happen. I’m too busy and too poor and too invested in my family to commit myself to an Ed.D. program at this time. So I just study and research by myself. I’m on my own, with my books and my copied articles, and my PDFs on my ipad, and my hop-scotching from topic to topic as I follow my inclinations without discussions or deadlines.

But now there is a new game in town. MOOCs, or Massive Open Online Courses, are crawling out of the academic woodwork and getting all sorts of attention. Game-changers, they’re being called by some enthusiastic folk (especially the people involved in the three most prominent at the moment: Coursera, Udacity, and Edx). Of course, there is dissent as well, this being the internet and all (this article sums up the pros and cons pretty well). They don’t really level the playing field and allow wider access–real access–to higher education. Well, they kind of do, but point taken. They are not real courses with interaction with classmates and the professor! Yup, with thousands of people enrolled,you can’t expect to actually get to talk to anyone. They are just advertising for big name unis! Probably true, at least partially. I don’t care that much. The fact is that they are a niche right now for people like me who want to learn, who are learning on their own anyway, to add a little structure and direction to their studies. And that’s a good thing. The world is like that anyway now. You can learn almost anything you want on your own; but you only get a degree if you pay and work at it long and hard enough. It is the way it is.

So, I’ve enrolled myself into two courses at Coursera. I want to give it a spin. I’m hoping that I learn something, something that I may have overlooked if if I tried to learn the subject on my own. Anything good that emerges from the experience will be welcome, and unexpected, and precious. I know, like everyone knows, that MOOCs are not the answer to the problems of higher education. But they are a nice gesture, from a lot of people who don’t really need to make such gestures. For the time being, I’m still thinking of it as a slightly less lonely version of learning alone.

Demotivating

Any English teacher in Japan can read the following quotations and identify with the ideas:

…during the years between elementary school and high school, many students disengage from English and don’t regain their interest—to the detriment of their later schooling, and even their adult careers…

…by [the end of the first year of middle school], the pupils described English as less valuable than they had [at the beginning of the year], and reported that they were investing less effort and persistence in the subject than they had before…

Except that these quotations were not taken from a study on student attitudes and motivations toward English in Japan, but rather math in Australia. I just lifted and altered the quotations from a recent Mind/Shift blog posting that introduced a large study done in Australia by a University of Sydney professor named Andrew Martin. He and his colleagues looked at 1601 learners in 200 math classrooms in 33 schools. The similarities to English in Japan are powerful, however.

The two graphs above are from a 2009 study done by Benesse in Japan. The top one shows the results of a question that asked when learners in jr high felt most positive/motivated to learn English. Look at the two first numbers, the second highest and the highest on the graph. The first one is “just before classes started in jr high 1st grade” and the second one is “right when classes started in jr high 1st grade”. By summer of first year, only three months after beginning formal English education (these learners did not have regular/formal English classes in elementary school), the course/curriculum/teachers/textbook had managed to obliterate more than 90% of the delicate innate motivation these learners were feeling. And they never got it back. Not even close.

The second graph may help to explain the first one a little. It shows the responses to the question, “Are you good or bad at English?” Only 8 % say they are really good  at English. 29.5% say they are sort of good at it. 32.5% say they are sort of bad at it. And a larger-than-should-be 29.3 % state theyare really bad at it. We should be getting something more bell-curvsey here, I think; instead we get a pile of academic corpses in our lower regions. Somebody is doing something wrong almost everyone agrees.

Perhaps Mr Martin et al can help. The number one factor they determined in getting kids interested in maths was facilitating self-efficacy, that magical, elusive feeling that one is competent and able to solve problems. They suggest fostering this by restructuring learning to offer opportunities for success. Success is motivating. I couldn’t agree more, but jr high schools in Japan are so much in the business of separating learners out along the continuum that they don’t offer enough of these opportunities. It seems rather that schools give large numbers of low scores deliberately, like  switches to the backs of  heads of meditating monks  losing concentration. It works sometimes, but with many learners it does not. Many learners can only take getting hit with low scores so often before they give up. It doesn’t have to be this way.  Language, unlike math, can be successful without being completely correct. But in most jr high schools, communicative success is not rewarded. Close on English test question answers,  unlike hand grenades and horseshoes, does not count. When the spelling is wrong, the answer is wrong. Wrong, period. No points. And when the answer is different from the one in the text book (Wrong: this is the train to Machida; Correct: this is the  train for Machida, for example) it’s WRONG. NO POINTS.

The Australian math motivation researchers also found positive effects for family and school cultures that intrinsically valued math. That is, the learners got a clear message that math was important and math learning was important. When we try to apply this finding to English in Japan, we find some differences. Kids and (probably especially) their parents are embedded in a culture where doing well at English (not necessarily learning English as a language) is seen as a way to get into a better high school. Doing well at school has become the rationale for learning the language. They (the parents) expect their kids to actually pick up English later in their academic careers, but in jr high, they want them to learn vocabulary, reading, and test-taking skills. I can understand this up to a point–getting into a high school is a super important event in someone’s life and academic career. But wouldn’t you want your kid to learn to swim if they took swimming lessons, instead of memorizing the size of the pool and the names of the muscles involved in the activity? Wouldn’t you want your kid to get in the water sometimes? As the parent of a recent jr high graduate in Japan, I have been witness to this sorry condition, though I am sort of lucky in that I never expected the school to teach my daughter English; she’s mostly bilingual, and as a native English speaker and an EFL teacher, I have resources. But I feel sorry for the other students in her classes. They get an endless stream of seemingly discreet particles of the language that they just have to process and remember accurately. They never make meaning. Mistakes are never forgiven. They never feel progress, except when they do well on that high school entrance exam. And then it’s on to the next level. They never get the chance to realize that learning a language offers opportunities for expanding your world, meeting people, and exploring yourself. They just come away with the message that English is hard and they are not good at it–though all that has really been proven at that point is that they are not good at memorizing which letters go in which order and which words go in which order in short meaningless sentences. Oh, and they learn what a painful and pedestrian slog workbooks and classes are, and how juku (cram school) can drill that crap into you if you give up your free time and your parents can part with a lot of cash.

In many ways, Mr Martin’s finding could be more easily put to work with English than with math. Letting learners experience success is much easier. It is easier for partial, or flawed, or broken communication to still be successful communication. But it will take a mindset shift, definitely. In Julie Dirksen’s book on instructional design called Design for How People Learn (click here for my review) she has one chapter on designing for knowledge and one on designing for skills. The approach to each is fundamentally different. Designing for skills acknowledges that it is much more of a process, and it requires more practice and more formative feedback. Jr high school teachers do not seem to be approaching English as a skill, but rather as knowledge. That is both the crux of the problem and the opportunity for change.

 

 

The Creativity Gap

Rick Martin, writing in the Tech In Asia blog, recently posted the result of a study done by Adobe about attitudes toward creativity. The results are summarized in the article and in the graphic below.

And the results are eyebrow-raising. Japan is seen by everyone as an incredibly creative place–except by the Japanese themselves! And an astonishingly large number of people in Japan (78%) said creativity is something reserved for people in the arts community. I don’t mean to sound alarmist (OK, actually I do), but this should set off the same  bells and buzzers and active responses that PISA results do. But as Mr Martin and some of the response posters to his blog article state, Japan is a fantastically creative place. In fact, perhaps it may be because Japan has such a high level of creativity in almost everything, that people have a different view of creativity in general. This needs to be considered further…

Two Articles on Innovation: The Blue School and Sony

Innovation is the creation of better or more effective products, processes, services, technologies, or ideas that are accepted by markets, governments, and society.

The above quote comes from Wikipedia. It came up as I checked the spelling of “innovation”. What sent me looking for the spelling of innovation was an encounter with two articles from the New York Times, one from April 13 called At the Blue School (from which I also borrowed the image above), and the other from a day later called How the Tech Parade Passed Sony By. Both of these articles are very interesting and worth a few minutes of your time if you are interested in education and Japan. And both are focused on topics getting a lot of coverage recently.

Articles on neuroscience appear almost daily in the news, and several groups/sites/schools/programs have come into existence in the last few years. There’s the Neuro Education Initiative at John Hopkins University (mentioned in the article), The NeuroLeadership Institute (associated with author and Blue School board member David Rock, and also mentioned in the article), USC’s Brain and Creativity Institute,  Harvard’s Mind / Brain / Behavior initiative, and an Annenberg Learner Resource called Neuroscience and the Classroom, to name some that I’ve come across in the last half  year or so.

Sony has also been in the news, partly for the transfer of power to a new CEO (from Howard Stringer to Kazuo Hirai), and partly for making a record loss ($6.4 holy smoke billion!). The Times article focuses on Sony’s problems, the subject of so many books (here and here in Japanese), TV spots, and articles that there are probably families who discuss it at the dinner table regularly. Well, maybe not dinner table, but certainly it is present at the heart of the debate about what is “wrong” with Japan economically and what can/should be done about Japan, Inc., the economic model many people grew up with.

Seeing these two articles on the same day got me thinking, wondering if there is any connection we can make between a school that tries a new initiative, garnering both academic praise (my neuroscience Twitter gallery went quadruple post on that link) and serious parental acceptance (it costs more than $30,000 a year to send your young elementary-age child to the Blue School), and a company that seems to have forgotten how to innovate. It is tempting to make the leap that schools in Japan, like their business compatriots, are resistant to change and are struggling to find their way in the face of a changing global environment (blah they can’t lose money but they can certainly waste it on ineffective English language lessons blah blah). It is tempting, but probably a gross (as in icky) generalization, and mostly incorrect. It is tempting because in my job as a teacher trainer for a local government in Japan, lack of innovation is something I see quite a lot of. It is tempting because the Sony article makes the following claim that seems to hook the two articles together:

Sony’s woes mirror a wider decline in Japanese electronics. Though executives here are quick to blame a strong yen, which hurts exports, a deeper issue is that once-innovative companies seem to have run out of ideas. And when a nation can no longer compete on abundant labor or cheap capital, ideas and innovation are paramount.

It is probably incorrect to make these connections too quickly because a single boutique school does not represent a nation, and while applying neuroscience findings to classroom settings is something I am obsessing about myself recently, I’m not sure a) people have completely figured out how to do so effectively yet, and b) good teachers probably do a lot of what neurofanatics say teachers should be doing anyway. Compare the Blue School classroom depicted in the article with this elementary school classroom in Kanazawa, for example. It may very well be true that the focus on creativity and the process of learning practiced at the Blue School may be exactly what more Japanese need educationally to get out of the past and into the global future. Certainly I would like to see more of that, more application of skills and less rote learning, in language classes in Japan. And I think we can say that adding more fun, personalization, emphasis on affect, and the collaborative, social side of learning, would make lessons more bearable for a lot of learners. But how well these things can be instituted, and how effective they can be when they are instituted, is still not certain.

When I read the definition from Wikipedia, that last part really stood out: “…that are accepted by markets, governments, and society.” I copied it and pasted it here because it raised a few questions, both inward and outward. Like most people (I think anyway…) I had always sort of assumed that innovation was all about creativity and newness. But acceptance is a crucial part of innovation, not necessarily at first, but at some point, or else it is not innovation. Innovation is the process of  social acceptance of creative initiatives. That makes me feel better about my job (where I do face  rejections of my initiatives). Change is a process, not an event (a quote I found attributed to Barbara Johnson, but repeated often). Yup. What the Blue School and Sony have in common is they have to go through the same process. Size, culture, structure, personalities, and the power of the idea behind the initiative all impact on this process.