What can Data do for EFL?

image of glasses

In the US, something very interesting is happening as an indirect result of standards-based testing and the establishment of charter schools: a certain interesting set of research conditions has emerged. Charter schools are themselves often experiments, established in part to be “…R&D engines for traditional public schools” (Dobbie & Fryer, 2012). As such, they often eschew many elements of traditional education practices, and instead attempt to institute the results of research into educational effectiveness from the last several decades. For many charter schools, this research-informed pedagogy assumes a unifying or rallying role, and the ideas are woven into their mission statements and school cultures, as they try to take underprivileged kids and put them on the road to college, through grit training, artistic expression, or higher expectations. Even from a brief visit to the websites of charter systems such as Uncommon Schools, MATCH, or KIPP, you can see what they are intending to do and why. And some of these charter schools have become extremely successful—in terms of academic achievement by students, and in terms of popularity. And both of these have led to new research opportunities. You see, the best charter schools now have to resort to lotteries to choose students, lotteries that create nice groups of randomly-sampled individuals: those who got in and received an experimental treatment in education, and those who didn’t and ended up going to more traditional schools. And that provides researchers with a way to compare programs by looking at what happens to the students in these groups. And some of the results may surprise you.

Let’s play one of our favorite games again: Guess the Effect Size!! It’s simple. Just look at the list of interventions below and decide whether each intervention has a large (significant, important, you-should-be-doing-this) impact, or a small (minimal, puny, low-priority) impact. Ready? Let’s go!

  1. Make small classes
  2. Spend more money per student
  3. Make sure all teachers are certified
  4. Deploy as many teachers with advanced degrees as possible
  5. Have teachers give frequent feedback
  6. Make use of data to guide instruction
  7. Create a system of high-dosage tutoring
  8. Increase the amount of time for instruction
  9. Have high expectations for students

Ready to hear the answers? Well, according to Dobbie & Fryer (2012), the first four on the list are not correlated with school effectiveness, while the next five account for a whopping 45% of the reasons schools are effective. Looking at the list, this is not surprising, especially if you are aware  of the power of formative feedback.

Some people might be a little skeptical still. Fine. Go and look for studies that prove Dobbie and Fryer wrong. You might find some. Then look at where the research was done. Is the setting like yours? Just going through this process means we are putting data to work. And that is much, much better than just going with our own instincts, which are of course based on our own experiences. I work teaching English in Japan, and I know that is a far cry from the hard knocks neighborhoods where Dobbie and Fryer looked into the effects of interventions in Match schools. But I think there are enough similarities to warrant giving credence to these results and even giving them a try at schools Tokyo. I have several reasons. First, extensive research on formative assessment, high expectations, classroom time, and pinpointed direct instruction is very robust. Nothing in their list is surprising. Second, in Japan, English is often as foreign from the daily lives of most students as physics or math are from the lives of many American teens. The motivation for learning it is likewise unlikely to be very strong at the beginning. Many of the students in the Match system are less than confident with their ability with many subjects, and are less than confident with aiming at college, a world that is often quite foreign to their lives. Many English learners in Japan similarly see English as foreign and unrelated to their lives, and the notion that they can become proficient at it and make it a part of their future social and/or professional lives, requires a great leap in faith.

But through the Match program, students do gain in confidence, and they do gain in ability, and they do get prepared for college. Given the demographic, the success of Match and the other “No Excuses” systems mentioned above is stunning. It also seems to be long lasting. Davis & Heller (2015) found that students who attended “No Excuses” schools were 10.0 percentage points more likely to attend college and 9.5 percentage points more likely to enroll for at least four semesters. Clearly the kids are getting more than fleeting bumps in scores on tests. And clearly the approach of these schools—in putting to work proven interventions—is having a positive effect, although not everyone seems to be happy.

And it’s not just that they are making use of research results. These schools are putting data to use in a variety of ways. Paul Bambrick-Sotoyo of Uncommon Schools has published a book that outlines their approach very nicely. In it we can find this:

Data-driven instruction is the philosophy that schools should constantly focus on one simple question: are our students learning? Using data-based methods, these schools break from the traditional emphasis on what teachers ostensibly taught in favor of a clear-eyed, fact-based focus on what students actually learned (pg. xxv).

Driven By Data book cover

They do this by adhering to four basic principles. Schools must create serious interim assessments that provide meaningful data. This data then must be carefully analyzed so the data produces actionable finding. And these findings must be tied to classroom practices that build on strengths and eliminate shortcomings. And finally, all of this must occur in an environment where the culture of data-driven instruction is valued and practiced and can thrive. Mr. Bambrick-Sotoyo goes through a list of mistakes that most schools make, challenges that are important to meet if data is to be used to “…make student learning the ultimate test of teaching.” The list feels more like a checklist of standard operating procedures at almost every program I have ever worked in in EFL. Inferior, infrequent or secretive assessments? Check, check, check. Curriculum-assessment disconnect? Almost always. Separation of teaching and analysis? Usually, no analysis whatsoever. Ineffective follow-up? Har har har. I don’t believe I have ever experienced or even heard of any kind of follow-up at the program level. Well, you get the point. What is happening in EFL programs in Japan now is very far removed from a system where data is put to work to maximize learning.

But let’s not stop at the program level. Doug Lemov has been building up a fine collection of techniques that teachers can use to improve learning outcomes. He is now up to 62 techniques that “put students on the path to college,” after starting with 49 in the earlier edition. And how does he decide on these techniques? Through a combination of videoing teachers and tracking the performance of their classes. Simple, yet revolutionary. The group I belonged to until this past April was trying to do something similar with EFL at public high schools in Japan, but the lack of standardized test taking makes it difficult to compare outcomes. But there is no doubt in my mind that this is exactly the right direction in which we should be going. Find out what works, tease out exactly what it is that is leading to improvement (identify the micro-skills), and then train people through micro-teaching to do these things and do them well and do them better still. Teaching is art, Mr. Lemov says in the introduction, but “…great art relies on the mastery and application of foundational skills” (pg.1). Mr. Lemov has done a great service to us by videoing and analyzing thousands of hours of classes, and then triangulating that with test results. And then further trialing and tweaking those results. If you don’t have copy of the book, I encourage you to do so. It’s just a shame that it isn’t all about language teaching and learning.

Interest in using data in EFL/ESL is also growing. A recent issue of Language Testing focused on diagnostic assessment. This is something that has grown out of the same standards-based testing that allowed the charter schools in the US to thrive. You can download a complimentary article (“Diagnostic assessment of reading and listening in a second or foreign language: Elaborating on diagnostic principles”, by Luke Harding, J. Charles Alderson, and Tineke Brunfaut). You can also listen to a podcast interview with Glen Fulcher and Eunice Eunhee Jang, one of the contributors to the special issue. It seems likely that this is an area of EFL that will continue to grow in the future.

Making EFL Matter Pt. 3: The Challenges and Benefits of Discussion

image of balls in a tray

 

Well, what do you think? This question and answer form a basic opinion exchange that is sometimes called a discussion. And it is, sort of. But just as a single decontextualized sentence is of limited use in understanding grammar, so too a brief opinion exchange does not have enough context–with all its intentions, personalities, and sociolinguistic depth–to really be called a discussion. A discussion is more complex, and ultimately more powerful, because it has a goal and requires input and interaction from multiple members, which should allow them to collectively generate better ideas (solutions, plans, etc.) than any one of the participants could have done alone.

This sounds good in theory, but it is difficult to achieve in classrooms–especially EFL classrooms where learners have a layer of linguistic difficulty on top of the conceptual and procedural challenges inherent in establishing a system of rich academic discussions. The first thing we must acknowledge is that academic discussion skills, like Rome, are not built in a day. As I mentioned earlier, they need to be incrementally developed, starting with basic conversation and interaction skills.  Without basic conversation skills, discussion is not attainable. Students who are used to pairwork and are able to use the basic greetings, openings, and closings of common conversation “scripts” (Hi. How’re you doing? So, what did you do on the weekend? Well, nice talking to you!), and can react to each other’s utterances (Uh-huh, Really?! Oh, I love that!, Really? How was it? etc.) will find discussions accessible. Absolute speaking beginners will struggle and likely fail. Speaking must be taught, skills must be developed, and regular opportunities for fluency development given, or else activities like academic discussions, and the opportunities to flex critical thinking muscles that go with them, won’t be achieved. A little bit of speaking tagged on to the end of a lesson won’t get you there (as programs in high schools in Japan are slowly waking up to).

So now we know it’s difficult and requires a program of incremental skill development starting with a foundation in basic interactive conversation skills. One question we might ask is: is it worth the trouble. Given a limited amount of time, why should so much be devoted to conversation and discussion skills development? Well, the answer comes from sociocultural learning theory. As Daniel Siegel puts it in his forward to the wonderful Social Neuroscience of Education: “We evolved in tribes, we grow in families, and we learn in groups.” Walqui and van Lier (2010), in listing up the tenets of sociocultural learning theory for their QTEL approach, focus on some of the key points: “Participation in activity is central to the development of knowledge; participation in activity progresses from apprenticeship to appropriation, or from the social to the individual plane; and learning can be observed as changes in participation over time” (pg. 6). That is to say, we learn through active participation (engagement and collaboration) with others. “Language is primarily social”…and “…learning…is essentially social in nature” (pg. 4-5). This learning does not happen by chance, however. The really really hard thing to do is to get students into that sweet spot where they are developmentally ready and linguistically scaffolded  up to the point where they can function and learn. Development becomes possible when “…teachers plan lessons beyond the students’ ability to carry them out independently” (pg. 7), but create the proper community and provide the proper scaffolding to allow for success with such lessons. To answer the question that started this paragraph, the potential benefits of learning in groups are great enough to warrant using this approach. Students can learn content and language, and collaboration skills, essential skills for the 21st century according to Laura Greenstein (who also helpfully provides a rubric and suggestions for assessing collaboration, as well as other skills).

One more potential objection to focusing on academic discussion comes from Doug Lemov. Actually, it’s not so much of an objection as request to rethink and balance your choices. In Teach Like a Champion 2.0, he suggests that both writing and discussion can be strong tools for “causing all students to do lots of the most rigorous work,..but if I had to choose just one, which admittedly I do not, I would choose writing. Hammering an ideas into precise words and syntax and then linking it to evidence and situating it within a broader argument are, for me the most rigorous work in schooling” (pg. 314).  Writing is great, and cognitively more “precise” perhaps, and definitely needs to be part of the syllabus. Discussion is by nature more social. You can do both and you should do both; finding the time to do so is challenging, however.

So what do we teach? And what are discussion skills that students need to learn and develop? The best list can be found in Academic Conversations by Jeff Zwiers and Marie Crawford. It needs to be said, however, that this book is made for L1 learners and you will need to adapt as you adopt. But the basic list and framework make it easy and intuitive to do so. Here is the list of skills:

  1. Elaborate and Clarify: Make your thinking as detailed and clear as you can, carefully explaining the rationale behind your thinking.
  2. Support Ideas with Examples: Use examples to illustrate thinking. It is a particular and powerful way of elaborating and clarifying ideas.
  3. Build On or Challenge Partners’ Ideas: Actively respond to and develop the ideas that arise, either by expanding on them or tweaking them, or pruning out the bad ones through well-considered disagreement.
  4. Paraphrase Ideas: As ideas arise, paraphrase them to both show your understanding and create a springboard for idea development and improvement.
  5. Synthesize the Discussion Points: Bring all the ideas you’ve been discussing to a conclusion. Produce a group decision or plan.

These skills need to be introduced incrementally. Some of them are more difficult to teach and practice than others, particularly with students who lack proficiency or fluency, but also for cultural reasons. Some students in Japan find it challenging to disagree, and may have trouble ranking or pruning some ideas from their synthesis. These can be trained and taught. In my experience, students are not used to making their thinking so explicit and considering  ideas so carefully. Once they get the hang of it, however, they clearly become better listeners, better collaborators, and better thinkers. And after gaining some fluency with the formulaic expressions required to do academic discussions, they sound considerably more proficient.

Assessment is not as difficult as you may imagine. Ms. Greenstein’s rubric provides a nice overview of expected performance and can be used together with teacher observations, and peer or individual reflection and feedback. Performance tests are easy to organize because we can check several students at once in their group. Focusing on the success of the group in terms of process and outcome is actually not that hard to measure.

This post is part of a series considering ways to add more focus and learning to EFL classrooms by drawing on ideas and best practices from L1 classrooms.

Part 1 looked at the importance of goals.

Part 2 looked at using data and feedback.