Blended Learning with Khan Put to the Test #Edtech #KhanAcademy #k12 @Edsurge
Image Source: pharmrep.findpharma.com
In this post, that will likely be a long one, I examine the latter exploration because the team that is carrying out this study is calling it an experiment. Experimentation in any social science is difficult, and it is particularly difficult in education. There are few well controlled experiments in education, and those that are well controlled are usually limited in scope. I recently dissected a pilot study of a blended learning environment with School of One technology in the New York City public schools. Below, my scalpel is even sharper. I do this not because I am a curmudgeonly union stiff. Quite to the contrary, I really want this to work. I am critical because I want it to really work.
I am excited about the possibilities offered by well-used, appropriately designed EdTech. However, far too often in education, the real experts, classroom teachers, become the subject of someone else’s untested idea. There are many reasons for this, but I will leave a thorough discussion of such issues to those in education politics. Let’s talk tech!
I am entering this conversation two weeks into the Envision academy “experiment,” and all of my information is coming from the blog where they are documenting their progress.
The Blend My Learning Team that is doing this important work in Oakland recognizes that what they are doing would not stand up to academic standards because of their small sample size; two classes of 25 students each. Nonetheless, the study appears to be controlled in three significant, albeit insufficient, ways. Kudos to the team for randomly assigning students to the experimental and control groups. More praise to the team for recognizing that the study would have to be done with the same teacher in both the control and experimental groups to have any meaning. And lastly, the team selected for a pre and post test a standard algebra concepts understanding metric, the MDRT, to measure the effectiveness of their treatment.
Unfortunately, this is where things fall apart. Based on the following paragraph lifted from the BlendMyLearning blog it is apparent that there is a team of researchers/educators in the experimental, blended Khan class.
-
“One of the questions we are considering with Khan is whether we truly let students work on whatever content they need. Khan suggests that all students start at the beginning of their “star chart” with single digit addition and then build up steam as they get a lot of problems right, earn “energy points” and badges, and get comfortable on the platform. That said, what about the students who have big numeracy gaps in fractions, decimals, percentages, etc.? If they only have five weeks of summer school to remeidate algebra, we are pondering whether it is “okay” for them to focus on their numeracy gaps but not get as much of the algebra content. Our plan is to give them a week or so to focus on where they need it most and then monitor the data. But it may be that these students get more of a foundational shoring up rather than true algebra intervention.”
More than one adult in a classroom gives a class an advantage. Even if that adult is an impartial observer, and it does seem that the BlendMyLearning team is involved in a not insignificant way in the classroom, students behave differently and are likely to be more task oriented with more supervision. Is the team equally present in the traditionally taught class?
The research team decided to control the experiment by excluding all technology from the traditional class. This is unrealistic. Most teachers employ technology in some fashion. Thousands of math classrooms across the country use interactive white boards. Many teachers do some kind of online lab work with students, and there are a growing number of classes employing remote response pad technology. Preventing the teacher from using any technology in the traditional class, the control, is akin to attempting to control an experiment on the effectiveness of using GPS in navigation by preventing the control from using a map.
Experimental effect. The experimental group is being treated as special in several additional ways. First, every student has a brand new Google Chromebook. Did they have to be new? Students are asked to give feedback on the learning process on a regular basis. Are students in the traditional class also asked how the class might be improved to better meet their needs? The experimental group students have been interviewed on video and asked to reflect on their learning process. Is this valuable reflection also happening in the traditional class? Students in the experimental group were told that they are part of a “revolutionary” new technology program. No doubt, buy-in to the process was necessary. I know. I am a high school teacher. But the experimenters have already biased the students toward success by telling them that they are a part of something that will change them.
In short, while the Blend My Learning study at Envision Academy may provide the Khan Academy team (who, by the way, further tainted the experiment by visiting the school during the process) with valuable insight into what works for kids in a blended environment, any results the team attempts to publish should be consumed with scrutiny. For example, nothing but a significant gain on the MDRT metric should be considered, well, significant. Any qualitative analysis about student buy-in will have to be disregarded entirely because of the adult attention the experimental group has received – “Are we going to be famous?”
Nonetheless, I was impressed with the reports of the on-task behavior, and the comment about zero classroom management issues in the experimental class. Blended learning, so far, has shown great promise in high school science classes when used as an inspiration to inquiry, and when the video is not (paradoxically) a straightforward explanation of a concept. Blended learning has also been used successfully for years as a credit recovery device for students, like these Envision Academy students, who are repeating the class having failed it at least once already. I am skeptical but hopeful about the possibilities for differentiation that blended learning offers in high school. If Envision Academy students show significant progress on the MDRT there may be reason to get curious, but please contain your excitement until the teacher reviews start coming in.
I’m a high school science teacher and while the Blend My Learning comments about zero classroom management can be impressive, they should be taken with a grain of salt. This is probably a byproduct of using computers & headphones. I’ve seen and occasionally filled in as a sub for computer classes at my school that have a similar setup (computers + headphones). You put a screen in front of a kid and headphones on them and they are in their own world for better or worse.
I feel uneasy with this version of “no management issues.” All to often I find both myself and other teachers defining management issues solely as students speaking out of turn or misbehaving in a highly visible way that disturbs the class environment. When pushed to think about classroom management further, I’m reminded of classes with the opposite problem. Classes where I wish the kids would throw something, speak out of turn, or anything to let me know they are still alive (see http://youtu.be/dxPVyieptwA for a classic example from Ferris Bueller’s Day Off). As we begin to use technology in the class, especially on the individual student level, I think we need to expand our definition of classroom management to include student engagement or time on-task as well as the more traditional form acting out. After all, a business manager would be seen as a failing at management if all of his/her employees did absolutely nothing all day.
Not every student rebels by acting out, some disengage. Multimedia technologies encourage minimal physical inactivity, with the exception of Wii, Kinect, and similar technologies. This means that there is little difference in the physical behavior of on and off task students. This further supports the idea that we must redefine what it means to have classroom management issues to reflect changes in the classroom environment due to new technologies.
Thanks for your insight, Ben. I agree that quiet does not necessarily mean well-managed. Sometimes the cacophony is the symphony.
I taught math to seven gifted second-grade children one hour a week last year. I’m not a professional educator. I have an advanced mathematics degree.
The first half of the year, I ‘lectured’, we worked in groups, activities, reading and problem solving.
The second half, we used Khan Academy.
My own very very small scale experience mirrors some of the results above, including the classroom management elements. In my class, students didn’t use headphones unless they specifically needed to watch a video, the rest of the time, headphones were off.
I spent significantly more time working individually with each student using the Khan website; almost never was every student in the same place in the learning process, so this was a HUGE gain for students and me, we could talk one-on-one about needs as the class proceeded, rather than everyone waiting while one person’s needs were filled.
Anecdotally, I think I could have done 10 to 12 kids with Khan, 4 was comfortable without it.
All that said, I’m not sure why there seems to be resistance to using things like Khan Academy in the class; my students’ motivation was much higher, learning was faster, and I was happier as a teacher. The entire methodology is so different from so many of the graded educational supplements I have seen and used, and it works well, at least it did in my experience.
Peter V, I think many of us would agree with you but I suspect the blog owner would question the validity of your methodology.
When I read something like “would not stand up to academic standards”, the first thing that comes to mind is how low our academic standards are so it’s hard to really care.
Exactly. The reality is that we are reviewing semi-personalized instruction combined with reward systems that are known to be addictive (a-la Warcraft), and comparing them to.. self-paced worksheets? Delivered in classes with 30:1 student:teacher ratio?
I have a hard time understanding how any reasonably good teacher would turn down free assistance like this. In my case, the teacher and administration helped sideline a number of district-wide blocks on Youtube, just so that the students could participate — they loved it, and were HIGHLY motivated to get it going.
Jack, I appreciate your tight focus on research methodology in education. The abysmally low standards in educational research are in part responsible for the lack of any real objectivity in the adoption of new teaching methods and technologies. All too often decisions in education are the results of incompetent and corrupt officials succumbing to the lobbying of immoral education corporations who need only to provide the semblance of objective research to legitimize their over-priced and deeply-flawed products.
Research methods aside, I am a former middle-school math teacher from an East Oakland school that belongs to one of the most succesful CMOs in California. My students achieved state test scores in the top ten percentile of California public school students despite being mainly children of poor non-English speaking immigrants. I attribute three factors to my school’s influence on their success: the extraordinary talented and cooperative school community, my own skill as a teacher, and my implementation of educational technologies. As an early adopter and evangelist of many educational technologies, Khan Academy being one of them, I have personally found that well-developed technologies; i. g. ALEKS, Khan Academy, Manga High; when smartly adopted by a highly competent educator, produce extraordinary objective results . Unfortunately, I have seen these results rarely replicated due to lack of one of the aforementioned components. I see poor technologies chosen, i.g. anything Pearson produces. I see technologies horribly implemented, the repugnant way Florida is saving money by abusively throwing large numbers of students into giant computer labs with little direction. I see new or developing teachers ineptly using technology as they struggle to also master the craft of teaching. All of these failures are then compounded by the lack of objective educational research standards. All it takes is a district bureaucrat or company official to fudge the data and a failed implementation is suddenly successful and marketable.
The problem we now face is not what educational technologies we should implement and how. All our work will be in vain until we establish objective standards for judging these technologies. However, along with establishing these standards, we must liberate education in America from the bureaucratic, corporate, and political forces that maintain the dysfunctional and antiquated state of the public school system for selfish and nefarious ends. No real educational change can happen without citizens taking back control of public education. All supposed reforms: NCLB, the charter school movement, pay-for-performance, mandated testing are worthless distractions meant to neutralize any force for the real change we need. We need an educational revolution.
I am honored by your thoughtful and probing comments eduruptor, Peter V, PWB. There is more intelligent discussion of this issue at this thread. It is great to see so many interested in how we can really judge the effectiveness of educational interventions.
http://news.ycombinator.com/item?id=2913243
Thanks to Mike Lee for the reference.