Edtech is worth fighting for (2 of 2)

The case against blanket "Edtech" bans and a discussion of the evidence that's being manipulated and ignored in "The Digital Delusion."

This post is a follow-up to a previous one on Ms. Emily Cherkin and Dr. Jared Cooney Horvath’s testimony to the United States Senate. Here, we’ll look at Dr. Horvath’s claims as presented in his self-published manifesto: The Digital Delusion.

Like Ms. Cherkin, Dr. Horvath makes some valid points that we in the Edtech industry should heed. Unfortunately, they’re buried beneath cherry-picked statistics, correlational evidence, logical fallacies, straw men, and moral panic.

In his book, Dr. Horvath addresses “5 Lies” told by “Edtech.” In the spirit of that rhetoric, I’ll address 5 lies told by Dr. Horvath.

Effect sizes of different technologically mediated interventions, as measured by Dr. John Hattie in Visible Learning 2.0 - more proof that“Edtech” is not a single intervention, but a medium that augments many interventions.

1. Student-facing-device-as-instructional- intervention

Within the first 5 pages, the central flaw of The Digital Delusion becomes clear: Dr. Horvath views ‘EdTech’ as a singular instructional intervention.

This is wrong. A laptop is an instructional medium, not an instructional intervention.

Dr. Horvath should know this, given some of the research he cites.

In his landmark Visible Learning: The Sequel, Dr. John Hattie dedicates a chapter to instructional technology. After recounting generally positive effect sizes for the different ways in which it can be used to actualize pedagogy (“The message throughout this chapter is not that technology has not made a difference. It has, and that is what an effect of 0.3 to 0.5 signifies” (p.408, Visible Learning), Dr. Hattie says the following. Bolding is mine:

“If there is to be a third edition of Visual Learning, there would not be a chapter on technology influences, as it is time to think of technology like curricula, books, and other resources, and focus more on the implementation effects when including technologies and teaching methods, the when and how within the teaching cycle, the dosage and the fidelity of various apps, and so on….”

The UN Report previously cited by Ms. Cherkin makes a simular argument. To quote that report: “This confusion between the tool and the solution, between the means and the end, is what this report invites us to address…Technology should not be viewed as the solution, but as a supportive tool.”

“Edtech” isn’t a single treatment, it’s a blanket term for thousands of different digital curricula and applications. You can’t aggregate the impact of all of them simply because they all have a screen and connect to the internet. To do so is bad science, as is argued by Dr. John Hattie, the United Nations, and many other researchers.

To me, this undercuts the entire thesis of the book. It also leads directly into some of the logical fallacies that follow.

2. Straw Men

Dr. Horvath argues that “Edtech” perpetuates 5 lies:

1. Education is broken

2. Multimedia enhances learning

3. Free choice leads to better learning

4. Kids learn best on their own

5. Intelligent tutors make kids more intelligent

Ummmm. Okay. First, who is “Edtech”? As someone building an instructional technology tool, I disagree with points 1, 3, and 4. I think points 2 and 5 can be true, depending on the tool, the problem it’s being used to solve, and the context of the problem. Given that I consider myself a part of the Edtech community, I was curious to learn which individuals or organizations are perpetuating these lies.

Dr. Horvath doesn’t provide sources.

For example, while explaining the first “lie” that Edtech perpetuates, that education is broken, he says: “This narrative has been repeated so many times in op-eds, on TED stages, and in keynote speeches that it almost feels self-evident.” He then fails to cite a single op-ed, TED talk, or keynote speech.

It seems he’s referencing Sir Ken Robinson’s famous Do Schools Kill Creativity? If so, what does that have to do with technology?

Somewhere in the middle of slaying each of these 5 straw men, I started to wonder: Who published this? How could they let him say all this without citing specific companies, people, or organizations perpetuating these lies? Then I looked at the copyright page and realized it was self-published.

Charts from an OECD report cited by Dr. Horvath. Not included in his analysis: 15-year-old students who use digital devices moderately for learning at school tend to perform better and report a greater sense of belonging. Sense of belonging is one of the most important factors in improving learning outcomes for students.

3 & 4. Correlative, cherry-picked evidence

Chapter 3 of Dr. Horvath’s book is its most important chapter, as it presents the corpus of evidence he feels justifies “pulling the plug” on Edtech.

Dr. Horvath cites declining PISA scores (the world’s largest standardized test) from 2012, 2015, and 2018, which correlate with the widespread adoption of educational technology in developed countries: “The more time students spent on screens at school, the further their scores fell. Those who used a computer for more than 6 hours per day scored 66 points lower than their peers who didn’t use them at all.”

This argument is a logical fallacy: Post-hoc-ergo-propter-hoc. After, therefore, because of. Correlation is not causation. Just becuase EdTech use increased, and scores dropped, does not prove that EdTech caused the drop. People who carry umbrellas are more likely to wear rain boots, etc., etc.

None of the research on test scores presented is causal. Let’s look at it anyway, as we’ll find that even the correlational evidence was cherry-picked to fit a narrative.

Here is the OECD analysis correlating screen time and PISA scores (see charts above and below). The study showed a correlation in which students who spent 5–7 hours per day on “digital devices” for leisure at school scored 10–12 points lower than students who spent 3–5 hours per day.

First, Dr. Horvath has left the “for leisure” qualifier. Second, he’s left out that this study lumps all digital devices under one label. No distinction is made between personal cellphones vs. school-managed laptops. These are both critical distinctions if we’re going to use this study to justify school laptop policies.

So, I guess in a sense, he’s right that spending the majority of your school day using a device for leisure correlates with negative learning outcomes. That said, I don’t know of a single technology integration specialist or Edtech developer arguing that kids should be spending 7 hours per day using a digital device for leisure.

Regardless, other important data have also been left out (see chart above):

• Students who spend one to five hours per day on digital devices for learning at school achieve twenty PISA score points higher in mathematics than those who spend no time on such devices.

• 15-year-old students who use digital devices moderately for learning at school tend to perform better and report a greater sense of belonging at school.

• (again, because it bears repeating) Most of the negative correlations were related to student device use for leisure, not student device use for learning.

Why were these points left out? Dr. Horvath even acknowledges them later in the book (see p.78-79), but argues “learning isn’t unique to computers,” so these researchers are wrong. If the researchers are wrong, why are you citing them?

This becomes common when you engage with anti-edtech folks. The conversation goes:

Them: Show me the evidence that technology can help learning.

Me: Okay, here’s some evidence.

Them: …that evidence doesn’t count.

Another critical point to note: Many great technology tools aren’t primarily designed to improve standardized test scores! If a tool’s theory of change isn’t focused on raising PISA scores, then declines in those scores don’t falsify their value.

I’m building a digital writing tool. Writing ability is not directly measured on PISA. How do declining PISA scores falsify the value of the writing tool I’m building?

These illogical conclusions happen when you ignore the advice of researchers and erroneously lump all digitally mediated interventions under one misleading “Edtech” banner.

Dr. Horvath continues with the correlational evidence between declining test scores and technology usage using Trends in International Mathematics and Science Study (TIMSS) and Progress in International Reading Literacy Study (PIRLS) data. He simply references scores on the respective websites of these tests. If he were to seek out studies of these scores, here’s what he’d find:

Researchers warn against his exact approach. Simply correlating aggregate tech use and test scores is not useful. Researchers rightfully argue, similar to Hattie and the U.N. previously, that we should instead study effective tech use & scores vs. ineffective tech use. They argue for this precisely because there are so many experimental and small-scale studies showing positive effects of technological applications on student performance in various subject areas such as literacy, mathematics, and science (Chauhan, 2017; Cheung & Slavin, 2012, 2013; Sung et al., 2016). I would also add this recent meta-analysis by Dr. Rebecca Silverman at Stanford, which, among other things, showed a positive effect size for digitally mediated early childhood literacy tools. Because of this, Dr. Silverman concluded that “investment in educational technology to support literacy is warranted, and the prevalence of educational technology in schools is justified.”

Not only does Dr. Horvath not include this mountain of research, he does the opposite of what the researchers in these studies advocate for. Why?

Dr. Horvath does acknowledges Hattie’s meta-analysis showing positive effect sizes for Edtech. He then does mental gymnastics to deny them, essentially arguing “yes, they are positive, but other interventions have higher effect sizes, so we should use those instead.” To quote him: “So yes, students can learn with EdTech: the effect sizes aren’t negative. But this learning is almost always slower, shallower, and less durable than what’s achieved through human-centered methods like explicit instruction (+.64), guided discussions (+.82), and jigsaw teaching (+1.16).”

The misunderstanding of technology as a intervention rather than a medium has again led into a logical fallacy: False dichotomy: You either use “Edtech” OR you use explicit instruction, guided discussion, etc. This fails to acknowledge that…YOU CAN USE EDTECH TO AUGMENT EXPLICIT INSTRUCTION, GUIDED DISCUSSIONS, AND JIGSAW TEACHING!

I’ll share two examples that I have personal experience with.

The Modern Classrooms Project uses technology exceptionally well to improve explicit instruction. Because lecture and modeling are recorded, students can pause, rewind, speed up, and consume content asynchronously (that last one is critical in high-poverty districts with high absence rates). This allows students to work through guided and independent practice at different paces in ways that would otherwise be impossible with only live, teacher-led instruction. Research done by Johns Hopkins University found this approach significantly improved teachers’ ability to differentiate instruction, reduced stress, boosted student self-efficacy, and improved self-directed learning skills. I used this framework while teaching in Chicago and experienced all of these benefits. They wouldn’t have been possible without access to technology.

For guided discussions, I’ll submit the writing tool I’m building. Our theory of change at Short Answer is, in part, to improve guided discussions precisely because they are so valuable for learning. As an example, we recently launched an activity in which AI-characters argue about students’ writing, then the class discusses and votes for which character they agree with. Teachers tell us that kids will come up to the front of the room and read aloud the feedback in the voice of the characters (i.e. Surly Sam, Cool Cali, etc.) and that debates are so lively, they have to cut things off to make sure the activity doesn’t run too long.

These are two of hundreds of examples we could share as evidence of the false dichotomy Dr. Horvath presents. Given they’re somewhat anecdotal (and in the case of Short Answer, could be viewed as self-serving) I would encourage you to check out the previously listed meta-analyses showing causal evidence of improved learning outcomes associated with different digital learning applications.

That causal evidence is ignored by Dr. Horvath. It doesn’t fit his narrative.

5. Neuro-garbage

Dr. Horvath concludes the evidence section of the book by arguing that chemicals and structures in our brains make technology an inherently flawed medium for learning. He leans heavily into his background as a neuroscientist here (“..rulesets must be loaded into a small area called the lateral prefrontal cortex (LatPFC)” etc.).

Some of what he argues is true. Split attention is bad for learning. Technology can be distracting. Empathy and belonging are built around human connection, which screens cannot offer. Technology can sometimes make things too easy.

This is all overstated, as it ignores context.

Most of the studies cited in this chapter were done at the neurological level or in the context of large, undergraduate courses on personal devices with no content filters. Most importantly, neuroscience ≠ classroom practice, because teaching is inherently social and complex.

Dr. Daniel Willingham has referred to these sorts of arguments as nuero-garbage: using neuroscience language and concepts in education that overstate what brain findings can tell teachers about how to teach. To quote Dr. Willingham:

“We can’t take lab findings and pop them right into the classroom… lab findings consistently show that repetition is good for memory. But you can’t mindlessly implement that in schools--”keep repeating this til you’ve got it, kids!” Repetition is good for memory, but terrible for motivation.”

Neurons are not classrooms. They describe what is happening inside a brain; classrooms describe what is happening between people. The behavioral outcomes teachers care about (e.g., reading fluently, solving equations, writing, etc.) are shaped by instruction, motivation, and social context. The technology tools teachers use operate at that same level.

Pointing to a brain region that lights up during a grammar lesson doesn’t tell a teacher what to do on a cold Thursday morning in February when Bart keeps making fart noises and Lisa is mad at Jessica because Steven told her something he wasn’t supposed to, so now Lisa is acting out and distracting Jackie, who is now behind and confused about her grammar lesson.

There’s a gap between knowing that a neural process exists and knowing how to design instruction around it. It’s never one-to-one in the way that Dr. Horvath presents. There are too many ever-changing variables in teaching to reduce it to laboratory science and neurological chemicals. It’s why “good teaching”, and even intervention effect size, are deeply contextual.

The best I’ve seen this explained was by Dr. Dylan Wiliam in his Why Teaching Will Never Be a Research Based Profession lecture.

Brain science doesn’t show that technology is incompatible with learning. It shows that attention, retrieval, feedback, and cognitive effort drive learning. Technology can both support and undermine those processes, depending on how it’s designed and implemented.

For those interested in what good instructional design looks like in digital applications, check out Dr. Karen Forsell’s work. Most of what I know about it comes from her. For those interested in what good technology integration looks like, I’d encourage you to speak with one of the amazing instructional technology coaches at your local school district.

Conclusions

I became aware of Dr. Horvath’s work when I presented at a ResearchED conference in 2024. He gave an inflammatory keynote.

I never published my post about the experience, as I felt it was unproductively inflammatory, too much in the vein of Dr. Horvath’s rhetoric. Well, you’ve just read much of what was going to be in it, so I want to close by explaining why I’m sharing this.

If Dr. Horvath and Ms. Cherkin were simply writing books and selling their consulting services, we could ignore them. They’re not. They’re testifying in front of the United States Senate encouraging lawmakers to strip educational resources from the hands of educators.

Twelve states, including California, Florida, and Missouri, are considering moving beyond cellphone bans and towards laws that will limit “instructional screen time” in the vein advocated for by Dr.Horvath and Ms. Cherkin. Some of these are as blunt as “teachers can only use technology for 45 minutes per day,” regardless of subject, grade level, or instructional purpose.

These blanket screen bans would be a profound mistake. They ignore the difference between poorly designed technology and thoughtful design and implementation. They would remove tools that tens of thousands of educators rely on to differentiate instruction, provide feedback, reduce administrative burden, and respond to the very real assessment challenges posed by AI. They would do so based on cherry-picked data, correlational evidence, logical fallacies, straw men, and moral panic.

We should improve how technology is used in K12 classrooms.

We should push back on arguments that treat technology as inherently incompatible with learning. Basing policy on this narrative would be a mistake.

Put another way: Educational technology is worth fighting for.