An FDA for EdTech and Other Policy Reforms
Three ideas to shift our focus from medium to quality.
30,000 people have read my last two posts arguing against the anti-educational technology movement. I’ve done interviews with NBC News and Edutopia in response to them. Dr. Horvath himself came at me in the comments on LinkedIn. He even offered to meet and discuss the piece, which I was excited to do. That, of course, fell through when public optics were no longer a factor.
On one level, this was all heartening. I’m glad so many folks connected with the research. Many educators reached out with messages of support for the articles and educational technology in general. It was a hell of an ego boost to have national news outlets asking for interviews.
On another level, it’s depressing that the most-read thing I’ve ever written is a tirade that I was uncomfortable posting to begin with. I didn’t fully appreciate how much controversy drives attention. No wonder Penguin Random House has decided to publish Horvath’s manifesto. Controversy sells.
What controversy doesn’t do is serve the best interests of students and teachers. To that end, in this and future posts, I want to share some ideas about what healthy edtech reform might look like.
Right now, it seems that policy is the most pressing domain for reform, as 17 states have now introduced or passed laws limiting or banning “screen time” in classrooms. These proposals are misguided for a simple reason: They ignore quality of instruction. Whether a teacher is using research-backed, high-quality resources and practices is deemed irrelevant. The medium of instruction is, somehow, more important than the quality. ISTE president Richard Culatta pointed this absurdity out recently.
So, under some of these laws, a PreK teacher who wants to use, say, Khan Academy’s free (!!), research-backed digital reading curricula and practice activities would be forced to replace them with a paper-and-pencil alternative. This is true even if the paper-and-pencil alternative is low quality. Coloring in letters of the alphabet with crayons? It’s somehow better simply because it’s analog.
This is nonsense. It’s what happens when we misunderstand a device as an intervention rather than a medium.
A more helpful question for lawmakers: How do we bend edtech towards high-quality instruction? Here are three policy ideas:
1. Require ESSA Level 4 efficacy.
Every edtech product must have at least a research-based rationale as to why its product will improve outcomes for students. In order to become a registered vendor with a district, we could be required to share our ESSA certification along with our W9 form.
The problem here is that certification is not possible without a credible certification body. The current political climate and gutting of the Department of Education make federal oversight difficult, but the need for an objective arbiter has never been more urgent. To move beyond claims, we need…
2. A Food and Drug Administration for EdTech
We need a government body responsible for verifying the educational efficacy and privacy practices of educational technology (and, for that matter, all education interventions). Edtech companies should be responsible for supporting and funding their own efficacy research, both through their own revenue and grants, but there are also misaligned incentives that need to be addressed with this. It’s in our interests for research to come back favorably. It can therefore be tempting to skew, misrepresent, or hide findings.
This happens all too often.
TurnItIn recently misrepresented the efficacy of their AI writing detection product. An independent study showed that TurnItIn “should not be used in academic settings,” because of how wildly inaccurate it is (wrong over 20% of the time). TurnItIn took that research, cherry-picked quotes, and used it to conclude in a white paper that their product is, in fact, “reliable and effective.”
I experienced something similar at a conference recently. A competitor had an adjoining booth in the exhibit hall. Over the course of two days, I heard about how their product “improved writing scores by 20%.” I searched their website and found a case study in which one teacher’s state test scores rose 20% from one year to the next. So the 20% rise was across two entirely different sets of students. There was no control group, statistical analysis, attempt to isolate the software from other factors, etc.
This is all it takes to manufacture “research” in edtech.
These sorts of claims need a “these-statements-have-not-been-reviewed-by-the-Food-and-Drug-Administration” stamp equivalent.
Because, here’s the real rub: I’d be lying if I said, at the end of those long days in the booth, I wasn’t tempted to put together our own case study showing something similar. When a teacher comes up to our booth and says, “Their product improves writing by 20%. Does yours?”... it leaves you in a tough spot.
Do I tell the truth, or do I survive?
It’s easy to say “tell the truth.” But who would you rather work with?
A tool that “improves writing instruction by 20%” or
A tool that “.. is grounded in a research-backed theory of change, as it was my master’s project, but we’re a young company and still in the process of working to understand our efficacy, so we’re still working on causal evidence, but early survey data shows very strong self-reported results from thousands of educators across widely different contexts.”
You can see the race to the bottom forming. Without a uniformly recognized external validation body, claims become equivalent to results.
This is not a new idea. ISTE has a product index. EdSurge has theirs. Common Sense Media had one, but couldn’t keep up with the volume of applications and updates. Johns Hopkins has attempted to pick up the slack after the gutting of the Department of Education and IES, but given that their current program doesn’t even measure instructional interventions related to something as foundational as writing, I have questions about their ability to keep up with the speed and size of the market.
More importantly, I’ve never been asked about any of these indexes during procurement conversations. A government body is needed to create incentives so that I am not just asked, but required, to show that my product actualizes practices that are at least supported by a research-backed logic model.
3. Experiment with outcomes-based contracting
Here’s the problem with everything I just said.
Edtech is ever-changing. We could run a massive, randomized control trial on Short Answer as it exists today and establish causal evidence of improved writing outcomes as a result of using it. Do those findings matter if over the course of the year(s) it took to run the study, we launched 15 new features that completely change our tool? We end up measuring tools that no longer exist.
This is the conundrum of tech and research. Tech moves fast. Research moves slow(wwwww).
A way around this might be outcomes-based contracting (OBC). I became aware of this system through the work of The Center for Outcomes-Based Contracting.
In OBC, schools pay a portion of the full price for edtech products (say 60%). They pay the other 40% when the product demonstrates improved outcomes. Vendors and schools define what expected outcomes look like: “Writing scores on the state test will improve by 15%.” The necessary level of implementation is established: “All teachers will receive at least an hour of training on how to use Short Answer by X date. They will use it at least 3 times per week and complete all content for their assigned grade level by X date.” How each step is documented is determined in advance.
If the implementation plan is followed, but results don’t accrue, companies don’t receive the remainder of their payment.
This aligns incentives.
Schools are incentivized to train staff and use the products they purchase, which has been a problem for a while. If they don’t, they’ll have to pay. Early results show that schools that use outcomes-based contracting have 10X better implementation than the national average. For their part, edtech companies would now also have to focus more on improving outcomes rather than just juicing engagement.
This system is intriguing, but also flawed. Many schools don’t have the bandwidth to manage it. There are just too many stakeholders involved in edtech procurement (the school board, instructional leadership, business office, tech department, building principals, instructional coaches, teachers, etc) to easily align this work. Early research echoes these concerns.
As someone in the trenches of edtech procurement, I can tell you from first hand experience that many districts struggle to even manage existing purchase order processes. The idea that they could handle conditional constraints on contracts, measured based on a series of implementation checkpoints and assessment measures, seems like a recipe for a mess.
It’s not just schools though. Many companies would struggle to manage this process. The edtech market has always been a challenging one to build a successful business in. It’s given rise to more micro-companies like ours at Short Answer: Intentionally small teams that don’t scale as the business scales. This structure allows us to better focus on impact and sustainability, but the idea that our small team would be able to handle the logistics of outcomes-based contracts at scale seems far fetched. We’re already stretched to the limits of our bandwidth.
It’s why I’ve gone back and forth on this. In my interview with Edutopia, I actually talked about why I think outcomes-based contracting is a bad idea. But as I’ve read more about it, I think it could be a good option to help deal with the asymmetrical nature of educational research and edtech development. We’ll need more experimentation.
In sum
The “BAN IT ALL!” laws sweeping the nation are misguided and lazy. They ignore the quality of instruction in favor of the medium. It’s nonsense, but it’s easy to legislate, so it’s what we’re doing.
Building an “FDA for Edtech,” enforcing real efficacy standards, and rethinking contracts is not easy. It requires bureaucracy to move faster and school districts to build capacity they currently don’t have. But the alternative is what we have now: a race to the bottom where the loudest claims win, a market in which operating with integrity starts to almost feel self-righteous, and a seemingly never ending cycle of controversy and debate.
Controversy generates article views, gets you interviews with the press, and sells books. It doesn’t improve outcomes for kids.
We don’t need fewer screens. We need higher-quality instruction and verified results.
Great piece as always Adam. One note about the outcome-based contracting from my perspective: it's almost impossible to confidently attribute results to single interventions in educational settings. Kids, classrooms, teachers, schools, systems, initiatives, home life, economic factors, and so much more serve to muddy the waters in ways that seem rife for manipulation. And the biggest companies with the most resources will be the best at manipulating things to look more reasonable than they are. Cynically, I'll also add that our school systems—of which I'm a part of—are frankly not very good at accurately and critically interpreting data. In my opinion, this is why so many edtech companies put so much focus on usage data instead of efficacy data when they meet with educational clients. They know that it looks and sounds compelling even if it doesn't mean much of anything as far as student learning is concerned.