A Few Thoughts on Impact and Efficacy in EdTech Buying Decisions

Ah, the beauty of the social media news feed. Dropping into Facebook or LinkedIn ensures a never ending source of content for all of us. Thankfully, if we choose our friends well, we can still get a few thought provoking pieces that pop up around all the sponsored posts and links to your buddy’s latest vacation photos. This is how I came across episode #82 of the EdTech Podcast and an interview with Bart Epstein, CEO of the Jefferson Education Accelerator at the University of Virginia. Host Sophie Bailey (the founder of The Edtech Podcast) talked to Epstein on the important — yet often emotionally charged — topic of the efficacy and impact that edtech products yield in classrooms today. In their discussion, Bailey and Epstein also touched on some related ideas of how efficacy and impact can more broadly frame the difficult buying decisions that educators and consumers face when considering edtech purchases. [Note: For those that want to jump right to Bailey’s interview of Epstein, it starts at the 6:00 mark]

With efficacy and impact it’s helpful to work towards precise definitions of what these terms mean in education. I’m going to switch the order here and start with impact, or as my Merriam-Webster online dictionary defines: “the effect or influence of one person, thing, or action, on another.” As an example, if an edtech product (a “thing”) can influence the attendance rate of a student (a “person”), we can use this construct to define the potential impact (or lack thereof) of a specific edtech product or service. Now, how about we try to marry impact with efficacy, or again as M-W describes: “the ability to produce a desired or intended result.” Here we need an entity, like say a school principal or district administrator, to make a determination that the aforementioned edtech product improved (or at least helped improve) the student’s attendance by an amount deemed sufficient by the principal or administrator.

So where should edtech buyers start on the efficacy and impact spectrum? First, focusing for a minute on school and district purchases, we need educators as industry experts to define the impact they consider most critical to measure in the form of student outcomes — e.g. improving attendance, homework completion rates, grades, high school graduation rates, or post-high school matriculation to higher education opportunities. Next, we need our educators to define the efficacy they expect to see from the investments they make in edtech that will define those purchases as critical to the long-term success of their schools or districts. Based on this framing, a few important concepts are worth expounding on per Epstein’s interview.

For all edtech startups racing against the clock to show meaningful business traction, research oftentimes becomes an expensive and difficult element to prioritize. But Epstein rightly points out the importance of investing to demonstrate impact and efficacy in edtech products. There are thousands of education focused researchers willing and able to help support the mission of education-based companies, and many of these folks will adjust their rates accordingly to the stage of the startup. Further, finding these researchers need not be difficult. Thanks to Epstein and his colleagues’ work at UVA, the National Education Researchers Database (NERD for short, of course!) provides a resource to track down research partners that best map to the edtech problem a company attempts to solve for educators.

In reality, every edtech company might argue that they are already doing lots of research on their products. Indeed, companies perform a good amount of what Epstein refers to as “internal” research that is geared to gathering product development insights and learnings that help define use cases, customer benefits, and ultimately the features that get built into an edtech product or service. However, the gap that still exists is the “external” research that helps product development teams narrow the distance between what the internal findings tell us and what external research finds linked to the myriad of situations and implementations that edtech products must simultaneously address.

Today, a ton of this external research happens every day as educators use edtech products and see results — both positive and negative — from the use of these products. What’s missing according to Epstein is a way to document the learnings from these thousands of implementations already taking place. Rather than relying on “word of mouth” recommendations where the n=1 sample size response from one educator saying to another “I think that product is great” serves as the basis for a buying decision, how about creating a simple database where educators can document their experiences and fellow educators can tap into this “wisdom of the crowd” to help make these important decisions. Think of a true “Consumer Reports” for edtech products and services (Epstein’s concept, not mine — but an interesting business opportunity for some young entrepreneur?).

As edtech companies we need to find ways to invest more in demonstrating impact and efficacy of our products, and we need to be particularly sensitive to the different situations and implementations that are important considerations for the buyers of our products. At the same time, however, we need these buyers— both institutions and consumers — to meet us on our path in the following ways:

• As highlighted above, buyers should be clear on the outcomes they value most. If a district or school, is that attendance, homework completion, grades, graduation rates, or something else? If a parent or teacher, is success measured in terms of a student reading at grade level, a student developing a growth mindset in how they learn, or a parent becoming more engaged in their child’s education? The more clearly buyers can articulate what success means to them, the more precisely those of us in edtech can engineer our products to help deliver those outcomes.
• Once buyers are able to define what success means to them, we need them to become more sophisticated buyers and, to the extent possible, base their buying decisions on research that demonstrates impact and efficacy. As easy and assuring as it can often be to rely on word of mouth and/or recommendations from one or two close friends, in many cases edtech product results will be dependent on unique situations and implementation. Furthermore, Epstein rightly highlights the dangers of applying a simple “what works” application to any edtech product that receives a stamp of approval from a single school or district. Thus, school, district, and consumer buyers should do adequate “homework” to ensure that their purchase decisions are valid per research that aligns with their unique situation and implementation.
• Finally, we need schools, districts, and individual edtech consumers to embrace products that clearly enhance student outcomes across all situations and implementations in terms of their efficacy and impact. Communication services like Remind.com that help build relationships amongst teachers, students, parents, and administrators represent such a foundational layer for improving outcomes regardless of the school, student profile, or other unique situational elements. Likewise, a robust educational instructional content offering like Khan Academy delivers benefits to students across the full spectrum of educational situations and implementations that students might experience.

Listening to Bart Epstein’s interview is a good reminder of not only how important the concepts of impact and efficacy are to how edtech companies think about their go-to-market efforts, but it also reminds us of the important role that buyers of edtech products play in ensuring that students benefit from technology services that are brought into their learning environments. Moreover, we are reminded that we must commit to a future in which every student has access to edtech products and services — not just the kids in affluent school districts but those working hard to achieve success in Title 1 schools as well.