An Implementer’s Guide: Running Low-Cost, High-Quality Studies to Generate Impact

By: Olivia Shoemaker and Dr. Abubakar Umar

This blog post was written by The Taimaka Project, one of Azurit Foundation’s first grantees.

The Taimaka Project got its start because our founding team was inspired by the promising results of several randomized control trials (RCTs) looking at food insecurity in sub-Saharan Africa.

Over the past several years, as we’ve implemented our own programs to tackle childhood malnutrition and conducted an array of studies, we’ve become evangelists for the value of academic partnerships in generating novel, evidence-based, and scalable solutions to some of the world’s toughest problems. But along the way, we’ve also learned a lot of things we wish we would have known when we started:

• When is it worth having an academic partnership vs. doing a more informal study in-house?
• How much work goes into an academic partnership? (spoiler: more than we thought)
• How much does it cost in time and resources?
• And of course, what’s the point? Does it lead to impact?

How We Ended up Here

Taimaka is a joint Nigerian-American NGO, focused on tackling acute childhood malnutrition – through treatment programs in northeastern Nigeria, and through innovation designed to make treatment available to every child who needs it. We run studies with academics to test scalable solutions to make malnutrition treatment cheaper, more effective, and ultimately more accessible for the 45 million children who need it.

Here are a couple of examples of our academic partnerships:

• We’re running a clinical trial with UC San Francisco to test whether swapping an antibiotic called amoxicillin with azithromycin can reduce mortality rates among malnourished children.
• We’re running an RCT testing whether automated voice calls can reduce treatment program dropout rates with University of Florida researchers.

The Opportunity for Impact through Academic Partnerships

We want every dollar we spend to do the most good possible, and to do that, we need rigorous evidence of impact and cost-effectiveness. Academic partnerships help in several ways:

1. Impartiality: Good academics are driven by a passion for pushing their field (not an organization) forward.
2. Rigor: Academics bring a set of quantitative and qualitative skills that help design higher-quality processes for data collection and analysis.
3. Credibility: Leading academics can bring the reputation and networks necessary to share the results with people and organizations who can adopt new solutions.

All three of these benefits act in service of generating evidence-based, scalable impact. Let’s take our field (childhood malnutrition treatment), for example. If we can reduce the cost of treatment through innovation by just 1%, we would free up $3.4 million currently put towards programs, which could be reinvested to treat 50,000 additional children every year, in perpetuity.

So how do you run a low-cost, high-quality study to generate impact? Here’s a high-level, step-by-step guide based on lessons we’ve learned along the way.

Step One: Decide Whether an Investment in an RCT or Formal Study is Worth It

Academic partnerships are only worth it if you have a research question that can push the field (rather than just your organization) forward. Most academics aren’t interested in simple program evaluations, especially if there’s already evidence in your field for your intervention.

A good way to gauge whether a research question is interesting to the field more generally is to reach out to researchers at universities to chat about what they’re working on. In many cases, we’ve seen meaningful partnerships grow out of quick emails.

Step Two: Focus on Cost-Effectiveness and Potential for Scale, not just Statistical Significance

The real value of partnership comes from holding implementation experience and the rigor of academia hand-in-hand. We’ve read a lot of fascinating studies that have great ideas but are ultimately impractical for our implementing context or too expensive per unit of impact to be worth implementing.

We’ve found that it’s useful to consider two factors in addition to statistical significance:

1. Cost-Effectiveness: could whatever is being tested result in cheaper and more effective impact?
2. Scalability: is this solution adoptable by other actors (e.g., NGOs, governments), and is it generalizable beyond a given context (e.g., could it work in other countries)?

Step Three: Budget More Time, Skill, and Resources Than you Think You Need

When we started one of our first partnerships, we had already run a small, in-house RCT and felt like we had a grasp on how much a larger-scale RCT would be. We were wrong.

Getting high-quality data requires concrete investments in time and data monitoring systems on a near-daily basis. Throughout the first couple months of collection, we were consumed by calling clients to perform back-checks, monitoring whether staff had recorded data correctly and accurately, and checking for anomalies and biases in sampling. Any lapses in our time and attention would have directly resulted in lower-quality data.

Because we were excited about the promise of better evidence, this time was 100% worth it. But ever since, we’ve been more intentional two things:

1. Realistically budgeting the time of our study leads. Oftentimes, we now hire additional data research assistants or field managers to support our study leads.
2. Setting study launch dates at least a month after we think we’ll be ready. Giving ourselves a bit more of a cushion for launch forces us to think proactively about where gaps in our personnel structure, supply chain, or tech are.

Step Four: Make it Cheap by ‘Paying’ for Direct Impact

RCTs can cost millions of dollars to execute in the field. We’ve run RCTs and clinical trials for around $20,000.

We can dramatically cut the costs necessary to run a study because we already ‘pay’ for direct impact through our program costs. In other words, we run our data-driven malnutrition treatment program in northeastern Nigeria regardless of the research that we’re doing. The routine data we collect is key to our program’s success, but also means that we have a ready-made setup for high-quality research. So when we develop a partnership, our only costs are the additional logistical tweaks to our program in line with study protocols and the cost of additional data collection.

This also means that even if a study doesn’t generate a significant, cost-effective, or scalable result, the vast majority of the funding we put into our program still yields impact as a result of the direct effect on our beneficiaries. If one in ten (or one in twenty!) of our studies, for example, resulted in a 1% reduction in the cost of malnutrition treatment globally, again, an additional 50,000 children could be treated every year. At only around $20,000 for a study, that’s a remarkably attractive value proposition.

Step Five: Generate Impact by Using What You’ve Learned

Of course, better evidence doesn’t matter unless it’s followed.

When we started, we made an organizational commitment to spend every dollar to do the most good possible. One of our first RCTs, an impact evaluation of a post-harvest loans program for smallholder farmers in collaboration with UC Berkeley, found that the program was less cost-effective at tackling food insecurity than we hoped. When we saw the evidence, we made the tough decision to shut the program down and reinvest in our more cost-effective malnutrition treatment program.

Evidence, regardless of whether it shows a success or a failure, should also be shared externally to other implementers, academics, and funders who are interested in the issue area. The value proposition we discussed is only true if everyone in the field learns from and follows the evidence.

Help us Continue to Learn

We are always eager to hear from other implementing organizations, academics, or funders who have experience generating impact from evidence! If that’s you, we’d love to connect. Check us out at taimaka.org or email external@taimaka.org to get in touch.