How effective are USAID programs?

Samantha Power is taking the helm of USAID at a time when the agency is being tasked to address numerous crises around the world. Foreign aid is a perennial punching bag on Capitol Hill, and USAID does important work that needs to be funded — but it also needs to do more to show that its efforts are achieving U.S. foreign policy objectives.

Since 9/11, USAID has been forced to expand beyond its historic model of doing long-term development work and to implement programming such as conflict prevention, countering violent extremism, stabilization — both during and after conflict — and quick-impact projects to support counterinsurgency operations.

With traditional development work, the results can be years, if not decades, away and so it can be difficult to measure the progress and outcomes of development work in the early stages. However, stabilization or conflict prevention programming is by design meant to have a more immediate impact.

Therefore, such programs require evaluation metrics and methodologies that can capture program impact in a shorter time frame. In addition, it is critical to have an effective feedback loop to determine whether a program achieved objectives and should be continued or expanded, or whether a program should be modified or scrapped.

This opens a parallel discussion about whether strategic goals are developed properly with achievable outcomes based on the conditions on the ground and available resources. Many would argue that process is flawed, and I can certainly point to projects I observed in conflict zones and fragile states in my time as a journalist and at the Department of Defense Office of Inspector General that were poorly designed, had unrealistic goals, and failed to further larger policy objectives. However, for now I want to keep the focus on the evaluation component, since better evaluation would improve the planning and project development process.

In the years I spent in Afghanistan, I asked multiple USAID country directors how the agency assessed the effectiveness and outcomes of projects that ranged from building health clinics to empowering women to social media summits. The answer was routinely that outcome measurement is difficult, especially in conflict zones, and therefore USAID measures quantifiable data referred to as “inputs” and “outputs” — the program cost and the number of “things” to be built or delivered.

For example, a hypothetical program designed to improve the capacity of Afghan lawyers to follow the rules of evidence in a trial would measure the number of workshops held, the number of attendees who completed the training, and the cost. If those inputs and outputs were met, the program would be viewed as a success. Rarely would there be any follow-on evaluation to determine if lawyers were implementing the knowledge from the training, if that was leading to fairer trials, and if that was improving the legitimacy of the Afghan government, which was the ultimate policy objective.

Similarly, education in Afghanistan has been touted as a success story because of quantifiable data — the number of schools built, the number of teachers trained, the number of children attending schools. Those data paint a positive picture. However, as I reported in 2013, that information does not tell you anything about the content, quantity and quality of actual education being delivered and the difference it is making in the lives of students. Spoiler: The education Afghan children are receiving is not nearly as substantial and impactful as advocates claim.

In my time in Afghanistan, I followed and reported on hundreds of millions of dollars of USAID programs. Many of those programs made a lot of “noise” and were trumpeted by the agency as “good news stories.” However, from everything I could tell, many programs did little to create lasting change or build the capacity of Afghans to run clinics or schools or social programs. Again, the program outputs might have been significant, but the outcomes were not.

Without outcome assessments, how can USAID (and the State Department as well, which also funds stabilization programs) make informed decisions about what programming to conduct? How can Congress determine appropriate funding levels? How can the U.S. foreign policy establishment determine if goals are being met and evaluate whether the goals are the right goals?

To be fair, USAID does have internal guidance and requirements regarding outcome monitoring, but I have not seen a lot of evidence of it in practice. Yes, outcome measurement is difficult. It is often subjective. It takes more time and requires more expertise. Just because it is hard is no excuse not to do it. Calling a program successful because it spent the program funds on time and conducted the required engagements is a recipe for enriching USAID contractors and implementing partners at the expense of taxpayers while doing little to help targeted communities.

USAID will continue to be tasked with doing stabilization work and programming in the Middle East, Africa, and other regions designed to prevent conflict and reduce the influence and appeal of violent extremists. That work is vital to U.S. national security interests and can reduce the need for more costly military interventions. There is finite capacity to implement that work on the ground, and there are limited dollars available. Taking the time and spending the extra money up front to assess the effectiveness of programs will improve the design and implementation of programs and allow for more efficient use of limited resources.

Sean D. Carberry is a foreign policy and national security writer. He most recently served at the Department of Defense Office of Inspector General as managing editor of the Lead Inspector General reports to Congress on overseas contingency operations. Previously, he was a foreign correspondent for NPR based in Kabul and has reported from more than two-dozen countries including Iraq, Syria, Libya, and Yemen. Follow him on Twitter @sdcarberry17.