Second Most Robust Quantitative Evaluation-Design #2

In this series on various types of quantitative project evaluation designs, let's look at another design that is considered very rigorous using quasi-experimental methods. This design is quite similar to Design #1 presented in an earlier blog, in that two groups (participants and a comparative non-participant group) are studied over the life of the project; HOWEVER, this desgin does not include the the 4th study that was in Design #1, the post-project follow-up, but rather has three: baseline, mid-point, and end-line.

As in the most rigorous project evaluation design, what makes this design rigorous are a) the use of a matched comparative group who help establishe the counterfactual [i.e., what would have happened if the project had not occurred], and b) measurements taken at three-points in time. What makes this design slightly less rigorous is without the post-project study the sustainabiliy or trajectory of the results is not known. In other words, after a certain period of time with no interventions, were the results among the participant at the end of the project able to be sustained, increased or did they eventually decline?

Many of the limitations that applied to the first design also apply to this design, which partly explain why this design is not often used among NGOs. First, it requies more time and costs to collect data at 3 points in time and among two groups. Second, the sample size must be relatively large to account for loss or attrition of members in both groups over this period of time. Third, data management and analysis can be a challenge.

In an effort to clearly demonstrate "What Works" this is a project evaluation design that should be considered more often than is currently being used, especially for longer-term projects that span 3 or more years. BUT, such designs must be included at the propsal development phase, otherwise trying to fund, arrange and organize such a project evaluation becomes difficult.