Thursday, December 3, 2009

Pre-test of Project Group with Post-test of Project and Comparison Groups, Design #5

The next set of project evaluation designs that I will presenting are considered weaker than the previous designs; weaker meaning that they can not provide good evidence that a project's interventions are directly attributable to the measured outcomes or to what degree the project contributed to the measured outcomes (net impact).

The pre-test with a project group combined with a post-test of both project and comparison group design is shown below. As the name implies, the pre-test (baseline study) is conducted only with the people involved in the project. There could be a number of reasons why this might occur: save money, project team didn't like the idea of a comparison group initially, or some technical reasons. However, later there may be the budget, interest or feasibility to include a comparsion group.

The advantages of this design are that it can assess reasonably well how a project is being implemented and whether intended outputs from activities were produced. IF the comparison group studied at the end of the project is quite similar to the project group in characteristics, and adequate mixed methods can demonstrate the comparison group were similar at the baseline as the project group on the outcomes that will be measured, then this design MIGHT demonstrate project effects.

The disadvantages are that is that it is difficult to conclusively determine if the differences between the project and comparison group at the end-line study are due to the project or other factors.  Another weakness of this design is that local context events can effect outcomes in the comparison group. For example, an agricultural project compares agricultural output of its farmers to a comparison group of farmers without knowing that the the comparison group of farmers received more irrigated water than previous years thus increasing their output. To monitor local context events, retrospective information can be obtained from the comparison group.

Truncated Pre-test and Post-test with Comparison Group, Design # 4

How do you evaluate a project if no baseline was conducted and the project is being implemented? One possible approach, if a project has an adequate budget and has a reasonable amount of time left before it is completed, is the use of a truncated pre-test and post-test with a comparison group.

The truncated, or shortened, quasi-experimental project evaluation design uses a mid-term study as a proxy measure for the baseline even while recognizing that the project has been underway for some time. Again, as in all the other more rigorous designs, a comparison group is studied so as to estimate the net impact of the program interventions; however, with no baseline this design is weaker than the previous ones.

The post-test or final study, is conducted at the end of the project. One advantage of this design is that smaller sample sizes can be used since the time frame is shorter which reduces the possibility of respondent loss (attrition). One of the drawbacks of this design is that without having a baseline measure it is not possible to know the total amount of change over the life of the project, rather inferences have to be made based on contextual analysis and incorporating mixed methods approaches.