Why NGOs Are Hesitant to Share Lessons Learned

Ricardo Wilson-Grau Consulting (17 April 2007 from CSO survey) asked a civil society organizations (CSOs) if they had lessons learned and, if so, how many were not from just "better practices" but just as importantly from "bad practices." Surprisingly, virtually none of the CSOs reported sharing "bad practices" as one aspect of lessons learned.
A follow-up question was asked: Why are so few CSOs willing to share their "bad practices" so that others can learn from them. The question had 5 closed-ended responses and 1 open-ended response. The result were:

1. Organizations are reluctant to think about negative experiences - 36.6% 
2. They are uncomfortable sharing weaknesses with a donor - 64.1% 
3. They are uncomfortable sharing weaknesses with other organizations - 57.3% 
4. Organizations have little information and knowledge available to explain failures - 36.6%
5. Organizations are interested in what does work and not in spending time on what does not - 35.9% 
6. Other: 36.6% were by and large nuances of the five multiple choice options above.
   

In a blog by Daniel O'Neil (The Change Agent), he cites the paper, "Lessons Not Learned: Why don't NGO workers collaborate more?" by Wade Channell. In this paper, Wade highlights why development workers are lousy at learning from each other. Although we might be friends and socialize together, the world of international development workers does not foster learning. He cited four problems of learning:

• Incentives for Knowing, But Not for Learning
• High Incentives for Repetition, Low Incentives for Innovation
• High Incentives for Guarding Information
• Disconnection between Performance and Awards

All too often staff are hired because of their proficiency in an area, and due to the pace of project implementation, there is little time or incentive to read, reflect and interact with others in a learning process. Access to conferences, workshops, and journals for those workers in the field are both a physical and financial challenge. Thus, bad practices can "creep" into projects because staff are not able to keep up-to-date.

As repetition, Daniel states, "Donors only want to fund proven successes and NGOs write their proposals to satisfy what the donor wants to hear. This is especially critical when entering a competitive bid. The NGOs seek to divine what the donor wants to hear, rather than to come up with the best approach. The Gates Foundation has made significant waves because they are willing to fund projects that take risky approaches."

When it comes to guarding information, occasionally, project staff and NGOs are not willing to share project evaluations due to potentially unfavorable findings. Also, there are not many forums to share project evaluations even with the same organization so "bad practices" can be avoided.

Finally, the fear is that to expose "bad practices" and the lessons learned from that will not be rewarded. Certainly, good planning should reduce bad practices, but no project can be completely flawless.

How Many In-depth Interviews Are Enough?

Have you ever had a limited amount of time and budget and wondered what are the fewest number in-depth interviews you could get by with but still get an adequate amount of data and information on a specific topic?

In the article, "How Many Interviews Are Enough? An Experiment with Data Saturation," (Field Methods, Vol. 18, No. 1, February 2006) the authors Greg Guest, Arwen Bunce and Laura Johnson investigate this question.

Specifically, these authors were interested in the minimum number of in-depth interviews does it take to get a reliable sense of themes and issues and variability. That is, does it take 6 interviews, 18 interviews, 100 interviews to render a useful understanding of most of the issues; or another way to ask the question is, when does adding more interviews not make a difference in rendering substantial more information? When is enough?

To answer these questions, they conducted a study among a group of women (sex workers) in two African countries. The in-depth interview guide consisted of six structured demographically oriented questions, sixteen open-ended main questions, and fourteen open-ended sub-questions. To determine the degree of data saturation (useful understanding of most themes/issues), the authors used the point in data collection and analysis when new information produced little or no change to the codebook.

After collecting and analyzing their data, data saturation occurred at when they had analyzed 12 interviews.That is, 92% of the total number of codes they developed for the entire study were developed by the 12th interview.

Benefits:

1. Maximum data saturation obtained with minimum number of interviews.

2. Time and cost-effective

Limitations:

1. Must be done with a purposive sample: people specifically interviewed because of their knowledge or experience related to the specific topic.

2. The individuals should be relatively similar (homogeneous), for example female sex workers, street children, etc.

Participatory Video: A Qualitative Method of Monitoring & Evaluation

The following have been taken from Insight Into Participatory Video: A handbook for the field, written by Nick and Chris Lunch, 2006. Published by Insight. (Find this publication under DME Documents.)

Participatory Video (PV) is a set of techniques to involve a group or community in shaping and creating their own film. The idea behind this is that making a video is easy and accessible, and is a great way of bringing people together to explore issues, voice concerns or simply to be creative and tell stories. This process can be very empowering, enabling a group or community to take action to solve their own problems and also to communicate their needs and ideas to decision-makers and/or other groups and communities. As such, PV can be a highly effective tool to engage and mobilize marginalized people and to help them implement their own forms of sustainable development based on local needs.

Insight has its own YouTube channel that allows you to view PV from around the world. Insight's YouTube channel is located at: http://www.youtube.com/Insightshare

Nick and Chris Lunch were recently interviewed as part of OneWorldTV's series focusing on pioneering individuals and organizations using video as a tool for social change.

How Does It Work

• Participants (men, women and youth) rapidly learn how to use video equipment through games and exercises.
• Facilitators help groups to identify and analyze important issues in their community by adapting a range of Participatory Rural Appraisal (PRA)- type tools with PV techniques (for example, social mapping, action search, prioritizing, etc. See ‘Chambers’ in Appendix 7, References).
• Short videos and messages are directed and filmed by the participants.
• Footage is shown to the wider community at daily screenings.
• A dynamic process of community-led learning, sharing and exchange is set in motion.
• Completed films can be used to promote awareness and exchange between various different target groups. Insight has worked with pastoralists, farmers, marginalized communities and youth in rural and urban settings, street children, refugees and asylum seekers,people with mental health problems, learning difficulties and physical disabilities (see Part Five, Case Studies). PV films or video messages can be used to strengthen both horizontal communication (e.g. communicating with other communities) and vertical communication (e.g. communicating with decision-makers).

Tichezerane AIDS Support Group Participatory Video

What Does PV Offer

PV engages: Video is an attractive technological tool, which gives immediate results.

PV empowers: A rigorous but fun participatory process gives participants control over a project.

PV clarifies: Participants find their voices and focus on local issues of concern.

PV amplifies: Participants share their voices with other communities, including decision-makers.

PV catalyzes: Participants become a community, which takes further action. PV is inclusive and flexible: Insight have worked with a wide range of groups in the UK and internationally.

PV is accessible: Findings, concerns and living stories are captured by communities themselves on video; projects

can be documented and evaluated; policy information and decisions can also be transferred back to the community level through PV.

PV equips people with skills and positive attitudes: Skills developed include good group-working skills, listening skills, self-esteem building and motivation techniques; PV projects encourage better awareness of community, identity and place; PV develops an active role for participants in improving their quality of life.

PV disseminates good practice: A range of impressive initiatives and suggestions can be documented by those directly involved, cheaply and effectively, and shared across the country and even further abroad;policymakers can be deeply affected by powerful stories and images captured in this way at, and by, the grassroots.

Waiting for Water

Applications of PV

• Marginalised social group to wider community: showing a PV film made by one group and using as a tool to stimulate discussion and participation among other groups in society. Participants may want to conduct filmed interviews to gauge reactions among the audience and record feedback. Facilitators can use such screenings to identify and congregate new groups to work with using the same PV methods.
• Community to community: produced films shown to other communities and used as a tool to inspire and initiate same process of analysis and local action in the second community. Spreading impacts of the work and "PV strikes me as especially well suited to enabling rural people, after only a little training and at moderate cost, to create vivid accounts of their own experience. Very suitable for sharing with their counterparts elsewhere in the country or even abroad." Claire Milne, ICT Telecoms Consultant awareness raising, but also a chance to bring in new groups, highlight differences as well as similarities.
• Community to community PV exchange visits: introducing PV into this process as a tool for wider sharing, equitable exchange and team building (i.e. focusing on a shared task and having fun together!). Exchange visits can be costly and usually only benefit a handful of community members, with PV the learning and exchange can be documented enabling the wider community and other communities to benefit from the exchange.
• Policy to community PV visits: as with the community to community PV exchange visits above, but getting policymakers to the field. This can be difficult to arrange and maybe only one or two individuals can be prized out of their offices! A policymaker sharing a PV documentation task with the community members can be a good way to equalize relationships. They will have fun together and create something which the policymaker can show to his/her network of colleagues and superiors.
• Facilitating multi-stakeholder workshops using PV: A means of getting different groups together on a more equal footing, empowering populations who feel uncomfortable in a workshop setting, or are illiterate. Community members present their films and these become the starting point for discussion and group work which is all documented using PV tools rather than written notes. This also allows the workshop outcomes to be shared widely among communities, personal and professional networks of the workshop participants and the general public (if relevant).
• Campaigns: PV has tremendous potential to bring out personal stories to support campaigns and build understanding and consensus in potentially fraught situations. Decision-makers may respond better to the voices of people on the ground than to organizations, academics or activists campaigning on their behalf. Participatory videos are raw, direct and show a fuller picture of what is at stake.
• Participatory Research: Generate knowledge, initiate local action, raise awareness, monitor and spread widely.
• Community-led Research: Assist groups in the target communities to carry out their own research using the video as a tool for them to document local knowledge and ideas, as well as generate new knowledge and fresh solutions. Local people’s findings can be included in multimedia reports and publications, bringing their authorship into the process and developing a synthesis of local and scientific knowledge.
• Participatory Monitoring & Evaluation: Using video rather than an attitudes survey to look at progress during the research can put the community in control. It is visual and accessible to all. It allows the community to highlight issues and areas of interest that we could not necessarily conceive of as outsiders. Things emerge from the films they produce that open up new lines of enquiry and can also help shape the kinds of quantifiable questions partners focus on.
• Sharing Best Practices: The groups involved can document and communicate their achievements in their own words. The use of PV to collect and share the best practices and lessons learned. Often while collecting the lessons learned staff and experts obtain the information from the project implementing parties and having analyzed such information they may then prepare the manuals and adjust the vision expressed by the local communities while looking at such data from their own professional perspective. When receiving the project outcomes and developments, NGOs and local communities may have difficulty to fully understand the essence of the project outcomes. The use of PV can enable people to have a virtual interaction with their colleagues from other villages. While watching video material they obtain the information directly without the "university" filter of the professionals."

Participatory Video made by semi-nomadic shepherds of Kazakhstan.

Psycho-Social Programming for Children in Crisis

In 2004, Save the Children produced a handbook on psycho-social programming for children (Children in Crisis: Good Practices in Evaluating Psycho-social Programming, by Joan Duncan and Laura Arntson). You can download this handbook under the DME Documents section to the right. Those country offices doing, or considering psycho-social programming, should read this handbook if you have not already. It provides a good overview of psycho-social programming, from theory, definitions, issues, types of interventions, and measuring program/project outputs and outcomes. And, there is a very helpful chapter that discusses the differences between outcome and impact measurement.

Some short summaries from the handbook are:

What does psycho-social refer to? “The term “psychosocial” implies a very close relationship between psychological and social factors. When applied to child development, the term underlines the close, ongoing connections between a child’s feelings, thoughts, perceptions and understanding, and the development of that child as a social being in interaction with his or her social environment.”

What are the levels of severity children face in crisis? 1) Severely Affected Group- these are children in which their psychological and social functioning abilities may be severely compromised. While generally a small percentage of the overall population, this group requires intensive psychological attention because they are unable to manage on their own. Children forced to view and/or commit violent acts, such as child soldiers, are likely to fall into this group. More time-intensive, individualized approaches are likely to be the most appropriate responses, where social and cultural resources permit. This group is in need of one-on-one attention in order to address the more severe traumatic and/or depression disorders, for example. 2) At-Risk Group- A second segment of the community consists of those who have experienced severe losses and disruption, are significantly distressed, and may be experiencing despair and hopelessness, but whose social and psychological capacity to function has not yet been overwhelmed. Children in this category may be suffering from acute stress disorder (the most extreme, or exaggerated normal reaction to violence and trauma). They may have lost family members in the violence, they may have witnessed deaths, or they may be victims of violence. This group is at particular risk for psychological and social deterioration if their psychological, social, cognitive, and development needs are not addressed through timely community and social support mechanisms. 3) Generally Affected Group- The third and broadest segment of the population consists of individuals who may not have been directly affected by crisis events and whose families may be largely intact. Children in this group may be suffering from physical and mental exhaustion, for example, but are not experiencing the level of distress felt by those in the severely affected or at-risk groups. Community-based interventions that include not only normalization activities but also theme- and body-based activities can preserve and augment positive coping strategies among this population in a shorter time-frame and contribute effectively and more immediately to children’s and youths’ social, cognitive, and emotional development.

What are psycho-social programs/projects? Child-focused psychosocial projects are those that promote the psychological and social well-being and development of children. The orientation here is that child development is promoted most effectively in the context of the family, community, and culture. At its most fundamental level, psychosocial programming consists of activities designed to advance children's psychological and social development, to strengthen protective and preventive factors that can limit the negative consequences of complex emergencies, and to promote peace-building processes and reduce tensions between groups.

What are the primary issues psycho-social program attempt to address?

+ Secure attachments with caregivers - Child feels safe and cared for by supportive adult caregivers.

+ Meaningful peer relations or social competence - Child has the capacity to create and maintain relationships with peers and adults. Feels he/she is able to effectively navigate his or her social world.

+ Sense of Belonging - Child is socially connected to a community and feels he/she is part of a larger social whole. Child adopts the values, norms and traditions of his/her community.

+ Sense of self-worth and value, self-esteem, well-being - Child thinks of him/herself as worthy and capable of achieving desired goals. Child has a sense of empowerment and a sense of being valued. Child participates in larger community and feels in harmony with norms of his/her society. Child has the capacity and/or possibility to participate in decisions affecting his/her own life and to form independent opinions.

+ Trust in others – Child has a belief that he/she can rely on others for nurturance, help, and advice. Child feels that he/she will not be hurt by others.

+ Access to opportunities – Child has a sense of being in a supportive environment. Child has access to opportunities for cognitive, emotional, and spiritual development and economic security.

+ Physical and economic security – Child’s physical health, livelihood/economic security and environment are supportive and do not pose threats to the child’s emotional or physical wellbeing.

+ Hopefulness or optimism about the future – Childs feels confident that the world offers positive outcomes and a hopeful future.

Should psycho-social programs/projects be similar to each other? Some elements of psychosocial development are specific to a particular culture, meaning that there is not a “one size fits all” approach to psychosocial programming. A key challenge facing project designers is how cultural factors minimize or increase risk, and promote or impede resiliency. However, child development theory and research does point to a set of concepts that are useful building blocks for psychosocial projects regardless of where they are established. These include understanding what makes children resilient and the role that protective factors play throughout development. Identifying the ways these concepts are expressed within a particular culture should guide psychosocial project development and implementation. Through the study of children who have grown up under difficult circumstances, we have learned that some have certain characteristics and social supports that have enabled them to overcome adversity. Similarly, features of the social world have been identified that buffer the consequences of negative experiences on children. These features are often referred to as protective factors.

What are the content areas for interventions? Since children and adults experience and react to complex emergencies in unique ways, the types of projects designed to address their needs will also differ. Projects range and include those that are curative, preventive, and those that promote psychosocial well-being. Curative projects address the diagnosed psychological effects of complex emergencies on children and families, such as treatment of trauma. Preventive projects seek to prevent further psychosocial deterioration and may focus on a particular group or social environment. Lastly, projects may seek to promote healthy psychosocial development through, for example, opportunities to engage in educational, social, and spiritual activities that support the development of children.

What are the basic intervention approaches? There are different approaches to psychosocial programming, depending on the population being targeted and the project to be implemented. It is possible to identify three major groupings:

1. Psychological: Some projects focus more on psychological factors than on social factors. For example, some projects may provide individual counseling to children who have had traumatic experiences or provide training to key community members to identify, refer, or counsel children. These projects will most likely target children and caregivers who have been most severely impacted by crisis events and require a higher level of individualized attention than community-based interventions can provide.

2. Predominately Psychosocial: Some psychosocial projects are predominately or exclusively psychosocial in focus. The project is self-contained and not integrated into other projects with different foci health, food security, shelter) that may co-exist and are co-located. Examples include stand-alone recreation projects, art therapy, or various community-based interventions that promote positive cognitive, emotional, and educational development and functioning. Staff working in these psychosocial projects may have only minimal contact with staff working on other projects. Predominately psychosocial projects are likely to target their activities toward generally affected and at-risk populations, and provide screening and referral (to individualized mental health services or counseling programs) for those more severely affected by conflict or violence.

3. Integrated/Holistic: In some cases psychosocial interventions are integrated into a holistic and total response to the needs of a community. In this case, the “psychosocial” elements may not be as visible. For example, income generation or vocational training projects are not typically thought to be psychosocial. Yet, addressing the economic livelihood of families is fundamental to psychosocial health both in terms of reducing the daily stress of how a family will feed itself, and in terms of providing a pathway to stability and hope for the future. Similarily, such an intervention may have an educational component that supports cognitive development and at the same time fosters good peer relationships and social skills. An income generation project or vocational training project may be a conduit for improved self-esteem and self-worth and the establishment of peer friendships. The position here is that projects that are based on such a holistic approach are to be preferred since they maximize a mutually reinforcing effect when responding to different aspects of child development simultaneously. These projects are most likely to focus on those in the at-risk or generally affected group.

It is useful to organize projects into six broad areas that encompass the diverse social and psychological needs of children during and after a crisis: The Primacy of Family, Education, Engaging Activities, Economic Security, Community Connections, and Reconciliation and Restoration of Justice.

20 Essential Program/Project Evaluation Books

If you have an interest in having several books on program/project evaluation as a reference, you may want to include those book that are most often purchased together.

Using the online book store, Amazon.com, I began my search to determine which books might be best to have in an evaluation library by starting with the most known evaluation book, Evaluation: A Systematic Approach, by Peter Rossi. Amazon.com also provides a list of other books purchased by those people who purchased a certain book. Starting with Rossi's book, which is generally recognized as an evaluation primer, I made a list of all evaluation books people purchased with Rossi's Evaluation: A Systematic Approach. Then I looked-up each one of those books (as of October 2009) and made the same list of books purchased with each on of them until I had reached almost 550 citations for 110 books on evaluation. (Of course, there is a degree of selection bias in this method since this list will show primarily American readership and only for those who purchase books online.)

The map below shows the purchasing patterns for the 110 books on evaluation purchased in the US. The cluster in the middle of the map is those evaluation books most often purchased together.Taking those books that received the most citations of being purchased together I created the top 10 list, and the next 10 list, which are then the 20 most essential books on program/project evaluation.

Taking those books that received the most citations of being purchased together I created the top 10 list, and the next 10 list, which are then the 20 most essential books on program/project evaluation.

Top 10 Program/Project Evaluation Books Purchased Together

1. Research Design: Qualitative, Quantitative, and Mixed Methods Approaches, by John W. Creswell.
2. Qualitative Research & Evaluation Methods, by Michael Quinn Patton (Editor).
3. Qualitative Inquiry and Research Design: Choosing among Five Approaches, by John W. Creswell.
4. Evaluation: A Systematic Approach, Peter H. Rossi, Mark W. Lipsey, and Howard E. Freeman.
5. Handbook of Practical Program Evaluation, Joseph S. Wholey, Harry P. Hatry, Kathryn E. Newcomer.
6. Program Evaluation: Alternative Approaches and Practical Guidelines, Jody L Fitzpatrick, James R Sanders, Blaine R Worthen.
7. Utilization-Focused Evaluation, Michael Quinn Patton.
8. Logic Modeling Methods in Program Evaluation, Joy A. Frechtling.
9. Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation, E. Jane Davidson.
10. Evaluation Theory, Models, and Applications, Daniel L. Stufflebeam, Anthony J. Shinkfield.

Next 10

11. Program Evaluation and Performance Measurement: An Introduction to Practice, James C. McDavid and Laura R. L. Hawthorn.
12. RealWorld Evaluation: Working Under Budget, Time, Data, and Political Constraints, Michael J. Bamberger, Jim Rugh, and Linda Mabry.
13. The Program Evaluation Standards: How to Assess Evaluations of Educational Programs, James R. Sanders.
14. Evaluation, Carol H. Weiss.
15. Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, Huey Tsyh Chen
16. Case Study Research: Design and Methods, Robert K. Yin
17. Designing and Conducting Mixed Methods Research, John W. Creswell and Dr. Vicki L. Plano Clark.
18. Experimental and Quasi-Experimental Designs for Generalized Causal Inference, William R. Shadish, Thomas D. Cook, and Donald T. Campbell.
19. The Research Methods Knowledge Base, William Trochim and James P Donnelly.
20. Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, Juliet Corbin and Anselm C. Strauss.

Other books I found of interest:

• What Counts as Credible Evidence in Applied Research and Evaluation Practice?, Stewart I. Donaldson, Christina A. Christie, and Dr. Melvin (Mel) M. Mark.
• Program Theory-Driven Evaluation Science: Strategies and Applications, Stewart I. Donaldson
• Counterfactuals and Causal Inference: Methods and Principles for Social Research, Stephen L. Morgan and Christopher Winship.
• How to Measure Anything: Finding the Value of "Intangibles" in Business, Douglas W. Hubbard
• Quasi-Experimentation: Design and Analysis Issues for Field Settings, Thomas D. Cook and Donald T. Campbell.
• Damned Lies and Statistics: Untangling Numbers from the Media, Politicians, and Activists, Joel Best.

Use of Mobile Phone SMS in Program Delivery and Monitoring

Increasingly, mobile phones are becoming cheaper and thus more widespread around the world and owned and used by the poor. SMS is most likely the most widely used electronic means of communication in the world, with an estimated 2.5 billion active users.

Percentage of people covered by mobile services, by country in the MEE Region:

                         % population covered     Mobile phone service

Country                by mobile service          per 100 people

Armenia:                        88%                              63

Azerbaijan                     94%                               53

Egypt                            98%                               40

Georgia                         96%                               59

Iraq                               72%                              n/a

Jordan                           99%                              23

Kazakhstan                     94%                              80

Kyrgyzstan                     90%                               41

Tajikistan                        4%                                1

West Band/Gaza             95%                              n/a

Yemen                           68%                               3

Why use mobile phone SMS in development?

1. Very cost effective for communication (data and information).

2. Fast because SMS is sent in real time and immediately.

3. High exposure in that, generally, 90% of all SMS messages are opened.

4. Potential for large reach depending on how many use mobile phones.

5. Personal in the SMS can be personalized.

6. Interactive in that receives can respond.

If you would like to read more, below are links to sites that provide more information about mobile phones in development work.

Livelihoods and employment (Souktel http://www.souktel.org/ and http://mobileactive.org/improving-livelihoods-and-incomes-mobile-phones)

Farming advice (http://mobileactive.org/farming-advice-cell-phone)

Health (RapidSMS http://www.rapidsms.org/about/about/)

Nutrition surveillance (http://mobileactive.org/case-studies/nutritional-surveillance-rapidsms-malawi)

Pre-natal child care (http://mobileactive.org/prenatal-care-through-sms)

Emergency Preparedness & Emergency Warning (FloodSMS http://www.netsquared.org/projects/floodsms-–-early-detection-and-warning-catastrophic-flooding-sms)

Community mobilization (FrontlineSMS http://www.frontlinesms.com/)

Education (http://mobileactive.org/bridgeit-mobiles-classroom)

Literacy & community empowerment (http://mobileactive.org/how-tostan-using-mobiles-literacy-and-community-empowerment)

Cash transfer (http://mobileactive.org/concerns-cash-aid-mobile-kenya-evaluation

Why does SC use a Results Framework?

Often, country offices ask me why Save the Children recommends a Results Framework rather than some other type of program/project design tool, such as Logical Framework. Several employees of Save the Children (see attached article on the right titled, “A Results Framework Services Both Program Design & Delivery Science” under the Documents section).

Some of the reasons these authors cite include:

1. The entire program/project logic and “theory of change” can be visually grasped without extensive reading.

2. Different disciples or technical specialists (health, food security, livelihoods, education) can use the same
    basic model.

3. The ability to clarify assumptions as well as state hypotheses.

4. Facilitates in the design programs and projects.

5. Helps in the evaluation designs.

6. Informs action research

The Results Framework has the following components:
Goal- a) States the long-term end status that is to be achieved, b) Usually expensive to measure since it requires large population-based surveys.
Strategic Objective (SO) – a) Is the most ambitious result that programs can reasonably effect and for which implementing agencies are willing to be held accountable.
Intermediate Results (IRs) – These are essential steps toward achieving the SO. Save the Children recommends the use of the following 4 IRs, since SC’s programming is based on behavior change:

   IR-1: Availability & Access (as service must be available as well as spatially and economically accessible)

   IR-2: Quality (services meet technical as well as client perceived standards)

   IR-3: Demand (knowledge, skills, attitudes, or beliefs that hinder or promote service usage)

   IR-4: Enabling Environment (facilitates both the supply and demand side of services)
IR Strategies – specific steps to achieve the Intermediate Results
IR Activities – specific program/project activities related to each IR strategy.

Results Framework (Health Example)

What are some of the limitations of the Results Framework?

1. IR2, Quality, has many dimensions, such as technical (i.e., meeting national or international standards) and perceived (client’s perception of quality services). However, this model combines both types into one box even though these are separate dimensions.

2. The Enabling Environment (IR4) is often highly related to achieving IR3, a change in demand via more informed clientele.

3. The Results Framework, unlike the Logical Framework, omits external environmental factors (apart from those in IR4) that can ease or constrain achieving the results and that are beyond the reach of programmers.

4. Finally, the framework lacks the operational details (such as those found in Logical Frameworks) that managers and some donors need; however, standard detailed implementation and monitoring plans that are based on the framework provide these.

Overall, the Results Framework is a simplistic way to illustrate the relationships between higher level Goals all the way down to activities for small as well as large, complex programs/projects regardless of the sector. But, as the authors conclude, that simplicity has its disadvantages.

US White House announcement: Program Evaluations

In an 7 October 2009 memorandum, issued by the Executive Office of the US President, the issue of program evaluation is emphasized (http://www.whitehouse.gov/omb/assets/memoranda_2010/m10-01.pdf). In short, an inter-agency working group of evaluation experts under the Performance Improvement Council established will be revived. The purpose of the working group will be: (a) to help build agency evaluation capacity and create effective evaluation networks that draw on the best expertise inside and outside the Federal government; (b) to share best practices from agencies with strong, independent evaluation offices; (c) to make research expertise available to agencies that need assistance in selecting appropriate research designs in different contexts; (d) to devise strategies for using data and evaluation to drive continuous improvement in program policy and practice; and (e) to develop government-wide guidance on program evaluation practices across the Federal government while allowing agencies flexibility to adopt practices suited to their specific needs. A key goal of the working group will be to help agencies determine the most rigorous study designs appropriate for different programs given their size, stage of development, and other factors.

The question to me is: Will this also apply to programs receiving US foreign aid via USAID during the Obama administration? Is so, when?

Reconstructing a baseline via retrospective pretest

A common approach to project evaluation is using a pre- and post-test among beneficiaries. Although this type of design is not very rigorous (I will write about why in a later post), occasionally there are training projects that get started so quickly that only at the end of the training do staff realize that they did not conduct a pretest and subsequently do not have baseline to compare a change in knowledge from the training course.

In an article by Debra Moore and Cynthia Tananis (American Journal of Evaluation, 2009) these authors discuss the issue of not only how to reconstruct a baseline but also the validity and reliability of data when doing so. This method is called a retrospective pretest design.

Now, the authors clearly state that this is a method best used with short-term, intensive training programs and may not be as reliable in other types of activities and interventions.

The authors mention that in a both a pretest and posttest design, or a retrospective pretest design, one of the primary concerns is something called response-shift bias. Response-shift bias occurs when a participant understands the concept being measured at the pretest differently than at the posttest. For example, youth asked on pretest to answer questions about empowerment (the concept) before the training begins may answer differently when asked the same questions about empowerment at the posttest when the training ends because after taking a training course they understand the concept of empowerment differently. Thus, the authors wanted to test if the degree of response-shift bias when a pretest and posttest was conducted for a training course and when a pretest was NOT done and a retrospective pretest was used.

The basic research question was: Do participant’s responses more accurately represent their level of knowledge/awareness at the beginning or after the training?

For example, before taking a training course on DME I many think I know a lot and would willing respond on pretest questionnaire high levels of knowledge and abilities in doing DME. Then after taking a DME course, and being exposed to more detailed and complex issues that I was not previously aware of, I may reassess that my level of knowledge and abilities were not as great as I thought. But, sadly it’s too late to change my pretest responses.

The authors conclude that pretest scores tend to overestimate a particular level of knowledge or ability(larger response-shift bias) than with a retrospective pretest. They also report that other studies have found that self-report retrospective pretest scores are more highly correlated with scores on objective pretest measures of skill development or knowledge than the self-report pretest scores.

The basic message: In projects that include short, intensive training courses, reconstructing a baseline through the use of a retrospective pretest conducted at the end of the training may provide more accurate results than a pretest at the beginning of the training course.

Youth Livelihood Developmental Index (YLDI): Measuring Youth Assets and Competencies

For my first blog, I thought I would review several tools that I have been involved with the Egypt Country Office in measuring outcomes for their youth livelihoods projects, and has been used in Yemen, as well as in the Africa and Asia regions.

The YLDI is a set of three tools: 1) the Developmental Assets Profile referred to as the DAP, 2) the Livelihoods Competencies Profile referred to as the LCP , and 3) the Tangible Assets Profile referred to as the TAP. Each of these tools is a standardized index to measure the level of assets, competencies and resources that youth have at a given point in time.

The DAP was developed by the Search Institute and contains 58 questions that can be completed either by the youth themselves or in groups. The scores are totaled and the levels of the 40 developmental assets are categorized as low, fair, good, or excellent. Profiled results portray the type and degree of developmental assets among the youth. The DAP produces quantitative scores for two domains (External and Internal Assets) and on eight asset categories (Support, Empowerment, Boundaries & Expectations, Constructive Use of Time, Commitment to Learning, Positive Values, Social Competencies, and Positive Identity) as well as four Context Area domains (Personal, Family, School, Community, Social).

The LCP was developed by Global Youth Livelihoods and contains 69 questions that can be completed individually by the youth or in a group. The LCP is designed to measure a youth’s self-assessment of the level to which s/he possess one or more of 17 basic competencies needed to generate or maintain an income and livelihood. These competencies are grouped into four domains: Human Capital, Social Capital, Financial Capital, and Physical Capital. The scores are totaled and the levels the youth possess of the four types of capital are categorized as low, fair, good, or excellent. Profiled results portray the type and degree of livelihood competencies among the youth.

The TAP, again developed by Global Youth Livelihoods, contains 32 questions that can be completed individually by the youth or in a group. The TAP is designed to measure a youth’s self-assessment of the level to which s/he possess or has access to 8 tangible assets that can generate or maintain an income and livelihood. The 8 tangible assets are grouped into two domains: Financial Capital and Physical Capital.

The YLDI is accompanied by a database template that allows for easy data entry and some basic data analysis and reports.

Due to the total number of questions for all three tools they are administered at separate times. These three tools that comprise the YLDI can be used for three general objectives:

• To assess the status (prevalence) of developmental and livelihood assets of youth in a given area to assist with new program/project design;
• To evaluate a program or project via a baseline and end-line survey to measure developmental asset and livelihood competency outcomes and or results;
• To establish youth profiles based on various characteristics so as to better tailor program/project activities and interventions.

To date, the YLDI has been used in Upper Egypt for a project evaluation (with Mona Moneer) and in Yemen (with Lucienne Mass) for a general assessment to develop youth livelihood programming.

I have attached the English versions of the DAP, LCP and TAP (in the Documents list to the right). HOWEVER, please do not translate or use because they are copyrighted materials. If you would like to use them in a project contact either me or Sita Conklin (MEE Livelihoods Advisor).

If you have a livelihoods project or are considering a livelihoods component in a future project, and are interested in learning more about the results of the use of the YLDI in Upper Egypt or Yemen, feel free to contact me.

In my next post I will discuss how to reconstruct a baseline for a training program at the end of a project!