Chapter 2: Planning an Evaluation

Category

Agency or Organization

URL

Description

U.S. Federal Government

Agency for Healthcare Research and Quality

www.ahrq.gov 

This is an extensive site with a wide range of relevant resources, ranging from information and resources regarding COVID-19, to tools for improving patient care, to reports of “Impact Case Studies” that can be sorted or filtered by region, date, products, and more.

Centers for Disease Control

www.cdc.gov/eval

This is another extensive site with many relevant elements. For example, just in the “Evaluation Documents”, found by clicking on “CDC Evaluation Resources”, the following pages are noted: Evaluation Development Tools, Logic Models, Evaluability Assessments, Indicators & Performance Measures, Evaluation Reporting, Economic Evaluations and Tools, Health Impact Assessments, Databases and Data Resources, Health Communications Tools, Strategies, Interventions and Best Practices, Success Stories, Podcasts and Webinars, Technical Assistance for States, Program Management, Other Evaluation Tools, and Evaluator Self-Assessment. “Logic Models” provides instruction on using Logic Models as well as examples of models used in evaluations.

Department of Education, Office of Planning, Evaluation, and Policy Development

www2.ed.gov/about/offices/list/opepd/ppss/reports.html 

The focus of this site is on education, as could be expected. One particularly relevant point for evaluators is the “What Works Clearinghouse” at  https://ies.ed.gov/ncee/wwc/   that represents efforts to systematically use the results of evaluations to make known what approaches have the best evidence for success.

Department of Health and Human Services, Administration for Children & Families, Office of Planning, Research, and Evaluation

www.acf.hhs.gov/opre

This comprehensive site focuses on children and families. Some of the elements to note especially are: in the “Research and Data Topics” tab – Methods and Tools, and Head Start. The Conferences and Events tab provides current information on meetings and other gatherings. You can search for “Resource Library” which has a large number of reports and other documents that can be searched or sorted by a number of elements such as topic.

Office of Justice Programs, Bureau of Justice Assistance, Center for Research Partnerships and Program Evaluation

www.bja.gov/programs/crppe/index.html

An important source of information and resources related to the criminal justice system. The tabs on Logic Models, Performance Measures (guidance on how to choose good ones), and the Featured Resources are particularly helpful. As noted in chapter 6, there are useful links to guidance for finding relevant data online as well as to actual sources of relevant data.

State Governments

State of California Department of Justice

https://oag.ca.gov/recidivism/program-evaluation

This is an example of a site with a specific evaluation focus – evaluating recidivism, although the many pages provide information and links for many related topics.

Minnesota Office of the Legislative Auditor

www.auditor.leg.state.mn.us/ped/pedintro.htm

This site provides a few brief comments on evaluations. But for those interested in examples, there are hundreds of reports, dating back to 1977, many with both summary and full reports.

Foundations

Kellogg Foundation

www.wkkf.org  

The W.K. Kellogg Foundation has a long history of supporting Program Evaluation, and the site provides a number of resources, such as The Step-by-Step Guide to Evaluation, published in 2017, the Logic Model Development Guide, published in 2004, the Evaluation Handbook, published in 1998, and series of three guides focused on “Doing Evaluation in the Service of Racial Equity.”

Robert Wood Johnson Foundation

https://www.rwjf.org/en/insights/collections/evaluations-and-assessments.html

The Robert Wood Johnson Foundation also has a strong tradition of supporting evaluations. On the site, under the “How we work” tab, select “Research, Evaluation, and Learning.” A number of evaluation reports are available for download. In addition, there is a “Resources for Researchers and Evaluators” section.

Universities

American Evaluation Association’s list of University Programs in Evaluation

https://www.eval.org/Education-Programs/University-Programs

Although this list is just one of the many resources at the American Evaluation Association website, it provides the specific value of a current list of graduate degree programs or certification in evaluation with details such as the level of the program ( masters or doctorate) and hyperlinks when available.

Western Michigan Evaluation Center

www.wmich.edu/evalctr

This site is a good example of a University-based Evaluation Center providing various resources, such as general information on evaluations, helpful tools such as checklists, and webinars on a range of evaluation topics. New webinars are presented every 3 months, and past webinars are listed with links to video recordings of them.

Organizations

American Evaluation Association

www.eval.org

The American Evaluation Association is obviously an extremely important organization for evaluators to know and to be connected to. There is a wide range of information on the site for anyone, such as tips, webinars, blogs, details about conferences, and career information. There is even more available for members of the Association, especially on-line access to the main evaluation journals.

Canadian Evaluation Society

https://evaluationcanada.ca/

Rather than listing all international evaluation websites, the Canadian Evaluation Society site provides an important reminder of the broad range of organizations and resources on a global basis. Among the many valuable points across the site, the specific information about Canada’s “Credentialed Evaluator” process and Evaluator Competencies provides an important example and context.

Journal

Journal of Multidisciplinary Evaluation

http://journals.sfu.ca/jmde/index.php/jmde_1/index

Although many journals are available on-line for a fee or as a part of membership in an organization, the Journal of Multidisciplinary Evaluation is available free, and provides an important supplement to other information.

Model

Features

Strengths

Weaknesses

Notes

Citation

Accreditation and Certification

Accreditation and certification models are specific to training and service delivery programs that are connected with professional organizations that oversee and endorse them. There is a common structure of an initial self-study, a kind of preliminary internal evaluation, followed by review of the material and a visit on-site by representatives of the organization.

These evaluations are generally consistent across programs, and there is a clear focus to inform the public generally and clients of the programs specifically about the quality of the program. Further, those involved in the evaluation are generally particularly well-informed about such programs, even experts in the fields. Recently, there has been growing emphasis on the outcomes of the programs, which strengthens the previous attention to processes.

As these evaluations are driven by the professional organization rather than independent evaluators or agencies, there is the risk that following organizations’ political agendas can have undue influence or that connections between the site visitors and the program staff could compromise their objectivity. Further, the initial self-studies tend to be greater opportunities for programs to promote their successes rather than to weigh strengths and weaknesses fairly.

Stufflebeam & Coryn, 2014

Black Box

This approach is often used as a kind of caricature for the purpose of contrast – a very simple approach to evaluation that concentrates only on results while ignoring virtually everything else about the program, as if those details were enclosed in a black box that remains unopened.

The simplicity of this approach really is its only strength – that by concentrating on the outcomes without being distracted by any other issue, the clarity of describing those findings appeals to those who are looking for a direct answer to a direct question – What does the program do?

There are multiple weaknesses – this approach does not consider needs, implementation issues, stakeholders, side effects, improvement, or the mechanisms of change. For these reasons, it is often presented as a kind of cautionary example – that although some people may think that just asking about outcomes is a good approach, it is not at all a comprehensive way to evaluate a program.

Posavac & Carey, 2007

Capacity Building

Like some of the others, Capacity Building is defined more by the meta-goals of the evaluation than by the specific methods or tools used. It aims to work with program stakeholders in such a way as to teach enough of them sufficient evaluation knowledge and skills to be able to continue evaluating the program on a regular basis.

This approach seems particularly valuable to many evaluators in the sense that persuading program stakeholders to learn evaluation methods and skills so they can continue the work the evaluators began feels deeply satisfying. And to the extent that the culture and practices of the program change, it can be transformative. Further, the approach of teaching others to take over for initial efforts generally inspires a more collaborative, respectful approach than thinking others cannot understand.

The risk of this approach is especially that evaluators will compromise their objectivity because of their desire to emphasize stakeholders’ learning. Likewise, given limited resources of time, effort, and finances, situations can develop so evaluators must choose whether to concentrate on the evaluation or on teaching. Some evaluators find that the promise inherent in this approach is not realized as often as they wish.

Consumer Oriented Studies

The emphasis of this approach is providing those who benefit from, or consume, the products of a program with clear information regarding the comparative value of what is delivered – is it better than, the same as, or worse than the other options available?

Although many evaluations indicate that they include the perspective of clients or others served by programs, this approach lives up to that standard. Following Michael Scriven’s emphasis on value, this model’s integration of various elements about programs provides insight that is focused on what programs are worth to those who use them.

As important as a focus on consumers is, it risks having evaluators fail to connect with other important stakeholders in a variety of ways. The attention to providing clients with comparative summaries risks putting program staff on the defensive, believing that the evaluators are only looking to criticize them. This potential resistance can undermine the process of gathering accurate information about the program.

Stufflebeam & Coryn, 2014

Critical Evaluation Capital

It is described as a tool, not a separate approach, designed to be used with other models that consider the evaluator as an “engaged stakeholder”, rather than an objective observer. Still, it is similar to other models in addressing assumptions about the evaluand, the evaluators, and perspectives on power and privilege.

This tool explicitly incorporates Critical Race Theory and Community Cultural Wealth into the evaluation process, supporting evaluators who recognize that issues of power and privilege are deeply embedded in programs. Among other aspects, it supports questioning the assumptions underlying the existing goals and expected outcomes of the program, which allows considering the voices of those with less power in the program.

As may be obvious, this way of understanding programs and important processes within them is a good match for those who share its perspective on the way that power and privilege are typically un-examined components of programs. Evaluators who do not share this perspective will generally not be able to make use of this tool, and in fact, might mis-lead the program stakeholders by claiming to incorporate these insights when they do not.

Ginsberg, 2022

Culturally Responsive

This approach represents a comprehensive commitment to addressing cultural differences that are reflected in aspects of programs, whether because of patterns with clients served, with staff who work in the program, or broader elements like the setting relevant to the program. Some important features are a focus on strengths, promoting social justice, and taking a holistic view.

One important strength of this approach is its correction to the range of practices in past evaluations from being less than fully aware of cultural issues at the best to drawing erroneous conclusions due to a failure to understand important cultural differences and the varied meanings that different cultural groups make of the same events and perceptions. Growing awareness of how often cultural elements are not sufficiently understood means this approach is especially valuable currently.

Although it is hard to argue that taking cultural differences seriously has a down-side, it is possible for evaluators to incorporate a limited number of elements of this approach only to discover that such partial models may have little to none of the benefits of a fully Culturally Responsible Evaluation. Among other things, using superficial or stereotyped information about cultural groups can be worse than including no information at all.

Thomas 2004

Decision and Accountability Oriented Studies

Intended to be a comprehensive assessment of the full range of relevant issues, especially the challenging aspect of a program’s value, these approaches are exemplified in Stufflebeam’s Context, Input, Process, and Product (CIPP) model – a structured method guiding careful and thorough evaluation emphasizing value and worth that can be used for formative or summative purposes.

The thorough coverage of all relevant aspects of program evaluation provides important balance – neither neglecting nor over-emphasizing any element. In addition, as this approach is broadly grounded in current research, insights from a wide range of perspectives provide a solid foundation for addressing the elements in a fair way. The details distilled from careful use of many evaluators’ experiences provide especially useful guidance to newer evaluators.

This balanced approach avoids the weakness of models that emphasize a narrow range of elements. The actual weakness is more a reflection of how the model is used. Evaluators can choose this model but emphasize collaboration with program stakeholders so much they lose the objectivity to be clear regarding problems in the program, and the strong theme of improvement means that it is an easier match with formative evaluations rather than summative. As Stufflebeam and Coryn acknowledge, there is the (inaccurate) perception that the approach focuses too much on agency leaders.

Stufflebeam & Coryn, 2014

Deliberative Democratic

This approach involves a structured process of stakeholder involvement in the evaluation with equal representation built into the details. There is an important distinction in that stakeholders are provided with equal voice, but the evaluator retains the final word regarding the meaning of what has been learned from those voices in the ultimate report.

Like some other models, this structure illustrates a formal commitment to values that most evaluators share and even applaud. So ensuring that the evaluation will listen to voices from all stakeholders equally has an intuitive appeal, and the obvious commitment of evaluators to all stakeholders has the important benefit of generally leading to more complete co-operation and engagement by those stakeholders, especially those who might commonly be left out of discussions.

Although the theory of this model is appealing, it turns out to be overly idealistic in practice, and thus is not a practical approach for some, perhaps even many, evaluations. Among other things, providing equal voice to all stakeholders in some evaluations would prove to be much too expensive in terms of time and other resources. Further, to the degree that these realities mean these evaluations commonly need to be scaled back at some point, disappointment can prove more problematic than not having hopes for equality in the first place.

Stufflebeam & Coryn, 2014

Developmental

Developmental evaluation is targeted at innovative programs that are open to collaboration with evaluators to modify the program plan based on the evaluation. Following Patton’s Utilization Evaluation, it focuses on the use of the program by asking those who use it for insights, and it tries to avoid linear thinking by takings system approaches seriously.

Although developmental evaluation is not a perfect match with all programs, it is an excellent fit with those programs that share the central values of innovation, adapting on the basis of evidence, and collaboration between program staff and evaluators. The pragmatic elements of Utilization Evaluation also add to the value of this approach.

As noted in the strengths, this approach is not for all programs nor for all evaluators, as it requires a rather clear commitment to particular methods. Likewise, as an explicitly formative model, it is not a good match when summative evaluations are required.

Patton, 2011

Empowerment

Similar to Capacity Building, empowerment approaches are highly collaborative, aiming to teach knowledge and skills to empower current program stakeholders to take on evaluation roles. In addition, while both models aim to train internal evaluators, empowerment approaches incorporate social justice elements.

Like other approaches that engage stakeholders in a genuinely collaborative way, empowerment approaches embody some of the best evaluation values such as building a solid foundation for improvement. When stakeholders begin to take on the role of evaluators, they typically become more open to comprehensive insights, both positive and negative, and are more prepared to make changes because they understand the basis for them.

Along with some other approaches, this one risks compromising the objectivity and contribution of external evaluators if they over-emphasize the importance of stakeholders’ learning above the current evaluation products. It is important to realize that such problems are seldom recognized at the time – the decisions typically seem justified by the circumstances.

Fetterman, Kaftarian, and Wandersman, 2015

Expert-Opinion, also called Connoisseurship and Criticism

When programs involve matters that are best assessed by those with expertise in the field, an essential part of an evaluation will be the use of experts rather than other measures such as self-report. Different programs may have different forms of such assessment, such as how good a given outcome is or the quality of an intervention.

As noted, this approach applies to a subset of programs – those that require experts to determine one or more element of the program. In such cases, those who are not familiar with the field are likely to make important mistakes of judgment, so it is essential to use experts. A common example is that there is no checklist for ordinary people to use to determine the value of art – instead, those with training and experience in the field – experts – are needed.

The suggestion that some matters require specialized training and experience to support high-level evaluations is not an unreasonable one. The problem, of course, is that determining which matters need experts and which do not is not a simple one, and who makes such determination is likewise a crucially important point. Many evaluators would agree that there are probably far fewer needs for expert-opinions than some would have us believe.

See Note 1

Posavac & Carey, 2007; Stufflebeam & Coryn, 2014

Fiscal Evaluation, also called Cost Studies

These are a set of related approaches that focus particularly on the financial elements of programs - providing conclusions, for example, about how much programs cost per unit of outcome, such as one client no longer using a given substance. Variants may assign specific values to outcomes so the conclusion reports the return on investment, such as two dollars of benefit for every dollar of cost.

Not only is the focus an intuitively meaningful one that also meets substantial demands of program stakeholders, such approaches build on an impressive body of work detailing how to determine the various financial analyses. Among other things, as virtually all agencies have important financial constraints, comparative evaluations can provide specific information about relatively efficiency, such as noting that one program leads to twice as much value for the same cost as another.

Although the idea of evaluating costs in a way that permits comparisons and other secondary analyses is an attractive and valuable contribution, there is an important danger that the clarify of exact numbers do not just overlook other important elements, but overstate what has actually been found. Saying that there are two dollars of benefit for every dollar of cost could be based on estimates that range from five cents to five dollars with a median of two dollars, which would lead to very different decisions than if the range were from $1.95 to $2.05. And dollar amounts can oversimplify effects in people’s lives.

See Note 1

Posavac & Carey, 2007; Stufflebeam & Coryn, 2014

Goal-Free

This distinctive approach focuses on evaluators having a “blank slate” as they begin the evaluation – deliberately not learning the program’s intentions for the purpose of avoiding bias. It exemplifies a “bottom up” method of letting insights and conclusions emerge from the data and observations.

In theory, expectations based on what programs say they intend to do can make it hard for evaluators to perceive subtle evidence for other points in any aspect of a program, so not contaminating the evaluation with those expectations allows the actual data to speak for themselves. This approach would be most useful if there are reasons to believe that assumptions have kept previous observers from seeing important processes and outcomes.

Assuming that it is possible to approach any evaluation without expectations, an important weakness is that allowing all information to organize itself is a particularly costly method in terms of time, effort, and other resources. Further, it seems virtually impossible for evaluators to have no expectations for any program, as if they could really be unaware of the population, interventions, goals, and other aspects.

Posavac & Carey, 2007

Improvement-Focused

In many ways, this is primary an attitude held by evaluators that the program probably has value and is making an important contribution, so problems or weaknesses are most appropriately addressed in ways that detail how the program can fix problems or do even better in the future. This model is commonly added to other models, not replacing them.

With regard to the products of evaluations, such as reports, this approach is much more likely to provide information and insights that stakeholders will use because of the positive nature of the focus. With regard to the process, evaluators who have an attitude of improvement are much more likely to engage stakeholders in productive ways and to encourage more genuine co-operation. Stakeholders who feel less threatened are more likely to be more open about the full range of program matters.

The risk of a focus on improvement can be to understate negative findings and to downplay concerns more broadly. In a different light, although many evaluations see this model as a typical addition to other methods, novice evaluators may convince themselves that they follow an Improvement Focused Model, and avoid committing to a primary approach as a result.

Posavac & Carey, 2007

Industrial Inspection

Similar to the Black Box, this approach is sometimes described for the purpose of providing a contrast with more complete and more valuable evaluations. This methods examines the final products of a program and details what needs to happen with any that do not meet standards.

Similar to the Black Box, the main strength of this approach is its fit to the expectations of many people. There are those who assume that programs are already working well enough that the infrequent problems can be fixed after the fact. To the degree that these expectations are correct, this is an efficient model because resources are not wasted looking for complications. Of course, most evaluators believe such expectations are egregiously wrong.

The weaknesses are similar to those of the Black Box – its simplicity means that important aspects of the breadth of comprehensive program evaluations are simply missing – insufficient attention is paid to needs, implementation, stakeholders, side effects, improvement or the mechanisms of change. Further, although the method might make sense if problems were so rare that few corrections are needed, that is not the case with most programs.

Posavac & Carey, 2007

Meta-Analysis

Like meta-analyses in the general field, meta-analysis as an approach to evaluation takes the results of many individual studies and draws conclusions based on the multiple findings.

Combining results as a meta-analysis provides greater statistical power by effectively combining participants from multiple studies. Especially as evaluations are often under-powered and thus have increased likelihood of false negative findings, this approach provides a way to detect mild to moderate effects while also following traditional expectations for evidence.

Although the theory of meta-analysis is sound, the actual programs that are often included typically include enough variation in details to make examining a main outcome at least questionable. Much more, the focus on outcomes means that substantial areas of evaluations such as needs, implementation, stakeholders, side effects, improvement focus, and probably nuances are downplayed if not completely ignored.

See Note 1

Stufflebeam & Coryn, 2014

Naturalistic / Qualitative, also called Constructivist

Although some evaluations that are identified in the category may simply use qualitative methods to gather information, thorough-going Naturalistic evaluations follow a philosophical view that rich data collected in more collaborative ways is vastly superior to what is seen as a narrow focus with quantitative methods.

The commitment to comprehensive information and collaborative methods have substantial potential to make these evaluations rich in terms of insights and conducted in a manner that brings reports to an agency prepared to listen to the full range of insights and use the information to improve the agency. Many participants find that having someone interested not only in their information but the way they talk about their information helps them be especially transparent about program matters.

This approach can stretch limits of practicality as it tends to take more time and resources than quantitative methods. It also tends to be somewhat idealistic in assumptions that stakeholders value improvements more than their own approaches and individual benefits. So even when stakeholders may initially agree to this method, the actual details can strain the initial commitment.

Posavac & Carey, 2007

Objectives Based

The evaluation considers the program’s existing objectives as the standards and focuses on the degree to which the program meets these targets.

This approach is a common one, so there are many examples of its use. It is generally perceived to be especially fair – programs are evaluated based on what they themselves intended to achieve.

The objectives that a program chooses may not be appropriate or sufficiently rigorous, so the evaluation may conclude there are good outcomes when the program does not achieve good enough results.

Posavac & Carey, 2007

Ontologically Integrative Evaluation

This approach explicitly addresses the problem of worldviews based in western values and assumptions, and provides a framework for evaluators to explore their own ontological assumptions and to learn about the assumptions of the evaluand.

This model is a sophisticated and comprehensive one, so it is an especially good match for evaluators who are already committed to the challenging work of de-centering their own assumptions as they seek to learn the assumptions of the evaluand and to collaborate meaningfully with those who hold those different views.

Newer evaluators and those who are still learning about cultural humility and culturally responsive approaches to evaluation may do best by considering what the Ontologically Integrative Evaluation might contribute to their work, but incorporate this model in their approach gradually rather than all at once.

Billman, 2023

Outcome as Value-Added Assessment

This approach, commonly used with yearly standardized assessment of students’ achievement at multiple schools, concentrates on the gains made, permitting comparison of average gains across schools.

The structure of systematically collecting scores for all students and in multiple schools supports many meaningful analyses, especially regarding how high- and low-achieving students change over time, which is commonly not reported in other approaches.

The reliance on a single outcome measure over-emphasizes that element, ignoring other potentially important aspects. Although the comprehensive collection of data permits comparisons across settings, the meaningful factors that might be associated with differences are not necessarily assessed or considered. Stufflebeam and Coryn also note that this approach is viewed with substantial suspicion by a wide range of educators.

See Note 1

Stufflebeam & Coryn, 2014

Social Science Research

The approaches in this class share the common feature of using rigorous Research Design methods to test hypotheses about causality, providing strong evidence about the degree to which the program caused the observed results.

These approaches provide substantial confidence in findings with regard to the causes of outcomes, which is a highly valued point, given the increasing focus on evidence-based practice. It is important to note that such levels of confidence require a number of components such as sufficient statistical power as well as use of reliable and valid instruments and procedures, but not all “Research” approaches have all those components.

The weaknesses of these approaches are their generally poor fit with real programs. Important elements of experiments such as random assignment to conditions are at the very least difficult to apply to many programs, and at times appear clearly unethical. The focus on evidence for causal relations typically also distracts from the comprehensive evaluation of many elements of programs such as challenges to implementation and learning how programs can be improved.

Posavac & Carey, 2007

Success Case

The evaluation focuses on a generally small number of cases that illustrate the best current outcomes of the program, contrasting them with the least successful cases. The goal is to learn why these examples were so successful.

This approach is especially valuable in formative evaluations, as it concentrates efforts on learning what is going well or best, so those elements can be emphasized or enhanced.

Focusing on what can go right does not provide a full evaluation of the program, especially for a summative evaluation. A few successes generally do not make up for a large number of failures, so an indication of the average or typical result is a missing element.

Posavac & Carey, 2007

Systems Oriented

A systems oriented model is an ambitious attempt to view programs in a way that recognizes complex, multi-directional relationships. It also pays attention to the boundaries between elements and to multiple perspectives.

Systems oriented approaches have the potential to avoid the simplistic thinking about single causes and single effects that are common in many traditional research practices even when the research theories are less simplistic. The better fit of thinking about the evaluation to the complex world of programs may also appeal to stakeholders who deal with real-world complications.

A common weakness is that evaluators may say they approach matters in a system-orientation, and they may hold that conviction as a philosophical position, but the lack of clear guidance on how exactly to incorporate a systems perspective in an evaluation means that the promise of the model is often unrealized.

Williams & Imam, 2007

Theory Based

Theory-based approaches to evaluation examine the extent to which the explicit theory used by a program is supported by the evidence found.  

As theories usually specify particular mechanisms of change, these approaches have the advantage of clear expectations regarding those mechanisms, rather than either using measures that the agency has chosen for any of multiple reasons or assessing those elements most commonly used in similar programs.

Although movement toward evidence-based practice should mean that more programs are committed to particular theories, in fact, many programs use methods related to various theories, and individual staff may add further variation. Many programs also say they follow a given theory, but their practices are not that specific or consistent. So many programs are a poor match to the tight expectations of a theory-based model.

Posavac & Carey, 2007

Transformative

This approach involves a thorough commitment to principles and practices of social justice, cultural competence, and active support of marginalized groups.

Transformative evaluations are consistent with many evaluators’ values and ideals, especially in addressing perspectives and insights of groups that are generally not included fully in evaluations.

Substantial numbers of stakeholders are generally less sympathetic to the values of transformation, although these perspectives may often be expressed as concerns that others will not take the evaluation as seriously as a more traditional approach.

Mertens, 2009

Utilization Focused

Utilization focused approaches strongly emphasize the actual use of the evaluation by insisting that those who use the program are centrally involved in the evaluation process, especially by learning from them how the evaluation is most likely to be incorporated into future program plans.

The pragmatic nature of this evaluation, including the fact that select groups of stakeholders are asked to commit to collaboration with the evaluators, means that the common problem in other evaluations of lofty ideals not connected with actual practice is largely avoided. In particular, with the explicit value of use throughout the process, another common problem of reports sitting on shelves, un-used, is also avoided.

Although getting commitment from some stakeholders to be involved throughout the evaluation process is better than the typical situation of hoping for involvement only to be disappointed, this approach risks first that the limited group of stakeholders may not be a good representation of the entire program and second that as stakeholders change over time, the value of the evaluation will also fade.

Patton, 2008