Theory, Measurement, and Analysis
Implementing a Theory of Change Evaluation in the Cleveland Community-Building Initiative:
A Case Study
Sharon Milligan, Claudia Coulton, Peter York, and Ronald Register
Comprehensive community initiatives have proven difficult to evaluate because they do not lend themselves to traditional experimental methods. Nevertheless, many audiences are interested in the effectiveness of these efforts in actually creating change and improving the lives of residents. The theory of change approach to evaluation is intended to respond to these challenges.
In a theory of change approach, evaluators, program designers and staff, and other stakeholders work together to make explicit the important pathways of change they expect to follow. Further, they specify key steps along those pathways so those steps can be measured. The steps include both initiative strategies and the anticipated short- and long-term outcomes of the initiative. The evaluation determines the degree to which the change process is unfolding as expected and links strategies to outcomes.
Although this approach does not yield the type of impact estimate that can come from an experimental design, it does have several advantages. First, it can be applied to whole community interventions, within which untreated control groups are not possible. Second, it makes explicit many assumptions about the ingredients of community and system change and how those are expected to improve conditions for residents and their local institutions. Third, by tracking progress along the steps of the change process, it can provide corrective feedback that distinguishes failures of theory from failures in implementation. If the initiative’s programs and activities are shown to lead to expected outcomes over time, the evaluation begins to build a case for the effectiveness of the initiative. Even without a control group to provide a counter-factual, the order of occurrence of the outcomes and the resemblance of those outcomes to those predicted can support inferences about effectiveness.
The Cleveland Community-Building Initiative (CCBI), in collaboration with the Center on Urban Poverty and Social Change, has begun the process of implementing a theory of change approach to evaluation. CCBI and the evaluators have participated in the work of the evaluation committee of the Aspen Institute’s Roundtable on Comprehensive Community Initiatives for Children and Families and are one of the first teams to operationalize concepts from the theory of change approach.
Because there is little extant experience with the theory of change approach, this paper describes how the evaluators, staff, board, and community stakeholders launched the evaluation and what they learned from the experience. Specifically, the processes of involving stakeholders in the evaluation, delineating the elements of the theory, and choosing and defining the concepts to be studied are explicated. Before depicting these experiences, however, we present background on CCBI and the evaluation.
The Cleveland Community-Building Initiative
The Cleveland Community-Building Initiative (CCBI) builds on the work of the Cleveland and Rockefeller foundations to reverse growing and persistent poverty and related deteriorating conditions in urban neighborhoods. Through its six-city Community Planning and Action Project, the Rockefeller Foundation sought to analyze local data on these conditions and to find the appropriate local partners to galvanize communities to address poverty. In 1987, the two foundations provided support to Case Western Reserve University’s Mandel School of Applied Social Sciences (MSASS), which in turn created the Center on Urban Poverty and Social Change. After a detailed study, the center produced the 1990 report "An Analysis of Poverty and Related Conditions in Cleveland Area Neighborhoods."
The report articulated a growing consensus that it was time to take bold action. As a result, the Cleveland Foundation formed the Commission on Poverty, a 30-member group charged with developing a long-term strategy to address Cleveland’s persistent poverty. The commission’s work was guided by five principles:
In support of its first principle, the commission identified five program frameworks: health, investment, education, family development, and human resource development. Realizing that implementing these program frameworks consistent with the guiding principles required considerable community-wide commitment and effort, the commission issued recommendations on how best to implement this community-building approach in its 1992 final report.
- Cleveland’s plan must be comprehensive and integrated.
- Strategies should be tailored to individual neighborhoods.
- An individual community strategy should begin with an inventory of the community’s assets, not its deficits.
- Local communities themselves must be actively involved in shaping strategies and choices.
- The approach should be tested in pilot areas and carefully evaluated before being replicated in other Cleveland neighborhoods.
After the Commission on Poverty concluded its work, a group of six commission members and six other public and private sector leaders continued to work on ways to implement the principles through a long-term strategy to address persistent poverty in the city of Cleveland. This group, called the Cleveland Community-Building Initiative Council (CCBIC), worked in collaboration with MSASS’s Center for Community Development to prepare an implementation plan. CCBIC selected four geographic areas for testing the new approach. Referred to as "villages," the areas selected were East (Fairfax), Central (King Kennedy Estates), West (Ohio City and a portion of Detroit Shoreway), and Mount Pleasant. CCBIC, with continuing support from the Cleveland Foundation, determined that the most viable mechanism for implementing the recommendations of the commission report was a new, independent organization. As implementing organization, CCBI received its nonprofit charter from the State of Ohio in September 1993 and its 501(c)(3) designation letter from the IRS in August 1994.
CCBI was incorporated by the original twelve CCBIC members and is presently governed by a board of trustees composed of eleven of those original members and seven others, including a representative from each of the four village councils. Each board member also participates on one of the village councils. The village councils, which seek to be inclusive and representative of various stakeholders, are charged with developing action plans to address poverty in the local neighborhood. Board members provide links to financial, intellectual, and technical expertise to implement the village-based agendas.
CCBI is staffed by a three-member administrative team and a coordinator for each village. Once organized, village councils participate in the selection of the village coordinators. Each coordinator is responsible for working with the executive director and the council to develop and maintain the functioning of the council and to provide technical assistance in support of village-based change strategies. To support the village-level work, a capacity-building component focuses on developing a core training curriculum based on recommendations from the councils.
Each village is a distinct geographic area whose residents are linked by a cluster of local institutions or a commercial center. By definition, the village concept suggests the interdependence of people who live in or do business within a geographic area; the village council is the mechanism to bring groups of stakeholders together. The village councils were organized over a two-year period. Start-up activities included establishing the geographic boundaries of the village, strategic planning, expanding membership to be more inclusive and diverse, conducting an asset inventory, establishing operating procedures and by-laws, and launching pilot projects.
Each village council deliberated about whether to maintain or revise the geographical boundaries identified by the Commission on Poverty in 1992. Thus far, three villages have redefined their geographical boundaries to include a larger population in the intervention area. All have developed operating procedures and by-laws. The councils vary in size, ranging from 13 members in East village to 27 members in West village. The composition of the membership also varies, with residents, social service providers, businesspeople, educators, and clergy represented on most councils. The councils have given considerable attention to composition, with at least one village developing very specific membership guidelines.
Although the method of work differs among the villages, each council has embarked on a process to develop collective vision, strategies, and activities for social change. Two councils—East and Mount Pleasant—called this a community strategic planning process, which included sessions to develop the local mission, strategies, activities, and goals, and produced a document charting a plan of activities to reach their goals. In contrast, the West village council opted not to conduct an extensive planning process, choosing instead to develop a mission statement, council composition guidelines, common themes for change, an operations statement for community outreach and communication, and a committee structure to address priority issues. West village has also developed a workplan identifying potential partners located in the village and the larger Cleveland community.
Village Characteristics and Assets
The West Village focus area is located on the west side of the city of Cleveland, Mount Pleasant is situated at the eastern border, and Central and Fairfax are centrally located, contiguous to one another. The villages vary in population size, ranging from slightly under 1,900 people in Central village to more than 31,000 in Mount Pleasant (table 1 357k). The population is predominantly African American in Central, East, and Mount Pleasant, while West village is racially and ethnically diverse.
All four villages have poverty rates above city averages. Poverty is extremely high in Central village, especially among children. Mount Pleasant is the most affluent of the villages. All villages have unemployment rates that exceed city and county averages, ranging from a high in Central village of 59 percent to a low in Mount Pleasant of 17 percent. Central also has the highest rate of adults over age 25 without a high school diploma and the highest proportion of female-headed households.
The villages are also quite diverse in terms of housing. Central and West villages encompass two public housing estates each, which typically have high vacancy rates. Nearly all Central village residents live in public housing, where many units are being internally demolished and renovated through a major modernization effort called Central Vision/Hope VI. Hope VI is a national initiative to renovate and create defensible space in public housing estates. West village, however, has more private market housing and a substantially lower vacancy rate. In Mount Pleasant, the rate of owner occupied housing units (49 percent) is similar to the citywide rate (43 percent) but lower than the county rate (62 percent).
Despite poverty and other less desirable indicators, the villages have several assets located in or nearby their geographic boundaries. East village, for instance, has a number of large institutions in the neighborhood, which could provide a range of resources, from job training and employment to health care services, housing, and community and business development. Additionally, East village is within the Cleveland Empowerment Zone, a designation that has the potential to bring additional job training programs and employment, business development, and housing resources to the village. Each village has numerous places of worship, representing a range of denominations, many of which provide social and other services to village residents. Community development corporations and other institutions are working within each village to improve the supply and quality of housing units. Other common assets include public and independent educational institutions from early childhood education through high school. Located near the East and Central villages are a community college, a public university, and an independent university.
The CCBI Evaluation
When the Cleveland Commission on Poverty proposed a set of principles, guidelines, and actions to reverse persistent poverty in 1992, it was agreed that the initiative would have to undertake a program of cross-sector, cross-system reform (Cleveland Commission on Poverty, 1992; Connell et al., 1995). These actions were to be implemented and tested in a few areas, with the clear expectation that they would be transferable to other locations.
Evaluation was embedded in the commission’s report. The report identified the Center on Urban Poverty and Social Change as the entity to develop a full assessment strategy, based on an initial evaluation plan constructed by the center and included in the report. After further work with the Aspen Institute Roundtable and in collaboration with CCBI, the center proposed a theory of change approach for evaluating the initiative. Baseline data collection was conducted in 1995-96.
An important aspect of the change process in Cleveland has been the notion that all stakeholders should be involved in designing and using the evaluation. Neighborhood residents and public and private sector leaders are all key stakeholders, including those who are involved in the initiative, potential users, and those affected indirectly by the strategies designed. Agency staff and volunteer leaders are also important stakeholders and need to be closely involved in the evaluation process. Area institutions, agencies, and organizations also have a stake in the direction and effectiveness of the initiative. Finally, those outside the target area, such as city and county government officials whose operations may be affected by the initiative or who may have a role in transferring successful programs and strategies to other areas, have a stake in the evaluation process.
The theory of change approach to evaluation is compatible with the way Cleveland stakeholders have conceived their role in the change process. Stakeholders view this approach to evaluation as a natural extension of the collaborative process of the Commission on Poverty and CCBI. Stakeholders have welcomed the opportunity to work with the evaluation team to specify the assumptions and hypotheses that guide decisions about the structure and components of the initiative. Early involvement in the evaluation process seems to have built trust among the stakeholders.
A theory of change approach is complex, intermingling the initiative’s strategies and its anticipated outcomes. It is explicit about the short-term and interim outcomes that will lead to the long-term goals desired. Similarly, evaluating an initiative like CCBI is a complex undertaking, requiring an explicit plan for the collaborative and analytic activities that make up the evaluation design.
Terms and Definitions
To reduce the confusion that can arise from the complexity of a theory of change approach, evaluators and stakeholders may want to adopt common language for referring to elements within the initiative and its guiding theory. CCBI uses the following definitions:
Initiating the CCBI Evaluation
- Strategy. Set of actions to achieve a purpose
- Activities. Actions that make up the strategy
- Outcomes. Early, interim, and long-term results of the chosen strategies
- Benchmark. Specified point in an activity or strategy that is agreed to be a sign of accomplishment
- Indicator. Defined manifestation of an outcome
In designing a theory of change evaluation for the Cleveland Community-Building Initiative and laying out a plan of action, the evaluators envisioned a series of specific but related steps:
Although the steps themselves are well articulated, little is currently known about the procedures evaluators should follow in carrying them out. The CCBI evaluation therefore presents an important opportunity to consider the practical aspects of such crucial early activities as eliciting theories of change from key groups, defining and reconciling theories to track, operationalizing key concepts of those theories, and establishing measures and data sources. Our recent experience in implementing these start-up activities in Cleveland may prove helpful to other initiatives and evaluators.
- articulate theories of change as viewed by multiple stakeholders
- identify important concepts and key benchmarks along the change pathway
- design methods to measure important outcomes and the related strategies
- collect data to measure the outcomes and benchmarks at specified points in time
- conduct analyses to determine whether outcomes and strategies are related to one another as specified by the theory of change
- modify theories as needed, based on experience
- provide regular feedback to stakeholders on progress and results
Eliciting Theories of Change from Stakeholder Groups
To define CCBI’s overall theory of change, the evaluation team and CCBI staff jointly identified three groups of key stakeholders who would be asked to describe their theories of community change under the initiative: CCBI staff, CCBI board members, and members of the four village councils. Each group plays a relevant and distinct role in defining CCBI’s theory of change.
The CCBI executive director manages overall operations and monitors programs and activities according to CCBI’s mission, goals, and objectives. The four village coordinators aid their respective village councils by providing technical assistance and advocacy and by helping to link village stakeholders with assets located inside and outside the community. The CCBI board—consisting of Cleveland-area business, church, educational, legal, medical, and social service leaders—provides direct guidance and assistance for CCBI program planning and development and oversees the financial administration of CCBI activities. The four village councils are made up of local residents and leaders. The councils play a critical role in developing and implementing programs, projects, and activities, as well representing the recipients of the services.
Eliciting Theories from CCBI Staff
The evaluators and the executive director of CCBI began the process of eliciting theories of change from CCBI staff by conducting a small group interview with the village coordinators. Prior to the session, the evaluation team reviewed CCBI written materials and circulated a paper on the theory of change approach to evaluation to the coordinators. The executive director was interviewed separately to avoid undue influence on the views of staff members.
The initial meeting with village coordinators included the evaluation team leaders and consultants from the Aspen Institute. The evaluators oriented the group to the task by explaining the theory of change approach and indicating that the meeting’s goal was to form a beginning sense of how each coordinator expected the change process to unfold. The following questions were used by the evaluators to guide the group interview:
The evaluators guided the initial conversation toward short-term strategies and outcomes, recognizing that some of the steps leading to initial outcomes had already been put into motion. The intention of beginning with short-term strategies and outcomes was to provide participants with a foundation upon which intermediate and long-term strategies and outcomes could be developed. After eliciting immediate strategies and outcomes, ensuing strategies and outcomes were gleaned until long-term goals were finally identified. Questions like "What would you do next?" and "What would happen next?" served this process. The separate interview with CCBI’s executive director was conducted in a similar fashion.
- What kinds of activities are you doing or planning to do in your role in the Cleveland Community-Building Initiative?
- What are you trying to accomplish with the neighborhood, families, or individuals in the villages?
- What initial results do you expect from your involvement in the project?
- What short-term results do you expect in the next year for the neighborhoods, families, and individuals in the villages?
- What results do you expect in the next five years for the neighborhoods, families, and individuals in the villages?
- How will the actions or activities lead to the short-term results?
- What long-term outcomes do you expect from your involvement in the initiative?
- How will the actions or activities lead to the long-term results?
- What barriers do you foresee in implementing the actions or activities?
During the small group interview, evaluators diagrammed the evolving theory on large sheets of white paper taped to the walls of the room. The evaluators also took written notes, documenting the details of what participants said regarding strategies and short-term, intermediate, and long-term outcomes. Theory-eliciting questions were asked by the evaluators, as well as the evaluation consultants from the Aspen Institute. The evaluators conducting the group interview ensured that every participant contributed to the discussion equally and that personal experiences with community-building initiatives were brought to the fore.
The evaluators observed that the village coordinators were specific when discussing the short-term strategies and outcomes they sought but became more abstract as the discussion moved toward long-term strategies and outcomes. The long-term strategies and outcomes tended to be more universal, often reflecting commonly held values. Overall, the evaluators present at the group interview noted that, when long-term strategies and outcomes were being proffered, values played a strong role in the development of the staff’s theory of change.
Group members appeared to be enthusiastic, engaged, and excited about the theory-eliciting process, and all four village coordinators said they had enjoyed the discussion. Further, the village coordinators found that the process of eliciting theories was not unlike the organizational planning process they had been through with CCBI. They stated that previous CCBI planning meetings had fostered a high level of involvement and participation in the group interview by increasing their capacity for thinking about and discussing ways to achieve organizational goals.
The evaluators took the information gleaned from the group interview and the interview with the executive director and developed a diagrammatic representation of the staff’s theoretical model for change. The first draft was disseminated to all relevant staff members in preparation for a second meeting, at which modifications and further elaboration of the theories were solicited. The recommended changes were made by the evaluation team and sent back to the staff members for further input. No more changes were suggested. Figure 1 (229k) represents the staff’s theory of change.
Eliciting Theories from the Board of Trustees
To ease scheduling burdens and assure the full participation of highly vocal and more reticent members, the evaluation team and the CCBI executive director decided to elicit board members’ theories of change through individual interviews. The chairman of the board sent letters to all members in advance, informing them that evaluators would be scheduling interviews to seek information about activities CCBI was seeking to initiate, benchmarks indicating accomplishments, and the role of the board in the CCBI initiative. The letter also explained that the interview would take approximately 1.5 hours, that it would be conducted in-person, and that all comments would be confidential.
Each interview was tape recorded and documented through written notes. As a theory developed for each individual, the interviewer constructed a conceptual diagram of what was being stated. A draft of the board’s theory of change, in flow chart form, was later developed and sent to all board members.
An evaluation team leader presented the draft at a meeting of the board to confirm the accuracy of the evaluators’ interpretation. Although discussion was brief because of time constraints, the evaluators described a strand of the theory to explain the flow of the model and clarified relationships between activities and outcomes. Board members pointed out that two important concepts—training and entrepreneurial activity—were missing from the model.
The board delegated further refinement of the theory of change to its program and planning board subcommittee. The evaluators are working with the subcommittee to prepare a final draft of the board’s theory of change, for submission to the entire board for approval. Figure 2 (198k) represents the board’s current theory of change.
Eliciting Theories from the Village Councils
When the evaluation began, each of the four villages councils was at a different stage of development. One council was newly formed, while another had completed a strategic planning process. The remaining two were engaged in project activities, having gone through their initial planning processes. The evaluators and village coordinators decided that eliciting theories of change should be coordinated with these developmental activities and should not be introduced too soon to a newly formed council.
Evaluators and village coordinators deliberated about methods of eliciting theories of change from the relatively large and diverse village councils before deciding to use focus groups led by a professional facilitator. Although interviewing a smaller group of village council representatives was discussed, it was determined that convening the entire council would serve the process best. Several village coordinators also raised concerns about ideological differences within the councils and their possible effect on the theory-eliciting process. Last, village coordinators agreed their own presence might inhibit village council members’ input.
An additional concern discussed by the evaluation team and village coordinators was that, as research, the evaluation would need to comply with Case Western Reserve University regulations for the protection of human subjects. The university’s internal review board required that village council members be informed of the purposes of the evaluation and their rights as participants. An information sheet was prepared using straightforward, non-technical language for use in obtaining informed consent from village council members prior to the focus group meeting.
Although the focus groups were expected to provide important information for theory development, the evaluators subsequently realized that the village councils’ strategic planning work was also pertinent. The evaluation team believed that each village’s strategic planning process should be linked to the theory-eliciting process, and vice versa, if the theory guiding the evaluation was to incorporate activities being implemented within each village. The evaluation team therefore decided to observe strategic planning meetings and review planning documents, as well.
Mount Pleasant, the village farthest along in articulating a theory of change, had completed its strategic planning process prior to the decision to use a theory of change approach to evaluation. The evaluators and village council members therefore had strategic planning documents in hand during the first theory-eliciting focus group. The focus group began with an explanation by the evaluators of the theory of change approach and an orientation to the kind of discussion anticipated. Each member had a chance to ask questions and was asked to give informed consent. The council members then engaged in a facilitated discussion of their expectations and thinking about changing the Mount Pleasant community. Because the strategic planning document had already been ratified, they were asked to build upon it, adding additional thoughts about their vision and relationships between activities and outcomes.
Overall, the council members participated actively, elaborating upon most of the ideas represented in the planning document and contributing some new activities and outcomes. The responses of the council members were recorded in written form throughout the process. The theory elicited from this focus group was diagrammed by the evaluators and presented to a feedback focus group, which suggested some modifications. The current version appears in figure 3 (264k).
Defining and Reconciling Stakeholders’ Theories
Having begun to elicit theories of change from the staff, board, and village councils, the evaluators’ next steps were to define the overall theory for each stakeholder group and find common and unique elements within and among the theories. These elements were identified by reconciling and combining individual differences within each group, comparing theories within and between groups, and determining whether there were one or several theories or theories within theories.
Defining the Staff’s Theory of Change
The staff’s theory of change was developed primarily during the theory-eliciting group process. More specifically, the small group interview was structured so that questions pertained directly to strategies and outcomes for CCBI as the organization moved from the present to the future. This structure provided evaluators the opportunity to construct and diagram the first draft of the theory as the meeting unfolded. During the interview, each village coordinator was given an opportunity to propose strategies and outcomes relevant to particular points in the initiative’s time frame. If coordinators disagreed with each other about the strategy that best served a specific outcome and a compromise was not reached through discussion, then all the strategies were placed in the theory. For example, to reach the intermediate outcome of "successful projects are implemented," two strategies were suggested: "interest groups undertake specific projects" and "resources are obtained to support the agenda." Since the village coordinators believed these strategies to be important for the implementation of successful projects, they were both included in the theory. Similarly, if a strategy was proposed and agreed upon, but more than one outcome was proffered, then all outcomes were placed in the theory.
To complete the theory, the evaluators examined the executive director’s theory of change and integrated it with the staff’s theory. When aspects of the two theories were in agreement, no changes were made; but when the executive director suggested novel strategies and outcomes, these were added to the staff’s overall theory. After evaluators refined the staff’s first theory of change, it was presented to the staff for further modification. Then, the theory was finalized.
Defining the Board’s Theory of Change
The board’s theory of change was developed and diagrammed during an evaluation team meeting that followed individual interviews with ten of the sixteen board members. The evaluation team first examined all ten theories of change for common or similar strategies and short-term, intermediate, and long-term outcomes. In general, these were noted when more than one board member expressed a similar idea. An example of a common strategy was "to identify community leaders, assets, and resources," and the common outcome associated with this strategy was "the formation of a strong village council."
The evaluation team then examined each board member’s theories for unique elements. When a strategy or outcome was proposed by only one board member, the evaluation team discussed whether or not to incorporate it into the theory. Although most of these unique elements were included in the board’s theory, some were left out for further clarification during the feedback process. The first draft of the board’s theory, represented during the evaluators’ meeting on a poster-sized flow chart, was generated by computer and sent to board members for review and discussion at the next board meeting.
Defining Theories of Change of the Village Councils
Although the evaluators drew mainly on strategies and outcomes which an entire village council had agreed upon, the first draft also included elements that had been proposed vigorously by particular participants. For example, one council member felt strongly that more community residents should take advantage of service sector jobs. Others disagreed, stating that service sector job opportunities offer limited income potential and leadership opportunities. Evaluators incorporated service sector job opportunities into the theory with the assumption that the feedback process would ultimately determine whether or not it was included in the final theory.
In Mount Pleasant, planning documents and focus group notes were used to diagram the village council’s theory of change. The evaluators first listed all outcomes and strategies. Each strategy and outcome was then placed within the initiative’s time frame according to the village council’s specification, and links between strategies and outcomes were established. Last, Mount Pleasant’s theory of change was diagrammed using a computer presentation program and presented to the village council for feedback during a monthly meeting.
Reconciling the Stakeholders’ Theories of Change
As figures 1, 2, and 3 show, the theories of the CCBI staff, the CCBI board, and the Mount Pleasant village council are similar with respect to long-term outcomes. More specifically, the staff’s ultimate outcome, "community identity and optimism rise more," is similar to the board’s "people choose to move to or remain in the community" and the village council’s "village pride and energy revitalized." Each theory also contains ultimate outcomes pertaining to the development of strong families and youth and a more robust business and economic environment. Both the staff and board theories include long-term outcomes related to improving the quality and responsiveness of neighborhood institutions and services. The staff and village council theories include long-term outcomes pertaining to safety. The only unique ultimate outcomes are quality education, part of the village council theory, and improved housing and commercial areas, included by the board.
In general, the staff and board theories contain more similarities than differences. Many of the steps leading to village council formation are similar: both theories stress identifying community assets and leaders, developing a sense of common interest, training resident leaders, and recruiting a diverse group of volunteer council members. Also, after the village council is formed, the steps toward realizing ultimate outcomes are similar in the staff and board theories, including agenda development, capacity building or resident training, networking and collaboration, acquiring resources, and activity or project development. The staff and board theories are also similar in their lack of specificity regarding methods for conducting each step. For example, the "agenda is formed" step in the board’s theory and the "council sets an agenda" step in the staff’s theory do not explicate the "micro-steps," or activities needed to form an agenda. The evaluators did not see this as problematic because village council theories would likely specify those activities.
The Mount Pleasant village council’s theory of change does not differ from the staff and board theories as much as it elucidates steps that focus on capacity building, resident training, networking and collaboration, acquiring resources, and activity or project development. For example, both the staff and board theories mention the need to get access to resources. The village council theory elaborates the activities that Mount Pleasant intends to implement, including developing an interactive village roundtable, disseminating legislative and political information, forming a Jaycees organization, collaborating with the city, developing and disseminating resource information packets, and initiating a village networking program. Some of the council’s intermediate outcomes closely resemble staff and board general steps as they move toward ultimate outcomes.
Using the Theories of Change to Design the Evaluation
Although the theories of change are incomplete, they have been useful in guiding the early stages of the evaluation. Data collection and measurement strategies have been planned to establish baselines and cover the initiative’s initial stages, since the three sets of theories show considerable agreement and clarity about how early activities will lead to short-run changes or accomplishments. The evaluators believe that stakeholders cannot yet be explicit about steps in the intermediate and long term, but the theories will be revisited periodically to see when and if further elaboration becomes possible.
It is important that the evaluation move forward, since stakeholders are eager to know how they are progressing. Thus, the evaluators have produced an early, streamlined version of the theory of change, defined key concepts and definitions, developed preliminary benchmarks, and begun to gather baseline information about longer-term outcomes for which there is considerable support.
Streamlining the Early Theory
Some stakeholders expressed the view that the complexity of the theories made it difficult to see how those theories would lead to meaningful evaluation findings. When the evaluators confronted the task of operationalizing and measuring the elements of the theory, they too acknowledged the need to simplify. After examining the board, staff, and village council theories of change, the evaluators concluded that the elements could be grouped tentatively into sets of major strategies and outcomes, each of which could then be further specified for evaluation.
The general theory of change that seems to fit CCBI early in its development is shown in figure 4 (90k). The formation and operation of village councils is the first outcome. Each village is to establish an inclusive, representative, and collaborative structure that can effectively move the village toward its goals using the community-building principles established by CCBI. The process of establishing the village council and its work in identifying community assets begins to unleash greater participation of residents in community affairs, while also serving as a forum for collaboration as groups and institutions are brought into the planning activities. The work of the village council builds upon, celebrates, and enhances the assets of its community.
A focused agenda—established by the council—garners resources and directs the community’s energy toward comprehensive change consistent with its long-term objectives. For example, well-defined projects move the community toward its goals while building capacity and increasing involvement within the community. Growing civic involvement and the action projects themselves build social capital by creating new social connections and forms of organization. Communication channels become more effective as a result of rising participation and action projects such as newsletters and bulletin boards.
The growing implementation capacity serves as a magnet for additional internal and external resources. Some projects are deliberately structured to involve actors and systems outside the village, thus beginning to strengthen external relations. The agenda may also include expanding and improving services, which requires specific community actions directed toward systems and agencies.
The community’s growing capacity to achieve its goals allows advancement in areas it considers important, while stronger social structures sustain this movement. Thus, neighborhood identity, security, service quality, economic opportunity, and family development are all promoted by the strengthened social organization within the community and the improved connections to the larger society.
This general theory will change as several of the villages begin to put forward details of their strategies. Specific steps linking interim to long-term outcomes will be added as the initiative unfolds.
Before these strategies can be evaluated, they needed to be broken down into elements about which judgments can be made. In explaining this to stakeholders, the CCBI evaluators found the term benchmark to be useful. Achieving a benchmark is a sign that a strategy is progressing in a desirable and effective manner.
Benchmarks suggest implicitly that performance can compared against some standard of achievement or definition of what is acceptable or desirable. The establishment of benchmarks can draw on the experience of the stakeholders, expert opinion, literature on best practices, or scientific research. CCBI plans to use all these sources, but at present the process is incomplete. Even so, the evaluators have developed a beginning set of benchmarks, some of them drawn from what was learned in the theory-eliciting process and others based on literature and the advice of experts.
A list of benchmarks for the initial CCBI outcomes is presented in table 2 (282k). Each benchmark is related to an important aspect of the success of the strategy, and, where possible, a threshold for accomplishment is specified. For example, while stakeholders assume that councils must be inclusive in their membership, the benchmarks specify categories of individuals to be included and reference points for determining adequacy. At selected points in time, the evaluation can compare actual performance with these benchmarks. For each strategy, numerous benchmarks have been proposed, each of which requires consideration of the evidence—and the measurement procedure—that will be required to know its status. Where possible, established measurement methods will be used.
To ensure that stakeholders are aware of and agree upon these benchmarks as visible signs of progress, the evaluation team conducted meetings with all stakeholder groups. For example, the evaluation team met with the Mount Pleasant village council to verify their agreement with the benchmarks, establish priorities, and review the criteria for measurement. The council agreed with the concepts but had concerns about timing. Also, the council was dissatisfied with the dichotomous nature of the operational definitions, which measured only presence or absence of a phenomenon. The evaluation staff agreed that most of the benchmarks did not adequately specify an established standard and that a continuum should be used, rather than a dichotomy. It was also agreed that, for some villages, the current measurement would constitute a baseline.
The evaluators also met with the CCBI staff to discuss priorities regarding the benchmarks and criteria for measurement. The staff thought the benchmarks were helpful but suggested a few changes in wording.
Collecting Baselines for Long-Term Outcomes
CCBI has also begun to track long-term outcomes. Although the theories are not fully developed and the path to long-term results remains unclear, it is important to begin defining these ultimate outcomes so that a baseline can be obtained. Further, tracking these outcomes as part of the evaluation process should help keep the initiative’s long-term aims visible and aid in sustaining its focus.
The theories of change from all three stakeholder groups have specified long-term outcomes. An examination of these suggests that there is considerable agreement among the stakeholders on the following general outcome categories:
CCBI staff and evaluators have just begun to consider the possible indicators that could be used to define each of the categories. Initially, we expect to choose indicators that can be drawn from available data in order to create a retrospective baseline. This will help establish reasonable expectations for the rate and amount of change that can be expected as a result of CCBI’s work. Further, it is important that stakeholders be prepared to review the indicators and interpret the measurements realistically, so that the data are not perceived as discouraging or excessively negative. Both evaluators and stakeholders need assurances that the information will be interpreted in the proper context and used to promote positive planning and action.
- Economic opportunity. The economic potential and well being of residents and the economic vitality of the neighborhood is improved.
- Neighborhood identity and pride. The neighborhood is viewed positively and valued by residents and outsiders.
- Safety and security. Community residents and visitors are not threatened or harmed and their property is secure.
- Strong institutions and services. Educational, cultural, religious, service, and civic organizations are effective and serve and involve residents.
- Family, child, and youth development. Families are stable and supported, and children and youth are healthy and developing.
Establishing Measures and Data Sources
Although some important CCBI elements are only in their beginning stages, it has become apparent that certain basic data collection mechanisms must be implemented immediately. Otherwise, important information may be lost or forgotten. These mechanisms include:
Issues in the Theory of Change Approach
- Documentation of activities. Specific activities carried out by staff, village councils, residents, and others will be documented in a data base by CCBI staff. Documents such as minutes, reports, and memos will also be collected. Special data collection instruments may be developed but will be kept to a minimum.
- Key informant interviews. Measurement of early benchmarks will probably rely on questions posed to staff, board members, village councils, and residents. Questionnaire items will be developed and tested for validity and reliability. Interviewers will be trained to administer these questions and record and code the answers.
- Social and economic indicators. The Center on Urban Poverty and Social Change maintains a detailed longitudinal data base of social and economic indicators for all Cleveland neighborhoods. Drawn from administrative agencies and the census, these data will be valuable in calculating baseline and ongoing measures for many of the long-term outcomes. For some outcomes, the center will add new data sources, especially in the areas of educational and developmental outcomes for children and youth and service and institutional effectiveness.
- Oral history/archiving. Working with a historian, an oral history group of selected residents will record the process of community building within their villages. The transcribed oral records will be used by the evaluators and will also serve to communicate the principles and promise of community building within the village and externally.
- Resident surveys. A survey of neighborhood residents will probably be necessary to obtain measures of some outcomes. In particular, a survey could focus on residents’ perceptions of their neighborhood’s identity, cohesiveness, and effectiveness and on changes in individuals and families.
- Surveys of institutions and organizations. The theories as they now stand are not explicit about how institutions and organizations in the community will need to change in order to produce the desired long-term changes. Surveys of institutions and organizations may be needed to measure those concepts as they are specified.
Traditional evaluation designs can rely on well-established protocols and standards, but these are not yet available to evaluators using a theory of change approach. From our initial steps toward evaluating CCBI, we offer several reflections upon the method and some recommendations to the field.
Considering the Role of the Evaluator
The role of the evaluator—a key issue in the application of the theory of change approach to evaluating CCIs (Brown, 1995; 1996)—has also been an important issue in the early steps of the CCBI evaluation. The very nature of eliciting theories of change from stakeholders placed the evaluators into new and less traditional roles, as stakeholders and evaluators interacted mutually as teachers and learners through an iterative and ongoing process. Early experiences in evaluating CCBI raised several issues regarding relationships between the evaluator and stakeholder groups and the evaluator’s effectiveness as a trainer.
Relationships with the Board, Staff, and Village Councils
The CCBI evaluators had previous experience with the initiative and its stakeholders, having earlier provided information and data analysis to community stakeholders to increase their understanding of the nature of poverty and its consequences in Cleveland neighborhoods. Thus, the capacity of the evaluators to provide a detailed longitudinal data base on social and economic indicators from administrative agencies and census information was widely known and utilized by CCBI. This capacity to gather, manage, and analyze geographic-specific data continues to meet the needs of the board and staff in the planning and implementation process. Together, CCBI and the evaluators have examined trends that provide a better understanding of the villages.
All stakeholders were eager to participate in defining the CCBI theory of change. The evaluators, staff, board, and village councils worked together as collaborators and inquirers to elicit theories of change, and the meetings where theories were elicited functioned as open exchanges with opportunities for all present to participate in the discussion. Work to date has promoted stakeholder participation in the theory development process and clarified the roles and responsibilities of stakeholders.
In many ways the collaborative process has created demands for the evaluation work by stakeholders. The goal for the evaluation is not only to contribute to an understanding of the effects of the initiative, but also to use information for community and organizational decision making. Stakeholders have asked, "How are we doing on accomplishing our goals?"
Collaborative decision making has been central to theory development and measurement. Early on, village coordinators convinced evaluators to delay theory development work in two of the four villages where the respective councils were organizing their strategic planning process. The evaluators agreed, in order to be supportive of the growth of the organizations and to fit into a naturalistic process.
The Evaluator as Trainer and Consultant
Training is an important part of the evaluators’ role in the CCBI experience, especially in surfacing theories of change. Formal and informal training can help participants understand the theory of change approach while also demystifying the evaluation process, clarifying roles and expectations, building trust, and helping evaluators and stakeholders get to know one another and understand and acknowledge fears (Sonnischsen, 1994).
The evaluators had several opportunities to train and consult with stakeholders on understanding the theory of change approach. Each stakeholder group was given the opportunity to grapple with the concept in initial and feedback sessions, and board members and staff were given access to selected literature on evaluating CCIs. Moreover, in the iterative process of specifying the theories of change, evaluators have helped stakeholders reflect upon the plausibility of the strategies planned or being implemented and the outcomes stated. This process has put the evaluators in the role of consulting with stakeholders on the development of program strategies and outcomes and of articulating underlying assumptions. In one feedback session, for example, the evaluation team challenged the notion put forward by the village council that involving young people in physical clean-up of the neighborhood would build youth leadership. This challenge led to a discussion of other youth development strategies that the literature has identified as promising contributors to youth leadership.
The use of verbal and written progress reports is also part of the training process. The evaluators have met regularly with staff and board members to discuss progress and have provided quarterly written progress reports to the CCBI board and staff, thus encouraging multiple stakeholders to explore strategies and benchmarks. If the process of clarifying the theory of change serves as a model for subsequent processes, stakeholders can be expected to be closely involved in developing the evaluation design and in the final selection of measures.
Ultimately, the evaluator may also be instrumental in uncovering biases inherent in the various roles of stakeholders regarding the initiative. Such biases inform the perspectives of each stakeholder group, which in turn can yield significant information about how the initiative is working. The evaluator could also facilitate interactions among the groups and help them attain a unified view of the initiative.
Working with Funders
The evaluators, CCBI staff, board members, and village councils have been involved in eliciting the theories of change and the iterative process of defining the assumptions that underlie the initiative. Yet funders—another stakeholder group—have not yet participated directly in the CCBI theory of change process.
CCBI has worked with two funders, a foundation and a government agency. For the foundation, a program officer has been involved with CCBI since its origins through the Cleveland Foundation Commission on Poverty, and the chief executive officer has been aware of the development of the initiative. Recently, the foundation conducted an unprecedented all-staff site visit to provide program officers in different funding areas with first hand knowledge of the initiative. During the site visit, the evaluators had an opportunity to present the theory of change approach and to provide each program officer with literature about the approach.
Both funders have expressed an interest in the evaluation and see it as integral to the initiative’s implementation. Even so, the processes of eliciting theories and defining measurements have so far focused on institutional and programmatic assumptions rather than funder expectations. The CCBI evaluation may be somewhat unique in this regard, since many previous and current evaluations have had considerable funder involvement in their design.
Eliciting and Defining Theories of Change
Facilitating Effective Groups
In this first phase of evaluating CCBI, both individual and group approaches to eliciting theories were tried. We conclude that the group process added value: not only does it serve to build consensus and yield a common theory, but it also seems to allow group members to see one other’s perspectives more clearly.
Even so, there are limits on the ideal size of the group. We have found that, to allow participants adequate time to relate their theories and be pressed for clarification, groups should be no larger than eight. For larger groups, the task of working directly with the evaluators has been delegated to a subcommittee, which routinely reports issues to the larger group for discussion. This strategy can be supplemented with individual interviews.
Mapping Theories of Change
Communicating the developing theories of change to the groups at each stage has proven somewhat unwieldy. Detailed theories are very difficult to display on paper, especially at early stages when there is uncertainty about concepts and language. For example, it is difficult to tell at the outset whether people have truly different concepts or are using different words to describe the same idea. The temptation to simplify prematurely, just to fit the theory on paper, can be very strong.
To maintain underlying complexity without confusing the audience, we have experimented with layering levels of detail on computer screens that can be tied to more comprehensive concepts on a screen with a simpler diagram. Microsoft PowerPoint has proved to be a useful piece of software for this purpose. However, displaying these diagrams to a large group requires projection equipment that is not readily available to many neighborhood organizations.
The fact that many strategies and outcomes occur repeatedly and over time presents an additional difficulty in diagramming theories of change. In the CCBI theory, for example, "agenda formation" is to occur repeatedly, with the agenda modified on the basis of previous experience, new information, and changing values and perceptions. It is difficult to represent this repetition and link the activity to outcomes that are different at different points in time.
We have also struggled with urging stakeholders to be more explicit about detailed steps along the pathway of change. Even so, important links are missing between early strategies and long-term outcomes. Although stakeholders have a fairly clear vision of the changes that they want to see in their communities, they find it more difficult to see the steps that must be taken to arrive at these outcomes. Unfortunately, turning around economically distressed neighborhoods, making institutions stronger and more responsive, and promoting family and child health are very difficult and complicated challenges. The powerful strategies and know-how required to effect those changes are not revealed in the initial theories.
We anticipate that, as staff and village residents experience success with initial action projects, they will gain confidence and capacity to take bolder steps. At that stage, they will be asked to reconsider the long-term outcomes and make specific choices about the pre-conditions for change. Since expanding local capacity is a fundamental tenet of community building, CCBI will need to involve consultants and advisors who can contribute to the staff’s and residents’ abilities. The theories can then be more fully developed to guide the ensuing evaluation.
Determining Benchmarks and Indicators
Distinguishing Outcomes from Strategies
Traditional evaluation has distinguished between process and outcomes. Strategies and their related activities have been considered process, while longer-term results are generally seen as outcomes. We have found the distinction between process and outcomes difficult to apply in our work with CCBI.
CCBI’s early work is clearly about putting structures and processes into place that are intended to build the communities. Within the theory, however, these are pictured as strategies, the accomplishment of which are early outcomes. For example, putting into place a recruitment process for village council members contributes to village council formation and operation. Having an accepted and effective method for ongoing recruitment can be seen as a short-term outcome.
Rather than getting bogged down in these distinctions, we are currently using words like benchmark and indicator, which seem to be more familiar to stakeholders. We have tried to describe processes in terms of observable outcomes that can serve as signs that the processes have been accomplished well. Thus, outcomes will be the primary focus of our measurement.
Adopting an outcome focus precludes the detailed descriptions of processes that have been characteristic of many previous CCI evaluations. We are concerned, however, that the evaluation may miss important stories about individuals and the efforts they make or problems and barriers that occur in the political and social contexts. We hope that the oral histories will record these events and help portray the rich fabric of CCBI.
Establishing Standards and Thresholds
The idea of benchmarking and the interpretation of indicators on an ongoing basis implies a comparison against a definition or standard. In industry, benchmarking often takes place by comparing a company’s performance with that of its competitors. In public health, the comparison may be within the same population over time. Standards also may reflect fundamental values and societal goals or be based upon scientific determinations.
In the field of CCIs, values and goals are rarely stated as clear thresholds or levels. The experience of practitioners may be a useful source for a threshold for what works, but that experience is not always codified or readily available to evaluators. Comparing communities with one another or with themselves over time may be a useful way to establish benchmarks, but there is little existing experience with these methods. The field should invest in developing its knowledge base toward this purpose.
Choosing an Appropriate Level of Detail
If we knew the one or two most powerful ingredients of any strategy, the lists of benchmarks and indicators could be greatly simplified. Regrettably, the knowledge base about community change is insufficiently developed to point to those key elements. For example, experience suggests that inclusiveness is related to the success of coalitions, but to what categories of people does that suggestion apply? Does inclusiveness demand that all geographic areas be represented at the table? At this stage, it seems wise to include as benchmarks elements that may make a difference in the initiative’s success and ones that could be improved if the evaluation finds them wanting.
Making the Evidence Compelling
The evidence coming out of the CCBI evaluation must be compelling and useful for the purposes to which it will be applied. Those purposes may be different for the various stakeholder groups.
Staff and residents are most likely to need evidence in a form that will tell them how to improve what they are doing, or when something has been accomplished and it is time to move on to something else. For them, compelling evidence must distinguish between strategies that were incompletely implemented and strategies that were done according to the standards but did not produce the desired changes. They will want to know whether a correctly implemented strategy did not work because of an external force or because the theory was flawed to begin with. When outcome indicators move in the positive direction, they will want to know which strategies were most helpful and should be continued and which ones have accomplished all they can.
Board members and funders are likely to want answers that can guide future resource allocations. For them, compelling evidence should distinguish between strategies that have produced good results and those that have been less productive. They will want to know whether the invested resources have produced something of value that would not have occurred otherwise.
Our experience thus far suggests that the theory of change approach will be useful in providing the compelling evidence of improvement needed by staff and residents. Seeing their own thoughts operationalized should guarantee that the evaluation will provide findings they can act upon, and frequent feedback of findings and opportunities to make modifications should prevent the evaluation from departing from the reality of the initiative. Sensitive, well-timed measurements can help evaluators distinguish weak implementation from theory failure and fairly describe implementation problems, so they can be resolved.
It is less clear how the evaluation can build a compelling case that the theory itself is powerful and valid. The CCBI theory as it now stands is not adequate to account for competing explanations and the influence of factors not included in the model. This will not present an insurmountable problem for testing the early stages of the theory, when most of the predicted effects are internal to the initiative itself. Later predicted changes in the community and its residents, however, are less internally controlled or explained. Much greater consideration will need to go into fully specifying the model, so that the influence of external factors can be ruled out or explicitly brought into the change process. Without the detailed steps that are currently missing from the theories, it will be difficult to produce the compelling evidence stakeholders need in allocating resources among promising initiatives.
Building a Methodology That Works
The CCBI experience in implementing a theory of change approach to evaluation confirms that the method is a good fit with the way CCIs work. It allows the evaluation to be shaped by the ideas, values, and aspirations of stakeholders at all levels and at many points. The evaluation design is not imposed from without, but rather is gradually shaped through a collaborative, analytic process. This deepens stakeholders’ understanding of and commitment to the evaluation plan. It also increases the likelihood that the evaluation will be used by the stakeholders to improve the initiative.
Considerable work remains to be done in specifying and refining the theory of change method. This will occur as the technique is applied in many CCIs and the experience of evaluators and practitioners is codified. Several methodological issues stand out in particular among those that need to be addressed:
These and other methodological challenges require serious pilot investigation within the context of multiple communities. The complexity of evaluating CCIs, combined with the considerable investment in them and a vital need to know their effectiveness, suggests that this work has some urgency. The evaluation committee of Aspen Institute’s Roundtable on Comprehensive Community Initiatives for Children and Families has served the vital function of stimulating, supporting, and synthesizing this type of work, but the promise of theory of change evaluation is just beginning to unfold. Only through sustained effort can we expect to refine the methodology and achieve its practical application.
- Measurement. Most of the benchmarks that have been identified for CCBI do not have existing, tested measures. As currently phrased, they seem to call for simple dichotomies, although more refined quantification will be needed to reveal subtle changes. It is only when more sophisticated and sensitive measures are developed that we will be able to analyze how change in one concept leads to another—the kind of evidence promised by the theory of change approach.
- Modeling change. Theories of change tend to be depicted as linear, although the real process of change is dynamic and reciprocal. The initiative may be moving along several pathways, and the effects of strategies may depend upon their synergy. Considerable work is needed to discover how data can be analyzed to represent these complex processes correctly.
- Extraneous influences. As the evaluation moves to examine interim and long-term outcomes, the problem of alternative explanations will become more apparent. Generating compelling evidence for the value of the initiative may depend on adequately taking into account social and economic processes outside the initiative that may be influencing outcomes, positively or negatively. Choosing the correct forces to include will require some type of higher order theory about the communities in which the initiative is working and how outside factors affect them.
Financial support for this paper was provided by the Aspen Institute’s Roundtable on Comprehensive Community Initiatives for Children and Families. The assistance of the following individuals and groups is gratefully acknowledged: Anne Kubisch, James Connell, and Karen Fulbright-Anderson of the Roundtable’s evaluation committee; Greg Brown, Tracy Robinson, John Ward, and Ginna Fleshood, Cleveland Community-Building Initiative (CCBI ) village coordinators; members of the village councils; the CCBI board of trustees; Emma Melton, group facilitator; and Carmen Griffey, program analyst at the Center on Urban Poverty and Social Change, Mandel School of Applied Social Sciences, Case Western Reserve University.
Brown, Prudence. 1996. "Evaluation of Comprehensive Community Building Initiatives." In Core Issues in Comprehensive Community-Building Initiatives, ed. Rebecca Stone. Chicago: Chapin Hall Center for Children.
Brown, Prudence. 1995. "The Role of the Evaluator in Comprehensive Community Initiatives." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Connell, James P., Anne C. Kubisch, Lisbeth B. Schorr, and Carol H. Weiss, eds. 1995. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: Aspen Institute.
Cleveland Commission on Poverty. 1992. The Cleveland Community Building Initiative.
Coulton, Claudia, and Jill Korbin. 1996. "Measuring Neighborhood Context for Young Children in an Urban Area." Journal of Community Psychology 24 (1):5-32.
Sonnichsen, R. C. 1994. "Evaluators as Change Agents." In Handbook of Practical Evaluation, ed. J. S. Wholey, H. P. Hatry, and K. E. Newcomer. San Francisco: Jossey-Bass.
Weiss, Carol Hirschon. 1995. "Nothing as Practical as Good Theory: Exploring Theory-based Evaluation for Comprehensive Community Initiatives for Children and Families." In New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts, ed. James Connell et al. Washington, DC: Aspen Institute.
Back to New Approaches to Evaluating Community Initiatives index.
Copyright © 1999 by The Aspen Institute
Comments, questions or suggestions? E-mail firstname.lastname@example.org.
This page designed, hosted, and maintained by Change Communications.