Project Outcome: Helping Libraries Capture Their Community Impact

by Denise Davis & Emily Plagman on September 15, 2015

By Denise Davis and Emily Plagman

Denise is Deputy Director, Sacramento (CA) Public Library, ddavis@saclibrary.org. Emily Plagman is a Project Manager for the Public Library Association, eplagman@ala.org.

Project Outcome is PLA’s latest field-driven initiative, helping libraries to capture their impact in the communities they serve. In 2013, libraries and researchers formed a task force with a mission to develop and test a simple set of outcome-based surveys for any library type to use when measuring the outcomes of their services and programs. Project Outcome builds on the task force’s work by providing resources and support to help any library set strategic goals, measure outcomes, communicate their findings, and successfully achieve their goals.

Project Outcome started as just an idea when then-PLA President Carolyn Anthony announced that she would make outcome measurement a primary initiative during her tenure. “What is needed,” she wrote in an August 2013 Public Libraries Online article, “is a set of performance measures that can capture the services public libraries are currently providing in their communities, with guidelines for conducting the measures to ensure consistency and validity. We also need to measure the outcome or impact regarding the difference that some of these services make in the lives of individuals and the well-being of the community.”¹

Shortly thereafter, a PLA Task Force was formed comprised of librarians and researchers. Their mission: to develop standardized measures of effectiveness for widely offered public library programs and to promote the training for implementation and use of the measures across public libraries. They kicked off their work in earnest in July 2013 and spent the next year developing surveys specifically designed to be easy-to-use, flexible, and accessible for libraries of all sizes and demographics. The group pilot tested the surveys in the fall of 2014 and finalized them in early 2015.

The seven topics are:

1. civic/community engagement
2. digital inclusion
3. early childhood literacy
4. economic development
5. education and lifelong learning
6. job skills
7. summer reading

The Task Force’s work caught the eye of the Bill & Melinda Gates Foundation, resulting in a three-year, $3 million grant to PLA to build a project around the surveys—complete with resources, support, and trainings. Funding will allow PLA to expand the program with a goal of developing an online framework for public libraries to easily enroll and start using outcome measurement as well as creating a framework for a field-wide shift toward making outcome measurement a business-as-usual practice. In January 2015 the grant kicked off.

Project Outcome Implemented

By spring the project was given the name Project Outcome and development of content began. On June 26, 2015, Project Outcome launched publicly at a preconference during the American Library Association (ALA) Annual Conference in San Francisco.

At the project launch, more than sixty libraries met to learn how to pair strategic goals with service needs; to measure programming impact; and to identify ways they can take action such as communicating the findings of the surveys and clarifying their needs to funders, staff, patrons, or community. Project Outcome offers the resources needed to do all this work in one place and is designed to be comprehensive, customizable, simple, and available to any size public library interested in surveying one program once or many programs often. It sounds like Project Outcome is really easy, right? That’s because it’s designed that way.

Project Outcome helps libraries implement outcome measurement, capturing the change in patron’s beliefs or skills as a result of using library services and programs. It will help library staff measure the knowledge a patron gained after taking a job skills or résumé writing class. It helps librarians quantify the patron’s value of early childhood literacy programming, a bookmobile, or a cultural learning activity. It provides a centralized online location where library staff report and see the results of their surveys. It supports their efforts to use that data as evidence of what they already know in their gut to be true—that patrons find significant value in the services they offer and that libraries are not just buildings with books, but integral players in healthy and vibrant communities.

What Is an Outcome?

Public librarians are already accustomed to measuring their value in the form of outputs—they measure circulation, computer usage, and door counts. An outcome, however, is a specific benefit that results from a library program or service. Outcomes can be quantitative or qualitative, and are often expressed as changes that individuals perceive in themselves—like new or improved knowledge, skills, attitudes, behavior, or status. Project Outcome is designed to help libraries measure and identify the value they bring through critical services and programs (see table 1).

Why Measure Outcomes?

Outcome measurement helps libraries make an even stronger case for funding, partnerships, adding staff, or garnering community support—because it offers librarians methods for presenting the patron’s perspective. Project Outcome simplifies the steps to measure a crucial component, the services they provide to the public, and successfully uses the findings to achieve goals. Outcome measurement can help libraries:

• To better measure and improve upon the impact in the community. The services libraries and library staff members provide are all intended to improve patrons’ lives. Outcome measurement helps quantify what they hear every day from patrons: that library services make a difference in their well-being, level of understanding, and change in knowledge or skills.
• To support planning and assessment over time. Whether working toward a specific goal within a library’s strategic plan or developing new long-term plans, information about how current services benefit patrons today is essential.
• To help better manage services and resources. Outcome measurement helps ensure that the money and time spent on a program or service is the best use of those limited resources.
• To justify funding requests. This is one of the most important reasons! Anecdotal evidence can help illustrate your library’s value, but libraries need more to attain resources. Adding outcome measurement data alongside anecdotal evidence and output data will strengthen your library’s reports and proposals and allow you to better advocate, providing library leaders with a more compelling case for continued or increased library funding.

How Does It Work?

Project Outcome provides simple survey instruments that library staff can use to collect insights and data about all the ways libraries are meeting the needs of their communities. In a few steps, public library staff set goals; use the Project Outcome portal to enter data; view summary results; generate reports; and access Project Outcome resources to successfully take actionable steps to meet their advocacy, partnership development, and strategic decision-making goals.

The process is designed to align with services and activities that libraries are already conducting, and is simple enough that just about every library system can use it. Project Outcome uses an easy four-step process (see figure 1).

Some timing factors that may help libraries decide when to administer Project Outcome surveys include:

• anticipated changes in funding that will affect the library’s ability to provide its current services—whether it is a scheduled budget review or a special situation;
• an upcoming opportunity to seek funding to grow or add existing services;
• the library is beginning a new service and wants to set a baseline to compare outcomes;
• in order to show progress in a priority service area that aligns with a strategic plan, goal, or need;
• the library is planning to make service additions/changes or reductions and staff want to understand which services have the greatest outcome for the community; or
• library programming staff are required to report their performance and want to use outcomes evidence to demonstrate value.

The Role of Staff

Each member of a library team serves an important role in the success of implementing Project Outcome surveys. Pilot test-users reported that the greater the involvement and buy-in of staff at all levels, the greater success in both breadth and depth of survey distribution, use, and response rates—and, the greater the adoption of changes by the staff themselves because they were willing to make changes to improve the effectiveness and quality of their programs.

Examples of staff roles include:

• Librarians. Administer surveys at the end of programs that they manage.
• Library managers. Make decisions to change programs and services based on survey results.
• Library staff that interact with partners and community leaders. Package and communicate results appropriately for each audience.

Survey Topics and Sample Layout

Project Outcome provides seven topic areas to measure with short, easy-to-apply surveys (see table 2). Users are encouraged to select surveys based on the strategic focus or need of the library. It is not anticipated that libraries will administer all seven surveys. The data collection depth and periods are determined by the library based on how they envision using the results.

Each survey is six questions long and asks patrons to rate their level of agreement about what they learned or gained after attending a service or program on a five-point Likert scale, ranging from “strongly agree” to “strongly disagree,” with an option of “not applicable” (see table 3).

Field Test Findings

In 2014, the Project Outcome Task Force field-tested the surveys with twelve libraries and a total of twenty-seven locations. Each participating library was encouraged to use the surveys according to their needs. Some chose to enroll only a few test libraries within their systems. Others chose to test one or several survey topics. Still, others chose to measure a single program or a program series. Libraries responded very favorably to being able to choose the surveys and methodology that worked best according to their needs and objectives. A follow-up survey was distributed to the libraries after the testing phase and the findings are summarized below.

Patrons Willing to Help Out

Overall, pilot testers received a positive response from patrons when they were asked to complete the surveys. Some appreciated and wanted to give feedback on services. Houston Public Library tested two programs—a business competition and toddler storytime—and found in both instances that patrons were eager to participate and provide feedback for each one even though the topics and audiences were widely different. Both patrons and libraries found the limited length and specificity of the surveys valuable, especially patrons with kids who shared their appreciation for its shortness.

Champions Drive Success

With many effective initiatives, a champion helps drive successful implementation and system-wide adoption. As expected, libraries that had champions leading the effort had an easier time enlisting staff to help with testing the surveys. And if the champion is upper-level library management, engagement and participation by other staff is broader and deeper. Finally, response rates tended to be better when surveys were administered by committed staff and not volunteers.

Wide Range of Libraries in Pilot Test

Pilot-test libraries consisted of a wide range of public libraries as far as size, location, and level of outcome measurement experience. Most of these test libraries reported that they were able to use the surveys to meet their testing purposes. As an example, Holbrook Public Library in Arizona is a small library with only 2.5 staff serving a population of about 5,000 people. They surveyed six events and received a 31 percent response rate for their 120 attendees. While they did not have capacity to test as many events as they originally intended, they still found the survey responses to be valuable—affirming that the services they offered were meeting community needs.

Quantitative and Qualitative Data Were Valuable

Participating libraries were able to use the resulting data in board presentations, branch activity reports, and to underscore advocacy stories. Skokie (Ill.) Public Library (SPL) found the quantitative ratings of the surveys used after a business class helped them understand whether they were teaching useful information to patrons. Arizona libraries—participating at the request of their state library systems—found high scores to have a reinforcing effect on the good work they already do. The open-ended questions at the end of the survey were valuable, as well. SPL staff also found that the survey’s comments helped them understand why people choose to attend online business classes and what they want to get out of the session. The feedback they received will help them make the class more effective compared to only knowing attendance prior to the assessment.

Shift Toward Outcome Measurement

Prior to the launch, PLA staff worked with ORS Impact, a research consulting firm, to interview libraries about Project Outcome and outcome measurement in order to help guide content development. They conducted twelve interviews with library leaders, who, in the 2014 Public Library Data Service Survey, had expressed a high level of readiness and interest in outcome measurement.² The libraries represented a variety of size and geographic locations. The findings reinforced what the Task Force learned in their pilot testing: the library field is poised to make an industry-wide shift toward outcome measurement. The interviews revealed:

• Library leaders believe the field will benefit from having access to the types of resources Project Outcome is developing, particularly outcome surveys and best practices related to survey administration, data entry and analysis, and data-sharing partnerships.
• Engagement in outcome measurement at libraries is often part of broader organizational orientations toward outcome-based thinking and the library’s role in addressing community priorities.
• Data and outcome savvy libraries “lead by example” in their communities, gaining the esteem of stakeholders to whom they are accountable and attracting community partners.
• Many library leaders engaged in outcome-based measurement have successful data-sharing relationships with aligned community partners, particularly school systems. They also believe they and others can benefit from learning about different ways to partner for the purpose of data-sharing and different types of data sharing arrangements.
• Understanding what constitutes an outcome may be “fundamental” or “basic,” but it is also quite challenging.
• There is an appetite among data savvy library leaders to share their experiences with other libraries, either as part of learning communities or via efforts to cultivate more interest in outcome measurement, in the library field.
• Libraries on the leading edge look outside of the library field for information and partners to support their measurement and use of outcome data.³

Project Outcome Benefits All Libraries

Any library—one that has been collecting outcomes for years or just getting started—can benefit from participating in Project Outcome. Libraries in the beginning stages of measuring outcomes will have “Getting Started” resources available to them once they sign up on www.projectoutcome.org.
Mid-level enrollees—those who may have used some outcome measures in the past but have not operationalized them in their work—will have access to one central data entry and storage system with ready-to-use surveys that can take them from the occasional outcome measurer to adopting it as a regular practice. And finally, a champion user—someone who measures as a standard—can benefit from the webinars and other resources made available by Project Outcome that will examine topics like how to take data-sharing partnerships to the next level. Regardless of experience level, libraries have free access to all of these resources.

At all levels, Project Outcome offers a few distinct advantages over a standalone measurement system. It providesunique data visualizations of both the library-specific data and aggregate peer-level data for libraries to use to help build upon their own story. It also offers a comprehensive set of tools with regularly updated informational and training content based on the learnings of its users. It relies on peer-level engagement to develop content instead of a top-down approach to learning. The more libraries participate in Project Outcome the better it becomes.

References

1. Carolyn Anthony, “New Measures for a New Era,” Public Libraries Online, Aug. 21, 2013, accessed Apr. 30, 2015.
2. 2014 Public Library Data Service Statistical Report Special Section: Performance Measures, accessed July 27, 2015.
3. Interviews conducted via phone by ORS Impact with twelve library leaders in March 2015.

Further Information

Visit www.projectoutcome.org for more information or to sign up for the service. Information on webinars and other related events can also be found on the website.

---

Tags: Impact Survey, outcomes, project outcome, strategic goals