Project Outcome Results in Action
SAMANTHA LOPEZ is Project Coordinator for the Public Library Association. Contact Samantha at slopez@ala.org. Samantha is currently reading The Night Circus by Erin Morgenstern.
PLA commissioned ORS Impact to interview participating libraries and community stakeholders to create five success case studies for its performance measurement initiative, Project Outcome (PO). PO is a free online toolkit designed to help public libraries understand and share the impact of essential library programs and services by providing simple surveys and an easy-to-use process for measuring and analyzing outcomes. What PLA learned from the case studies is that, even with limited survey responses, libraries are able to leverage their outcome data into actionable results. By using Project Outcome surveys, libraries are tracking their impact across time; improving and expanding programs and services to meet community needs; supporting new and deepen- ing existing partnerships; and increasing library championship.
Using Project Outcome with Summer Library Program to Track Impact Across Time and Strengthen Championship
Appleton (WI) Public Library (APL) first experienced the value of measuring outcomes through their ongoing use of the Impact Survey. Library leadership viewed Project Outcome as an opportunity to extend outcome measurement to their summer library program and support program improvement and communications with the library’s board.
Data-Driven Changes Strengthen Summer Library Program, and Evidence of Patron Outcomes Supports Library Championship
APL offers a range of activities for children, teens, and adults as part of its summer library program. Patrons can earn rewards through an online component by completing missions such as visiting a local nature preserve or the police department. These missions help patrons build connections with community resources and other community members through informal interactions. Project Outcome surveys conducted in 2016 showed that patrons benefited from the program, but found some of the missions confusing and hard to follow. APL has since improved the descriptions and layout of the missions, and they look forward to reviewing 2017 survey results.
APL reports on the impacts of library programs and services to its board based on the results of Project Outcome surveys and Impact Surveys. Library leaders have found that outcomes resonate with board members, strengthening how they voice their support for the library. One leader shared,
I think [reporting on outcomes] really transformed conversations with our board, so that they’re much more interested . . . It lets us tell a more complete story about the library, and what we’re offering . . . so they can understand it more fully . . . When you talk strictly in numbers, or outputs, it’s not something that people can hold on to as clearly. And so when you start talking about outcomes and the impact that a library is having, that’s where the true heart of what we’re doing is. And our board feels that, as well . . . It allows them to . . . have something more concrete to hold on to, and to talk to people about what the library does in a more complete way than “X number of people walk in the door every year” or “X number of books walk out.”
A board member shared, “The [outcome] data provides an objective story, backing up much more engaging stories from staff about serving the community with objective numbers . . . [and] includes things the board may not think to ask for. This adds dimensions to how the board considers the library’s success in serving the community.”
Survey Administration Adjustments Improve Response Rates
Getting patrons to fill out surveys has been a challenge for the library. When APL first started using Project Outcome surveys in 2015 for their summer library program, they made the surveys available only on paper; in 2016, they made them available online. Each set of surveys produced valuable information, but fewer than twenty surveys were completed each year, and APL wanted more responses to better understand how to improve their programs and be more confident in their assessment of impact.
For 2017, APL is using some new strategies to increase online survey completion. For example, when children come in to get their prizes for completing the summer library program, the adult accompanying them receives a slip of paper encouraging them to take the survey. The library has also dedicated a computer near the front of the library to survey-taking and staffed it with a librarian who encourages patrons to complete the survey; and “boosted” a Facebook post about completing the survey that appears on the news feeds of everyone who “likes” the library’s Facebook page. The preliminary results of this effort have been positive, with more than fifty surveys completed so far.
Success Factors
APL was an early adopter of the Impact Survey. Their experience with administering the survey laid the groundwork for Project Outcome’s traction in the library, as staff were already familiar with the value of using outcome data to explain their impact. As a library leader explained,
We’d already done the Impact Survey, and so [the staff] understood the importance of moving to outcomes rather than outputs . . . With our [traditional output] numbers sliding, it tells a much better and much more complete story for us than our disappointing numbers were for several years . . . I think it would’ve been different if our numbers were going up. [The Project Outcome surveys] could have been seen as criticism. And instead, with our numbers going down, it was seen as a way for them to tell the story of what the library was actually doing.
Several APL staff members took part in trainings offered by PLA, which helped build familiarity with Project Outcome and led to their early use of the tools in 2015. APL staff appreciated that Project Outcome offered ready-to-use surveys, which the library did not have the expertise to create in-house.
What’s Next?
APL plans to review this year’s summer library program survey results to assess the effectiveness of their efforts to reach and meet the needs of more economically diverse residents. These efforts were part of the library’s participation in POINT (Poverty Outcome and Improvement Network Team), a local collaborative initiative. New Project Outcome survey functionality allowing them to add custom survey questions specific to this effort was a timely addition. A leader shared, “We were able to add questions about the pieces [specific to] the POINT initiative . . . I’m really glad to have that flexibility. So when we’re doing something different, or something where we’re trying to target a population, we can start to look at those factors.”
Project Outcome is growing in importance in the library’s work. APL will continue to use the data to help improve their summer library program, strengthen their community impact, and provide their board with a comprehensive picture of their value in the community. APL will begin reporting data from Project Outcome surveys in their budget requests to the City of Appleton in 2018, when they will have three years’ worth of data, as required by the city.
Using Project Outcome with Summer Reading and Digital Literacy Programs to Support Partnership Development and Expand Services at a Small Library
Burnsville (WV) Public Library (BPL) is an important community anchor within the rural county it serves. BPL has used Project Outcome surveys to better understand the impact of the library’s programs, and has developed new partnerships and designed new programs based on community input.
Summer Reading Program Survey Results Support a New Partnership and Enhanced Programming for Children
Project Outcome surveys showed that caregivers of young participants in BPL’s summer reading program wanted tutoring and extra help for their children. Equipped with this information and evidence of program impact, BPL worked with the local school district to have two teachers offer tutoring at the library the following summer, for three days each week. A library staff person whose two children participated in the program shared, “[The children] work on reading, math, they get on the computer . . . It’s really benefiting them. And some of the kids are here because they need the extra help, and some of them are here just to try to beat the summer slide. It’s working very well.”
BPL also started a new after-school program because surveys showed an appetite for additional programming for children. Children shared that they wanted to learn more about science and technology, so one of the after-school classes will include hands-on Science, Technology, Engineering, Arts, and Mathematics (STEAM) activities.
Survey Data Helps Secure Technology Grant
Project Outcome surveys showed that a basic digital literacy skills class at the library helped adult participants feel more confident using technology and become more active email users. Participants also shared their desire for access to better technology. This feedback from Project Outcome surveys helped BPL secure a technology grant from the West Virginia Library Commission, and next fall the library’s public computer lab will be upgraded and expanded.
Success Factors
In a prior job in the nonprofit sector, BPL’s new director gained experience in the collection and use of outcome data and was excited when she discovered Project Outcome. Since adopting Project Outcome, BPL’s board has been interested in and supportive of the library’s use of Project Outcome. As one board member told us, “The reason we do it is we are trying to find out the interests of the community so we can serve the community better. Any library can use it.”
What’s Next?
BPL is committed to an outcomes-based approach to measuring the impact of their programs, and learning how to improve them. BPL recently included survey data in a grant request to support its new after-school program, and they discussed survey results at a recent town meeting. BPL will continue using Project Outcome to understand impact and community needs, and communicate the value of the library’s programs and services with various audiences, including town council members. A library leader shared,
It seems like [town council members’] support is a little more evident since we actually have the data to back up what we’re claiming, rather than just saying, “Well, we had 80 kids here this summer. . . ” We can say, “Well, we had 80 kids this summer, but we also asked them what they would want to change, and this is what they said.” . . . It offers some validation . . . it’s more transparent, [now] that we actually have the data to back up what we’re saying. They haven’t increased our funding or anything . . . but they are more vocal and supportive. They have given us a couple of letters of support for different grants that we’ve tried to obtain. And I don’t think that would’ve happened without . . . the knowledge that we are doing the outcome measures.
Using Project Outcome with Business Development and Job Skills Programs to Deepen Partnerships, Improve Services, and Increase Library Championship
For eight months, Pima County (AZ) Public Library (PCPL) has been using Project Outcome surveys to assess the workshops, classes, trainings, and drop-in sessions they provide in the areas of business development, job skills, and digital literacy.
Data-Driven Decision-Making Leads to Deepened Business Development Partnership, Increased Efficiency in Service Provision, and Increased Access and Impact among Community Members
Based on survey results, PCPL has added components to business development services, changed how they offer services, and, in one case, decided to eliminate a service. One program manager explained,
It’s not just a matter of measuring attendees, but in measuring the effectiveness, or the immediate impact that it has on patrons that attend these workshops. Some of these workshops. . . were well-attended, but the feedback was such that we no longer offer [them]. It’s a matter of making sure that it’s not just getting people in the door, but that people feel like it’s worth their time, and it’s something that they can use. That when they leave . . . they’re in a position where they feel they’re more comfortable with whatever the topic was.
Survey data has also led to changes in how the library works with its partners. As a project coordinator explained, “What Project Outcome has helped us do is be more strategic in how we use . . . partnerships.” One example involved a drop-in service for adults interested in starting a small business. Through the Project Outcome surveys, the library learned that patrons wanted a more structured learning opportunity in addition to one-on-one help. Library leadership approached an existing nonprofit partner, SCORE (Service Corps of Retired Executives), and, together, PCPL and SCORE decided library staff would teach content from the first of five SCORE foundational business workshops during the library’s drop-in hours. As a result, more community members are now accessing a continuum of services that match their evolving needs.
PCPL and SCORE co-market their respective business development learning opportunities; SCORE regularly refers people to the library, where they get the foundational training and access to a wide array of business development resources; and the library regularly refers people to SCORE when they are ready for mentoring on more advanced business development topics. In this way, the collaboration has helped SCORE with the problem of people going to them for a first appointment and never coming back, which helps SCORE “keep our volunteer mentors happy—otherwise they won’t want to do this stuff, because we do it for free . . . Our mentors like nice, meaty business problems, that they work with people who are enthusiastic.”
In this mutually beneficial collaboration, SCORE provides the library with materials, coaching, and occasional in-person support, and the library frees up SCORE mentors to provide higher-level training to patrons who have an increased level of readiness for it. Each partner is doing what they do best—maximizing the efficiency, accessibility, and impact of the services they provide. A vice president at SCORE explained,
What I’ve heard from people . . . is that [first session is] incredibly important for them…in a couple of ways. For the people that . . . decide that they are going to go out and build a new business, it gives them an understanding of the questions they need to answer and an understanding of the process that they can potentially use to gain those answers, and to develop their business plan. And then equally important on the other side. . . is that they go to that meeting and they decided “Wow, that’s not going to be nearly as easy as I expected.” Or the risks are going to be a lot greater . . . They might have to use their personal resources and are not willing to do that, and they decide not to create a business. And that’s just as valuable . . . Because the worst thing you can do is send somebody down the road on a path that they’re going to fail and potentially eat up all their personal resources in the process.”
Sharing Evidence of Business Services and Job Skills Program Impact Leads to Increased Championship of the Library and a Deepened Job Skills and Economic Development Partnership
Pima County’s Economic Development Plan includes PCPL’s contribution to “human infrastructure” through support of workforce development and small business creation and growth. In their quarterly updates to the county, as well as their recent annual report, PCPL included evidence of outcomes and anecdotes provided through Project Outcome surveys.
One library leader believes these data points contributed to “the county manager and varied department heads . . . seeing the library’s got it going on . . . The library does know what they’re doing . . . ” which, in turn, led the county administrator to refer the Pima County One-Stop Career Center to the library when the One-Stop asked the county for additional money. The library leader explained,
The county administrator said to them, “Well, are you doing any of that cool stuff like that’s happening at the library? If you’re not, maybe you need to be partnering with them in order to figure out how to make this work, and then I’ll consider giving you some money.” So right there, just the fact that we’re able to inform all of this reporting, and that the administrator is aware of it, now departmentally, we are interacting better.
The result is a new collaborative effort between the One-Stop and PCPL to provide youth with work experience, as well as college and career readiness programming. One-Stop youth participate in Teen 365 programming at the library, which helps teens develop skills, connections, and opportunities to create a successful and happy adulthood. The year-round program addresses all aspects of teen development, from academic goals to social and personal interests, and, combined with the One-Stop’s efforts to provide employability skills training, helps create pathways to college, career, and entrepreneurship.
A manager at the One-Stop shared,
We complement each other. They have strengths, and we have our strengths. We are highly connected with industry, with business, with folks that are actually making hiring decisions . . . The library has identified a strength that we are looking at piggy-backing on: technology. Youth today use LinkedIn, social media, they use technology to communicate effectively, and a lot of the employers are doing the same thing. We here at the county are still behind the eight-ball as far as . . . utilizing social media, technology, videos, YouTube. We are utilizing [the library’s] strengths.
Success Factors
Strong leadership and an existing appreciation of the importance of measuring library impact played a big role in PCPL’s success using Project Outcome. One leader got buy-in from another leader who “started talking about it, now our administrators see the value in it. And because of that, because we continually pitched, ‘Hey, we need to do this. Look: it’s just demonstrating how great . . . our library is, but also the ease of use. You’re able to just do this within the context of what you do every day already.’” Getting key support from administrators was partly due to being good spokespeople for the importance of measuring outcomes, and partly due to demonstrating the utility of outcome measurement by showing them the results of one of the surveys: “I think the reason that [the administrators] saw the value [of using Project Outcome] is because I just used it . . . We collected the data, and then I ran one of the reports. And that’s all it took, was for them to see how great that report looked, and the information that we got from it, and they were there.”
What’s Next?
PCPL is currently building staff capacity to scale up use of Project Outcome throughout the system. The library has coordinated a training on Project Outcome with library program instructors who are spread across its twenty-six branches so they can begin to measure impacts across a broader range of programs and services, and so the trained instructors can serve as point people for other staff at each branch. Library leaders spoke about the importance of starting small:
If we would have rolled it out everywhere with every program, the amount of data that we would have collected in that short time would have been very difficult to synthesize and [it would have been difficult to] figure out how best to use and implement changes. So now that we have our working model that’s scalable, we’re taking this forward, moving throughout the system.
Using Project Outcome with Storytime and Teen Programs to Improve Programming and Better Meet Community Needs
Plano (TX) Public Library (PPL) uses Project Outcome surveys with a wide range of programs and services so they can better meet the needs of their highly diverse community and strengthen outcomes for patrons. Serving children, teens, and families is a core part of their programming, and a large part of the library system’s outcome measurement has focused on this population.
Data Drives the Creation of New Storytime Programs, which Broaden and Deepen Community Access to Early Literacy and Social Skill-Building
After a month of participating in a library storytime program (including Babes in Arms, Toddler Time, Rhyme Time, and Preschool), PPL administered Project Outcome’s Early Childhood Literacy survey to gather basic feedback from caregivers and learn if they were experiencing the intended gains in support of children’s basic literacy skills. In addition to providing evidence of positive outcomes, some caregivers reported their children had challenges being in such large groups due to sensory issues. As a result, the library decided to offer “sensory storytimes” at two of their branches with a smaller group size of ten caregivers and children. A library leader explained, “We have gotten some incredible feedback from our parents about how we are helping their children by adding this program, and how their children are better able to interact with other children.”
PPL also learned from the surveys that caregivers really enjoy programming that includes books in different languages. In response, PPL added a new series called Storytime Around the World, in which library staff read books in Arabic, Chinese, French, and Spanish in three-week cycles. The added programming has attracted both new families as well as prior storytime participants, and staff has observed that families often stay to use other library resources. Excitement about the new series has activated patrons to spread the word. A participant in a Chinese storytime shared the event information on a Chinese community Facebook page, which brought in many new people who had not previously known about the library’s Chinese language materials. Consistent with these observations, PPL has seen an uptick in circulation of materials in the languages featured in the new series. A staff member explained,
Whether or not . . . they’re new library users, we’ve definitely seen them connecting with the library and library resources in a way that they haven’t before. I think at the Maribelle M. Davis Library where they had the Story Time Around the World series, the circulation of [the books in the Junior World Language collection] doubled within about a year.
Data Drives Improvements in Science, Technology, Engineering, Arts, and Mathematics (STEAM) Services and in Community Awareness of all Library Programming
PPL’s teen outreach program brings technology and related activities to local schools, and PPL used Project Outcome’s Digital Learning survey to measure impact and get feedback from the teens. Teens shared how much they liked the technology, so PPL created STEAM robotic kits designed specifically for teens. “And all of them are checked out and on hold and not available!” a staff member shared. PPL also put Project Outcome Early Childhood Literacy surveys inside STEAM kits for all ages. The results showed that parents and children were using the kits to learn something new together and parents really appreciated how the kits supported interaction with their children. The surveys also revealed that many parents were not aware of the STEAM programs offered in the library, so PPL published a brochure describing the library’s programming to be distributed three times a year and included it in the STEAM kits. “And a third to half of those brochures don’t come back in the kits… So [ultimately, the surveys] helped us to find a different way to relate to the patrons in the library,” a library leader explained.
Data Guides and Helps Procure Funding for Arts Programming
PPL used the Project Outcome Early Childhood Literacy survey to measure the impact of an arts program for preschoolers and to get feedback from caregivers. After learning that patrons would also like a version of the program for teens, the library used the survey results to get funding for the expansion and fulfill grant reporting requirements. A staff member explained,
[Caregivers shared] how beneficial the program was, how they’re doing projects they wouldn’t have thought of doing at home . . . using supplies that they don’t have at home . . . So having that information, I could write a grant that shows “This is what the community wants, and we want to provide it.” And then, I also use the survey [results] to report during the grant period, talking about what the patrons liked most about the service, or programs . . . and what else they’d like to see the library doing.
Based on survey results, PPL also added more information about artists and techniques to the program curriculum. A staff member shared that a patron said “their child wouldn’t have had an opportunity to do art without the library.”
Success Factors
PPL has one library leader who manages the survey work, including deciding when to conduct Project Outcome surveys so that patrons are not over-surveyed.
In addition, PPL has an organizational culture that supports the use of outcome measurement. The library takes part in the Edge Initiative, and staff are invested in using tools that help them understand community needs and tell the library’s story in different ways to different audiences.
What’s Next?
PPL plans to continue using Project Outcome surveys and reaping the benefits of such use, in the form of better-informed decision-making, adaptive programming, and stronger, more widely experienced impacts for their community members. PPL will also continue communicating the value of the library in monthly reports to the city leaders, including the deputy city manager and department heads.
Using Project Outcome with Digital Literacy Programs and Online Training Services to Improve Programming, Inform Resource Investment, and Sustain Partnership
Thomas Crane (Mass.) Public Library (TCPL) uses Project Outcome surveys to measure its impact and ensure that its programs are always improving and meeting the needs of an increasingly diverse community.
Data-Driven Changes Strengthen Digital Literacy Program, and Evidence of Patron Outcomes Helps Improve and Sustain an Important Community Partnership
Saheli is a nonprofit partner of TCPL that primarily serves South Asian women and families. At the library, Saheli provides digital literacy training to diverse groups of community members. Participants in the training complete Project Outcome surveys either midway through the course or at the end. One example of data-driven change is that after reviewing mid-course survey responses, the Saheli trainer added social media lessons to the training. A Saheli trainer explained,
Quincy was new for us. And we really wanted to . . . get in touch with the South Asian population . . . try to find out their needs, and how to incorporate them with the mainstream population. . . [There are] a lot of families [who] want someplace to go out and connect with other people . . . they want to be in places where there are other South Asians they can talk to. But there’s always this gap of not being able to connect with other people outside, and. . . that barrier opens up when you’re in a classroom like this, where you’re talking about not just academic or work-related stuff. . . The last year and a half, we did social media based on the surveys . . . and people started inviting each other to . . . online groups, and started connecting that way!
The trainer also learned that one participant started her own business using new connections she made through the class.
Saheli made other data-driven improvements to the program, including changing the time of the class, adding lab hours, and clarifying program expectations at the beginning of each class, which supported program retention. “Project Outcome lets us be real about the experience, and tweak it in real time, so we can improve it. Otherwise, we would have no other knowledge, or means to improve,” said a library staff person that works with Saheli. A Saheli trainer said Project Outcome surveys tell her “a lot about what the community needs, what each person’s needs are, what else we could bring in, as far as technology is concerned, which is important as technology is changing pretty much every day now. The amazing things about these classes is that people from so many different levels and backgrounds come in, and they sit down together in the class, and have a conversation.”
The positive outcomes reported by Saheli’s digital literacy participants helped TCPL decide to continue working with Saheli and successfully pursue funding for the continued partnership. As explained by a library leader, “It was information from Project Outcome that gave us the confidence to ask the Friends [to fund the program], and to justify asking for that money.”
Evidence of Patron Outcomes from Online Training Services Support Library Decision-Making about Resource Investment
Project Outcome survey results also informed TCPL’s decision to continue their subscription to Lynda, an online training service that patrons can access through the library. TCPL uses Lynda’s metrics to understand how often patrons use the service, but TCPL felt the investment in the subscription also needed to generate meaningful outcomes. Using email addresses Lynda users provided when they registered for a Lynda account, TCPL sent invitations to take the Project Outcome survey online. The results showed that patrons value the service and it helped them to develop their skills and knowledge across a range of topics.
Success Factors
TCPL’s successful use of Project Outcome surveys started with leadership support. The library’s assistant director appreciates the value of customer feedback and outcome assessment, and works to build a culture within the library that does the same. He shared,
I’ve actually designed a whole bunch of surveys just on my own . . . [to get] anonymous feedback most often, on different staff trainings we’ve done. And I’ve found that to be really valuable and . . . conducive to the kind of general culture of . . . “This is what we’ve heard, and this how we’re going to improve the next step.” I’ve found it really useful just to take this framework to apply in a lot of different arenas. And even though it’s not strictly Project Outcome, it’s certainly inspired by Project Outcome. And I hope that creates an environment that is conducive and supportive of doing more outcome-based assessment.
Further, he provides material support for staff use of Project Outcome by creating the surveys, helping to implement them, and discussing survey results. TCPL also attributes success to using a staged approach, which allowed staff to learn from successive survey administrations with several types of programs.
What’s Next?
TCPL will use Project Outcome surveys to assess progress on the goals in their new strategic plan during the next five years. One priority in the new strategic plan is to inspire curiosity and lifelong learning, and Project Outcome surveys will enable them to track the percentage of parents and caregivers who say they learned something new that they can share with their children at early literacy programs. A library staff leader noted, “Project Outcome is really just exploding here in the next several years.”
Editor’s note: PLA is dedicated to helping public libraries become data-driven organizations and will continue supporting libraries in outcomes measurement. To learn more about Project Outcome or to register for free, visit www.projectoutcome.org.
