Iterate and Assess with Community
Overview
During the crises of 2020, most library staff we worked with quickly pivoted to virtual programs or continued current services with minor modifications (i.e. curbside pick-ups of materials, to-go kits, etc.). In many cases, the in-person programs shifted to a virtual platform, without determining if the programs were needed by community members or if modifications were needed to fulfill crisis-related needs. To design services that are community-centered during crises, library staff need to assess relationships with community members and iterate services based on those relationships. We learned that at crisis times:
- In instances where relationships with the community are frail, staff struggle to iterate existing programs to meet crisis-related needs, assess outcomes, and rethink offerings. This resulted in programs and services that did not respond to what community members truly needed during crisis. In these instances, we strongly recommend that library staff learn and co-create with their community (see Co-Create with Community essential tasks). We recommend using the outcomes articulated in your logic model to create or use readily available assessment tools that best measure the impact of your designed program or services.
- In instances where crises imposed additional challenges to the implementation of existing programs that fulfill the needs of the community, staff determined how to overcome these challenges through short interviews with their community assets, whom they already had successful relationships with, and continued to move towards the intended outcomes of the programs. We recommend this approach when circumstances are similar.
- In instances where strong relationships were already in place (see essential task: Learn with Community), staff leveraged community assets and consulted them on any imminent changes that needed to be made to existing programs and services. Program iterations happened organically as a result of short interviews, observations, and self-assessments. We recommend this approach when such strong relationships exist with the communities that you serve.
Regardless of where on the above continuum your library falls, it is crucial to measure the collective impact that you and your community partners have in your community. To achieve this, the assessment tools you select must align with the outcomes you articulated in your logic model.
Assessment methods must enable future decision making related to the design of crisis responsive programs and services. When aligning assessments to outcomes consider the following:
Once the data is collected, it must be analyzed and used to decide how to iterate existing programs/services, to determine if current programs/services are right for the community, or to inform the creation of new services. As with all other aspects of the work described in this Field Guide, you should not evaluate and iterate on your own. Work with your partners and community members to look into the work you’ve been doing and to make decisions about next steps. Share the impact story (if any) using video, blog posts, or other visualizations that demonstrate the contributions that were collectively made with the community, paying special attention to crisis-related needs that were fulfilled by the designed programs and services.
Tools
The Connected Learning Through Libraries project team published an Evaluating Connected Learning in Library Programming guide that is useful as staff evaluate and iterate their programs and services. This guide introduces the terms used in evaluation, the processes involved in evaluation (including goal setting and planning), when to administer which assessment tool, a portfolio of tools just right for measuring connected learning outcomes, and samples of data analysis. The team has also published specific case studies of evaluation with small and rural public libraries.
For the past few years, Subramaniam has been collecting instruments (with permission) that have been used to measure various learning outcomes. Library staff are welcome to use these assessment tools as a guide to creating their own customized assessment tools based on their intended outcomes.
Essential Task in Practice
In communities where needs during the 2020 crises could not be immediately identified by library staff, some of our participants engaged in learning and co-designing with their community. For example, at the York County Libraries (PA), staff learned about the ways in which teens could bring their interests and engagement to Black Lives Matter (BLM) activism. Through learning and co-designing with teens, staff engaged with adolescents to support participation in local BLM activities.At the Seattle Public Library, through pre-existing strong partnerships with the University of Washington, library staff continued through the crises of 2020 with KidsTeam, a program where children co-design technologies and learning experiences with library staff, peers, and UW graduate students and faculty members. The program successfully continued virtually because of the quality and vitality of the partnerships that were already in place and the need for after-school programs in the community. When the program shifted to a virtual platform, challenges did arise. This included youth’s inability to draw their designs in the chosen online platform while participating in a drawing activity. Through observation, interviews, and artifact analysis, program facilitators iterated activities accordingly (i.e. using the sharpie tool to draw online instead of the small pen tool). The iterations were crucial and helped alleviate challenges in the virtual environment.
The For Freedoms program was in full swing at the Waltham Public Library (MA) at the time of the pandemic. For Freedoms empowers youth to be civically engaged through the creation of “Freedom for….” or “Freedom in…” or “Freedom of…” or “Freedom from…” signs that are placed in front of the library and the City Hall. This program allows teens to have a voice, even though they cannot vote. Prior to the crises of 2020, the program had solid support from partners including funders, art organizations, and local public schools. This support continued throughout 2020. As schools were closed, library staff shifted from visiting school cafeterias at lunchtime to Google Meet “school visits” to connect with teens to demonstrate how to create the signs.
The library staff produced a short video (see below) which was shown during these virtual visits. The video answers potential project questions including how to create For Freedom signs and why creating a sign is a valuable activity. Through help from local teachers, library staff printed 240 teen designed signs which, as in previous years, were placed in front of the library and City Hall. The 2020 crises gave library staff the chance to try new engagement techniques, including the video, which will continue in 2021. Staff learned, through reflection and analysis of interviews with teachers and teens, how to build on the For Freedoms programming and continue to iterate it to serve teens successfully.
Do It With Others [DIWO]*
- Identify what iteration or assessment is needed during a crisis based on pre-existing community relationships and what you know, through those relationships, about the community. This step will help you to determine the starting point – initial co-design or iteration.
- Regardless of the starting point, create new or select readily available assessment tools that best measure the intended outcomes of your designed program or services.
- Collect data using the selected tool(s) at the predetermined intervals or times, as needed to measure the intended outcomes.
- Analyze data with community members and use it to iterate the existing program/service or create new ones.
- Share the impact story (if any) using video, blog posts, or other visualizations that demonstrate the contributions that were collectively made with the community.
*We use the phrase Do It With Others (DIWO) instead of Do it Yourself (DIY) which we derived from the maker movement vernacular (see Maravilhas & Martin, 2017) to reflect the nature of these steps that emphasize the focus on community and collectivism, further calling library staff to embrace the public servant mindset.
References
Maravilhas, S., Martins., J. (2017). Tacit Knowledge in maker Spaces and FAB LABs: From DO IT YOURSELF (DIY) to DO IT WITH OTHERS (DIWO). Handbook of Research on Tacit Knowledge Management for Organizational Success. Available at: https://www.researchgate.net/publication/316091165_Tacit_Knowledge_in_Maker_Spaces_and_FAB_LABS_From_DO_IT_YOURSELF_DIY_to_DO_IT_With_OTHERS_DIWO