OTI Releases Philadelphia BTOP Evaluation Instruments

The Open Technology Institute (OTI) and its partners in the Freedom Rings Partnership (FRP) in Philadelphia are pleased to release for public use a comprehensive set of program evaluation instruments designed to gauge the impact of broadband adoption and digital literacy programs. These instruments have been approved by two independent Institutional Review Boards (IRB) as acceptable protocols for use by the FRP in Philadelphia. OTI and its partners developed these tools with funding from a Broadband Technology Opportunities Program (BTOP) grant awarded by the National Telecommunications and Information Administration (NTIA), U.S. Department of Commerce.

OTI shares these instruments with other BTOP grantees in mind - both to aid in their development of evaluation tools and towards the goal of establishing common indicators across these programs nationally. In Philadelphia, OTI uses these instruments to assess the impact of the BTOP program on three major types of stakeholders: community members who use the BTOP programs, PCC facility assistants and SBA course instructors hired through the program, and the Philadelphia Freedom Rings Partnership staff.  Specifically, for community members utilizing the programs, we are exploring the impact of the program on participants' adoption, employment, education, and community engagement. 

The instruments represent a variety of quantitative and qualitative data collection methods. On the quantitative side, we are employing two standardized surveys for use across all Philadelphia BTOP-funded programs. On the qualitative front, we are using both focus group scripts and interview guides to collect success stories, challenges, and best practices. Below, please see more description for each instrument.

• Workstation User Survey (WUS): A brief survey collecting demographic information and some adoption questions, administered electronically at organizations providing PCC access and/or digital literacy trainings. The short WUS is collected on an ongoing basis while we will randomly deploy the long WUS for a week at a time. Available in English and Spanish. 
• Standardized Exit Survey (SES): A brief survey collecting participant outcome information, administered electronically after participants complete an NTIA-defined digital literacy training (classroom-based, instructor-led). Available in English and Spanish.
• Focus Group Scripts- only available in English. All scripts should be administered in-person by a trained moderator.
  • Participant cohort script: A series of open-ended questions for a cohort of BTOP participants, with the goal of tracking change/progress of the participants during the course of multiple discussions. Questions ask about participants' experiences of the programs, impact of the program on adoption of technologies and Internet, any changes to their employment/education status, and community engagement.
  • Participant non-cohort script: Similar to the participant cohort script, this script condenses the questions from multiple discussions for use in a single focus group session. 
  • PCC Facility Assistants and SBA Course Instructors cohort script: A series of open-ended questions for PCC facility assistants (FAs) and SBA course instructors (CIs), focused on their professional development experiences and growth. Administered during the course of multiple discussions with a cohort group of FAs and CIs. 
  • PCC Facility Assistants and SBA Course Instructors cohort script: A series of open-ended questions for PCC facility assistants (FA) and SBA course instructors (CI), focused on their observations of the impact of BTOP on participants. Administered during the course of multiple discussions with a cohort group of FA and CIs. 
• Interview Guides- only available in English. All scripts should be administered in-person by a trained interviewer.
  • BTOP Partnership staff interview guide: A set of open-ended questions for Philadelphia BTOP staff, focused on their experiences of the Partnership and the program. 
  • Allied organizations staff interview guide: A set of open-ended questions for organizations not funded by BTOP but working in partnership with Philadelphia BTOP grantees, focused on their impressions of the program and ideas for future partnerships. 

We welcome collaborations with other interested BTOP evaluators to analyze the data made available through the shared indicators. For questions or to contact us on this issue, please email oti.fieldops.research@newamerica.net.

July 12, 2012: We have amended this post slightly to emphasize that these tools were developed solely by OTI and its partners in the Philadelphia Freedom Rings Partnership (FRP).