Broadband Technology Opportunities Program (BTOP) Evaluation: Is a story worth a thousand numbers?

Now that the rush toward implementation of BTOP-funded public computer centers and digital literacy programs around the country is turning into a rush to make long-term sustainability plans, partners are looking for effective program evaluation strategies. As it turns out, establishing community-driven metrics is proving to be a challenge.

In November, OTI's monthly call with partners revealed that some BTOP grantees feel that the kind of data that best displays the impact of the program is qualitative and anecdotal: real stories about real people. Grantees pointed out that measuring broadband subscription numbers, on the other hand, fails to show nuance in how broadband access impacts people’s lives -- by giving them opportunities to use computers to apply for jobs, organize community events, or talk to loved ones far away – regardless of whether or not they have enough disposable income to sign up for a subscription.

Community-wide economic improvements are hard to document, and require deep research that may be difficult for BTOP partners to perform in the midst of program implementation or launching comprehensive evaluation efforts. On the other hand, this phase of BTOP program roll-out is a great time to start documenting the nuances in broadband adoption so that eventually communities can develop more appropriate metrics for evaluation.

One example of a program with enormous potential benefits that may not be easy to measure in the short term was described by one of the BTOP grantees on the call. In order to expand the benefits of digital connectivity, Neighborhood Learning Alliance’s Pittsburgh CONNECTS works with a local hospital to webcast their public sessions so that the community can be better informed about health-care information. These kinds of practices help prove how BTOP positively impacts community anchor institutions and other service providers in a community. What’s needed now is a standard way to document and measure the effects of interventions like this.

The BTOP partners that take part in this monthly call are talking about how this is a good time to start sharing tips, tools and stories with each other: what works, what doesn't work, what they are finding about the impact of technology and adoption on their communities. One of the great benefits of the BTOP program has been the sheer number and diversity of digital inclusion programs; it's a great time to explore the ecosystem we're creating, and to honor the range of innovations emerging across the country.

OTI invites grantees to share with us and each other via our monthly call, or to write to us with stories, ideas, or updates. To join the monthly calls, contact: losey@newamerica.net