Appendix: Testing

Testing had two distinct parts, messaging and tools. Messaging allowed us to dig deeper into insights around practical motivations and catalysts which we don’t often see reflected in current citizenship marketing campaigns. Tools allowed us to gauge reception to solutions that could alleviate barriers with the current citizenship process, particularly regarding the complexity and duration of the application.

We used a variety of testing methods to dig deeper into insights. Some of our exercises included quantitative measurements which were used as a subjective tool to understand LPRs’ perspectives and follow up on their range of interest. Even though scales are present in the testing, we represent these results broadly as “high marks” or “low marks” to indicate patterns in preference.

This portion of our research was done virtually with participants who had varying degrees of access to technology. We opted to use Google Slides as a widely accessible tool since some of our sessions were done via phone conferencing. Through slides, we introduced card sorting, rating exercises, a link to a Figma mobile clickable prototype for guided testing, and open-ended questions.

Our team wanted to learn about the preferences in citizenship marketing compared to the catalysts and motivations mentioned in their interview. To do this, we presented participants with a list of citizenship slogans which included a range of tones and motives. Some of the slogans were examples of common marketing approaches which appeal to voting, elections, or a general call to action. We also included slogans focused on strong motives and catalysts, such as security from anti-immigrant policies, security while traveling, traveling with a U.S. passport, and all benefits. With all 12 citizenship slogans on the screen, we asked each person to tell us which messages were motivating and why the message resonated with them. Additionally, if there were any messages they did not like, the participant was asked to note this.

As Seen from 18 Tests with Eligible or Recently Naturalized Participants:

ONLY HIGH MARKS

Chosen by most interviewees as motivating or highly motivating

MOSTLY HIGH MARKS

Chosen as motivating by many interviewees, and at least one participant reacting negatively towards it.

LOW MARKS

Viewed negatively by most interviewees

In addition to messaging content, we wanted to understand how citizenship marketing was received in different contexts in order to increase the reach of messaging. During interviews, we heard from many how citizenship is less of a priority than the Green Card and how general life events can delay residents who intend to naturalize. In order to get a sense of the environments and methods, we asked participants to rate (1 to 5) and discuss their comfort level of seeing citizenship messages in 13 different scenarios. Scenarios ranged from in-person to online within environments associated with strong catalysts.

All contexts presented:

• You recently traveled to see relatives outside of the U.S. and are now returning at the airport. While speaking to the U.S. customs officer, he mentions you're eligible for citizenship.
• It’s your second year as a resident. You are doing your U.S. taxes online with TurboTax and the tool gives you a message saying there are financial benefits to citizenship.
• You’re sending a package to relatives abroad and you see a poster at the Post Office about citizenship benefits and your receipt suggests a citizenship application tool.
• You’re visiting your primary care doctor. While reviewing your insurance information, a person from her staff asks you if you’ve considered getting your citizenship.
• You’re applying for state health insurance online. After submitting your application, the site suggests you could enjoy many benefits when obtaining citizenship.
• You’re applying for your first visa to the U.S. – either as a student, tourist or employee. With your visa approval, you receive information on the citizenship journey and steps.
• You’re buying air tickets online and you see ads describing the benefits of citizenship and applying online.
• You’re at the DMV and need to provide identification for your license. After using your foreign passport, the DMV staffer offers you a brochure on citizenship.
• You have a question regarding city services and call the city’s hotline. While on the phone with the operator, they ask if you’d like to learn about citizenship information sessions.
• Your credit card statement arrives. It includes a message about possible financial benefits to citizenship.
• You’re moving to a new home and want to be sure USCIS has your latest information. The USCIS site reminds you to save your travel and residence for five years so you can easily apply for citizenship.
• You take your children to enroll in school and meet with school staff for paperwork. The welcome packet from the school they give you includes information on how citizenship can benefit families.
• You’re in line at Western Union and you see posters about citizenship benefits and advantages.

As Seen from 18 Tests with Eligible or Recently Naturalized Participants:

ONLY HIGH MARKS

Chosen by most interviewees as comfortable or very comfortable.

MOSTLY HIGH MARKS

Chosen as comfortable by many interviewees, and one or more reacting negatively towards it.

MIXED MARKS

Received both positive and negative marks from different interviewees

LOW MARKS

Viewed negatively by most interviewees

Our team was eager to apply our insights to a tool that could help immigrants overcome the application complexity and length of the naturalization application form. We crafted a list of potential features that might alleviate barriers like a lack of knowledge about fee waivers, the overwhelming size of the application, or an uncertainty of timelines. We presented participants with the list of features and asked each person to discuss and rank the ones they were likely to use. Additionally, if there were any messages they did not like, the participant was asked to note this.

It’s important to note that a number of these features exist in current tools or websites, such as Citizenshipworks and USA Learns.

VERY HIGHLY RANKED: Chosen by all interviewees as useful.

• To Do List For preparing, filling and reviewing the app

HIGHLY RANKED: Chosen by most interviewees as useful.

• Help Finder For nearby classes or application workshops
• Civics Question Weekly Quiz
• Mock Interview Videos

RANKED WELL: Chosen by some interviewees as useful.

• Application Submit Goal & Countdown May occasionally messages me
• Travel Log Generator Scan email and photos for travel dates and locations
• Fee Waiver Eligibility Tool Find out if I qualify

REJECTED: Distinctly noted as not useful by many interviewees.

• Social Pledge Announce on social media about my intent to apply and goal date** removed after failing first six tests in a row
• Invite Others to Apply With You Select friends and family to message about applying at the same time
• Crowdfund Invite Select friends and family to send invites to contribute to your fee fund

All features presented:

1. Application Fee Goal & Budget Plan for my fee by setting aside some $ each month.
2. Help Finder For nearby classes or application workshops
3. To Do List For preparing, filling and reviewing the app
4. Social Pledge Announce on social media about my intent to apply and goal date** removed after failing first six tests in a row
5. Application Submit Goal & Countdown Which occasionally messages me
6. Create Support Team Select fam/friends to cheer me on
7. Residence Generator Gather prior home addresses via my amazon account or credit report
8. Travel Log Generator Scan email and photos for travel dates and locations
9. Civics Question Weekly Quiz
10. Crowdfund Invite Select friends and family to send invites to contribute to your fee fund
11. Fee Waiver Eligibility Tool Find out if I qualify
12. Invite Others to Apply With You Select friends and family o to message about applying at the same time
13. Mock Interview Videos
14. Comparable Timeline Tracker** added in late per suggestion from testers

We were also specifically interested in reactions to mobilizing support for eligible applicants within their own network since we have not heard of many solutions or campaigns to bolster encouragement from the friends and family of eligible residents. To do this, we created a prototype to gauge interest in solutions with a focus on initiating conversations regarding assistance. We took inspiration from a recently launched civic engagement tool – Vote with Me – which allows people to scan public voter registration records against their contacts list in order to encourage them to vote in upcoming elections. In our prototype, we focused on encouragement from either side, asking applicants to reach out for assistance from those they know or asking those who know eligible residents to acknowledge their interest in helping others with their citizenship application.

To determine what facilitating ongoing support might look like, we included features that coincided with an application checklist that could support conversations between applicants and supporters for the duration of the application process. Features related to a checklist also provided tangential insight into the role application management solutions might have on barriers related to application complexity.

Similar to our use of quantitative measurements before, our intention in testing with a prototype was to dig deeper into qualitative findings. The prototype did evolve subtly as multiple participants indicated patterns of positive or negative reactions. This iterative approach allowed us to conclude testing with an example that incorporated improvements from eligible or recently naturalized citizens. However, future groups making solutions for those seeking citizenship should perform additional and intentional product-focused research project with more design and testing if they would like to move forward with insights from this testing. The following appendix item shows our final prototype designs with annotations on reception from both immigrants and their loved ones.

When testing ongoing support, we asked participants to rate (1 to 5) and discuss their comfort level of receiving messages or notifications in five different scenarios. Scenarios ranged in tone and sender, whether from a loved one or the app itself.

HIGH MARKS

LOW MARKS

Prototype: Overview

What app set out to do: Gauge rage of interest in a tool to facilitate support, but checklist was a fun byproduct.

• Two workflows – supporter and applicant.
• Positive, supportive tone

Prototype: Application checklist and reminders

We tested a design for a checklist feature meant to help applicants prepare key documentation for the application.

Feedback

The checklist received only positive feedback from all testing participants. Some mentioned wanting to create subtasks or reminders depending on questions they had about the application or on documents they need for a unique situation. Many would like to be able to connect with expert help through the app too. Additionally, many assumed and were interested in the app being the centralized location to input their information – some also anticipated all input could then be sent to USCIS via the tool.

Testers were highly appreciative of the automated reminders for their citizenship tasks. Some were resistant to getting notifications from the app, but warmed up to the idea once they saw the checklist. Many appreciated the eligibility discovery component of the tool as well.

Prototype: Asking for support

Testing participants evaluated the ability to form a support team through the app. Supporters would be notified of the application progress automatically based on preferences set by both parties. Applicants would be able to connect with supporters by sharing their contact list with the app and then selecting from a series of pre-written messages to invite them to the app (displayed on this page).

Feedback

Most interviewees rejected the idea of creating a support team as a feature before looking at the prototype, but welcomed it when seeing it in context and understanding how it might work.

Most would choose a small support team with only one or two loved ones. Many participants mentioned they would choose their spouse or partner. Other possible supporters included close friends, parents, and adult children, or generally people they see in person frequently. One participant considered including co-workers who are also going through the process—these are not people they are close to, but who would empathize with their situation.

Interviewees were pleased with message suggestions, as it would be useful to frame the request for help.

Many said they would reach out in-person prior to using the app, as a way to clear the subject.

Some participants were hesitant to share contacts with the app following the message selection due to privacy concerns.

Prototype: Preferences for support team

Following feedback from one of our testing participants who felt that ground rules and boundaries for support should be defined for both supporter and applicant, we developed an interface to facilitate this conversation.

Feedback

Not many participants saw this screen because it was developed later in the testing phase, but it received overall positive feedback.

Prototype: Journey comparison

Testing participants evaluated a concept for connecting app users. It features sample journeys of other people applying through the app, with a choice to filter by country, by goal date, and by current location. The progress bars display how far the users are in their application. In order to test additional features, we included a forum for users to ask for tips on how others have prepared their application.

Feedback

Anonymity was very important for our testing participants. The progress bar functionality received overall positive reception, as long as details about the user's application were kept anonymized.

The forum functionality was also well-received, though a few expressed concern during interviews about seeing only posts from desperate, "crazy" users. Good community management would be important to keep the tone positive and helpful. Some users expected there to be experts available to answer questions about the application—it might be useful to supply the forum with expert-written articles about specific aspects of the form, or vet posts carefully and expertly for accuracy.

A future application could also consider methods of building trustworthy anonymous profiles, which would be helpful in retaining privacy in forums or if users are connected with each other.

Prototype: Supporter invitation to prepare for application with app

Some of our testing participants included those in the role of supporters of applicants. We showed them a different side of the app that was focused on helping the supporter follow the applicant's progress with their checklist and providing them with encouraging help-driven messages to avoid counter-productive conversations.

Feedback

Users were pleased with message suggestions, but were hesitant about how to reach out. Some preferred to do it in person because they had previously insisted their loved one naturalize or were worried about surprising their loved one with such sensitive subject matter. There is an opportunity to reframe this as suggestions on how to broach the subject with applicants to help them avoid falling into a nagging pattern.

Prototype: Following a loved one's checklist

For supporters, we showed the applicant's checklist that was synced to real time progress and provided ideas on how to help.

Feedback

The level of involvement a supporter should have in the application was varied, according to supporter testing participants. Most expressed that applicants wouldn't want much help unless they were having problems with a specific task. Some were interested in being cheered for completing a difficult task, but not all tasks. Seeing the checklist elicited a range of desire to be involved, with some supporters wanting to proactively input information on the applicant’s task. More research can be done to evaluate the best platforms and interactions to match the involvement of these different roles.

Prototype: Supporter sees ways to help

Testing participants in the role of supporters evaluated content around how to be most helpful to applicants without being a nag. They were told this content would be available for them for all tasks, with support and motivation being shown as an example.

Feedback

This feature received positive feedback from participants.