Colorado Financial Transparency User Testing

BrightBytes & Colorado Department of Education (CDE), May 2017

Project: public website

Contributions: user research

Team: UX design (2), interface design (2), product management (1), development (5)

Results: validated that the site was compelling with a high referral rate, and prioritized needed improvements for the next iteration

Enthusiastically supported by our client, user testing of the Colorado Financial Transparency site prepared both teams to plan the scope of improvements following the initial launch. This stage allowed for flexibility and larger breadth in our research; the product was finalized for launch, so our goal was to discover and prioritize opportunities for the next iteration. With a senior product manager advising me on the client’s concerns and needs, I constructed a research plan that used remote interviews with 10 demographically diverse Colorado residents to give us rich, qualitative data needed at that stage. Our research objectives centered on core task completion and comprehension of the site: could users search schools or districts, compare these using the tool, navigate regions of interest using the map, and evaluate and comprehend the complex financial data presented. After formalizing the plan with the client, script writing/reviewing, and scheduling interviews, I lead our interviews as the moderator. Using the notes that I scribed as I conducted the interview, I concluded each testing day by analyzing the day’s findings. I borrow this technique from ethnographic research; I find it extremely helpful for an accurate reading and productive interpretation. At the end the each week of interviews -- conducted separately in May and July -- I crafted a presentation of all findings and design recommendations, which the product management and executive team subsequently shared with CDE.

Several slides from the beginning of the summary presentation reported to CDE demonstrate the key areas a user testing presentation should always touch on. First, significant quotes of feedback serve to connect the audience to the feelings of the testing participants, so the findings are much more realized and understood. Next, findings should always be presented with some prioritization of severity, and very significantly, they should be separate from suggested improvements. Last, sometimes “quick and dirty” mark-ups on designs are the easiest way to make suggestions seem more open, light, and less pointed at shortcomings. Research is all about discovery for iteration! These ideas and more were discovered and prioritized in our product roadmap through this testing effort, and many became a part of the current site today.

The Financial Transparency user testing plan highlights some of the essential structures of a research plan for moderated, remote user interviews -- business case, objectives, participants, equipment, basic test tasks. These plans also typically include team responsibilities, scheduling recommendations, and links to other related documentation (notes, recordings, test activities etc.). Much of this is hugely useful for coordinating with an external partner, such as CDE, to ensure an understood scope of the research prior to beginning. I used this testing plan to help communicate areas I would need assistance in, like recruitment of participants. Mostly, my user testing plans serve as complete internal documentation for managing the whole research project as it runs, thus becoming a “source of truth” to return to when planning for product improvements in the future.