Ongoing Feedback collection

Our ongoing UX methods and activities to collect feedback from our visitors include:

  • Per page feedback to gather feedback at a page level
  • NPI to gather longitudinal data
  • Feedback survey to gather comments about the overall website
  • User experience (UX) audit to review user flows across pages

We reccomend a collaborative approach across different skill sets to execute each activity.

This rubric identifies skills and roles recommended to use each tool.


🗒️ Per page feedback

We capture user feedback at a per page level to better understand what pages are useful; what on the page is meaningful, confusing, or distracting; and how we can make them relevant to user needs. We use a native form with daily comments hosted on github and weekly synthesis and anlysis on Google Sheets, combined with responses collected through a related Ethnio survey.

Here’s how we create the per page feedback reports:

Export feedback data and import into systemThe data goes into Azure Cosmos DB (a cross region, managed database). We have a webpage with a few components in this repo that hits the cosmos db api and pulls out the data with query paramsSimple graph views based on our APIs hosted on github pagesEngineer
Review dataPublic facing: access data searchable by date, content captain and URL. Every captain reviews their pages and the Researcher reviews homepage.Per page feedback data siteContent captains and Researcher
Download data from EthnioDownload data from surveys on a weekly basisEthnio + spreadsheetWith an enterprise subscription we could use the API and create an automatic downloadData analyst
Import data to spreadsheetWe created a template to visualize the data from EthnioSpreadsheetAnalyst
Share report with teamEvery week we share the dashboard and themes and findings from previous weekEmail + SlackResearcher and data analyst

Here’s what per page feedback looks like in production:

Per page feedback

🌡️ NPI Survey

We use the NPI survey to gather longitudinal data from visitors to covid19.ca.gov and ca.gov to discover how attitudes and behaviors change in response to changes in policy, local, and/or state economy and other factors. We present the link to the survey to every 50th visitor on the covid19.ca.gov homepage and every visitor to the ca.gov homepage. This survey is maintained through SurveyMonkey.

Here’s how we create and update the NPI survey:

Design surveyThe Insights team designed the first version early on in March. Most questions remain the same to gather longitudinal data. We work with the engineering team to make changes to the frequency it’s shown to visitors.SurveyMonkeyInsights team and Researcher
Meet weeklyThe Insights team and some external volunteers meet to discuss results and future iterationsGoogle Meet or ZoomInsights team, researcher, volunteers
Dump data daily on spreadsheetEvery weekday morning we download SurveyMonkey data from the NPI survey from the previous day from 12:00 am to 11:59 pm and then upload it to the NPI raw data sheet with a new dateNPI raw data sheetAanyone with access to SurveyMonkey
Review open ended questions and gather quotesInsights team reviews open-ended questions to gather quotes for the GO book (delivered weekly). Quotes are chosen to reflect constant and frequent themes, new themes we want to surface, and edge cases we want to shed light on.Opportunity to follow up with interviews with participants who provided their email. We began collecting emails on November 1, 2020, but have not done much with them.Insights team, researcher

Example of our NPI

🚧 Feedback survey (footer)

We have a survey link in the footer for visitors to give general website feedback, suggestions, and comments. This is also maintained through SurveyMonkey.

Here’s how to update the survey:

Design survey or edit questionsWe have edited the questions to avoid cannibalization of other sources of feedbackGoogle Docs and SurveyMonkeyResearcher
Dump data dailyDownload the data from SurveyMonkey and upload it to the Google Sheet. Read the feedback each morning.SpreadsheetAnyone with access to SurveyMonkey
Review comments dailyShare meaningful comments with the team via Slack channelSlack or standupAnyone with access to SurveyMonkey

Example of our Feedback link

🔏 User experience (UX) audit

We perform a UX audit to identify common pain points across key user flows and roadbloacks and success points when performing a task. We can then improve the overall flow instead of just focusing on a specific page.

A team of a designer, researcher, and data analyst define key user tasks. They map out the entire user flow to get a clear picture of all the different pages and actions users take to perform the previously selected tasks. We use Miro, Google Docs, Google Slides, Spreadsheet, and Quicktime in this process.

Here’s how we create and implement an UX audit:

Design studyMeet to design the studyingResearcher, designer and data analyst
Create scenariosBased on Google Analytics, Governor’s Office priorities, and other feedback sources, create scenariosResearcher, designer, and data analyst
Assign scenariosEach team member choses one or more scenarios to perform the auditResearcher, designer, and data analyst
Execute auditIdentify the parts of the website that answered the user’s question in the scenario.Researcher, designer, and data analyst
Visualize auditCreate a map that shows all of the ways that a user might get there (including search). Visualize the steps through a flow chart, mind map, or something similar.Researcher, designer, and data analyst
Uncover pain pointsFor each step in the flow, identify potential pain points that make the process difficult (like confusing labeling)Researcher, designer, and data analyst
Find themesSynthesize pain points to determine themes/trendsResearcher, designer, and data analyst
Define heuristicsCreate heuristics to reuse in future auditsResearcher and designer
Score scenariosScore each flow with the heuristics/usability rubricResearcher and designer
Create list of quick wins and big improvementsFor each theme, create a list with small wins and big improvements. Try to introduce small wins immediately. Include big improvements in the next major redesign.Researcher and designer

Documentation for December 2020-January 2021 UX audit

UX Audit

Created and maintained by Office of Digital Innovation