A presentation at dscout: New Fundamentals of UX Research by Lauren Isaacson
1 Introduction to Survey Design Lauren Isaacson, MBA Market & UX Research Consultant Curio Research www.curioresearch.net @curio_research Photo by Curtis MacNewton on Unsplash
2 Thank you for being here! www.curioresearch.net @curio_research Photo by Ricky Kharawala on Unsplash
3 www.curioresearch.net https://iwsmt-content-ok2nbdvvyp8jbrhdp.stackpathdns.com/9152012004322nXqEh.jpeg @curio_research
https://c1.staticflickr.com/4/3064/4566278908_c97b3a557a.jpg 4 www.curioresearch.net @curio_research
Photo by Ken Treloar on Unsplash 5 Surveys! What Are They Good For? Absolutely Something www.curioresearch.net @curio_research
6 We love to send surveys www.curioresearch.net @curio_research Photo by Nong Vang on Unsplash
7 If you cannot measure it, you cannot improve it. LO R D K E LV I N www.curioresearch.net M_V76_D622_Lord_kelvin_with_his_compass.png @curio_research
8 easy www.curioresearch.net @curio_research
9 www.curioresearch.net @curio_research
10 We hate to get surveys www.curioresearch.net @curio_research Photo by Andre Hunter on Unsplash
https://i1.wp.com/www.lifecreativelyorganized.com/wp-content/uploads/2018/01/business-3070472_1920.jpg?resize=620%2C247&ssl=1 11 www.curioresearch.net @curio_research
https://cdn.pixabay.com/photo/2012/05/04/10/56/work-47200_960_720.png 12 www.curioresearch.net @curio_research
13 www.curioresearch.net @curio_research
14 www.curioresearch.net 06/5279525928_199804e5c9_b.jpg @curio_research
15 What Makes a Good Survey? www.curioresearch.net dgallery.com/pictures/survey.jpg @curio_research
16 Keep it short www.curioresearch.net @curio_research Photo by Meg Kannan on Unsplash
The shorter the better 17 Have a singular focus Keep the number of research objectives to a minimum. Use quantitative questions Open ended text response questions make the survey longer and more taxing on the respondent. No longer than 10-minutes to complete Fewer Questions = More Responses + More Completes www.curioresearch.net @curio_research https://pixabay.com/p-161539/
18 Mobile First 44% of SurveyMonkey surveys were taken on a mobile device in 2016* www.curioresearch.net *SurveyMonkey European Survey Research Association 2017 Presentation @curio_research Photo by Oliur Rahman on Unsplash
19 Do you really need to ask that? Really? Are you sure? If you knew this information, what would you do? www.curioresearch.net @curio_research a href=”hSps://www.flickr.com/photos/7342411@N02/4865201576/
20 Make all questions tweet* length questions Simply stated. Direct. Clear. Jargon free. www.curioresearch.net *Classic 140, let’s not get started on 280 @curio_research
21 Be friendly Not formal www.curioresearch.net @curio_research Photo by Alex Knight on Unsplash
22 Help respondents give you correct answers www.curioresearch.net @curio_research Photo by Kelly Sikkema on Unsplash
Below average, average, above average Don’t expect respondents to remember anything - Give specifics @curio_research
y.com/p-1955257/ Help Respondents Be Accurate • Offer an opt-out - • Other, None, Not Applicable Let people enter in their own answer - Other _________ • www.curioresearch.net 24 Only use this if you actually plan to use the data @curio_research
25 Do not make questions required www.curioresearch.net @curio_research Photo by Isaiah Rustad on Unsplash
26 www.curioresearch.net /ba1d87c311ebb69b3704a1d74acb193c—orient-savior.jpg @curio_research
Basic Survey Question 27 Do you have a non-human companion? Dog Cat www.curioresearch.net @curio_research
Basic Survey Question 28 What kind of pet do you have? Dog Cat www.curioresearch.net @curio_research
Basic Survey Question 29 What kind of pet do you have? Dog Cat Amphibian Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet @curio_research
Basic Survey Question 30 What kind of pet do you have? Dog Cat Amphibian Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet @curio_research
Basic Survey Question 31 What kind of pet do you have? Dog Cat Amphibian Reptile Rodent RANDOMIZE Fish Bird Other www.curioresearch.net I don’t have a pet ANCHOR @curio_research
Basic Survey Question 32 What kind of pet do you have? Dog Cat Short quesSon Amphibian Plain language Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet ExhausSve answer opSons Allows mulSple answers Randomized and Anchored It’s not mandatory @curio_research
Basic Survey Question 33 Do you have a non-human companion? Dog Cat Amphibian Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet @curio_research https://c1.staticflickr.com/8/7073/6878033654_cf98354007_b.jpg
34 QUESTIONS? S C A L E D E S I G N , B I A S M I T I GAT I O N , DATA Q UA L I T Y, D E P LOY M E N T, & A N A LYS I S A R E CO M I N G U P Photo by Andre Hunter on Unsplash www.curioresearch.net
35 Scales www.curioresearch.net @curio_research hSps://www.flickr.com/photos/40742367@N04/1643536194
Scales Very Dissatisfied Dissatisfied www.curioresearch.net Neutral 36 Satisfied Very Satisfied @curio_research
Scales Very Dissatisfied Dissatisfied Very Dissatisfied Dissatisfied www.curioresearch.net Somewhat Dissatisfied Neutral Neutral 37 Satisfied Somewhat Satisfied Very Satisfied Satisfied Very Satisfied @curio_research
Scales Very Dissatisfied Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Dissatisfied Plus www.curioresearch.net Neutral Neutral Neutral Negative Neutral Positive 38 Satisfied Somewhat Satisfied Somewhat Satisfied Very Satisfied Satisfied Satisfied Satisfied Plus Very Satisfied Very Satisfied @curio_research
Scales Very Dissatisfied Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Dissatisfied Plus www.curioresearch.net Neutral Neutral Neutral Negative Neutral Positive 39 Satisfied Somewhat Satisfied Somewhat Satisfied Very Satisfied Satisfied Satisfied Satisfied Plus Very Satisfied Very Satisfied @curio_research
40 Neutral is a valid answer www.curioresearch.net @curio_research Photo by Stanley Emrys on Unsplash
41 Good scales are: • Likert/Odd numbered • Balanced - • VerScal - • The same number of posi0ve and nega0ve op0ons with a neutral op0on in the middle Mobile first Labeled www.curioresearch.net @curio_research https://upload.wikimedia.org/wikipedia/commons/9/9e/Scales_of_Justice_%28PSF%29.png
42 www.curioresearch.net /ba1d87c311ebb69b3704a1d74acb193c—orient-savior.jpg @curio_research
Scale Question 43 Please rate your level of satisfaction with this conference: Excellent www.curioresearch.net Good Decent Poor @curio_research
Scale Question 44 How is the conference going so far: Excellent www.curioresearch.net Good Decent Poor @curio_research
Scale Question 45 How is the conference going so far: Poor www.curioresearch.net Decent Good Excellent @curio_research
Scale Question 46 How is the conference going so far: Poor www.curioresearch.net Decent Neutral Good Excellent @curio_research
Scale Question 47 How is the conference going so far: It’s really bad www.curioresearch.net It kind of bad Neutral It’s pretty good I’m having a great time! @curio_research
Scale Question 48 How is the conference going so far: I’m having a great time! It’s pretty good Neutral It kind of bad It’s really bad www.curioresearch.net @curio_research
Scale Question 49 How is the conference going so far: I’m having a great time! It’s pretty good Short quesSon Plain language Neutral opSon Neutral It kind of bad It’s really bad www.curioresearch.net Likert Scale Balanced scale VerScal @curio_research
Scale Question 50 How is the conference going so far: I’m having a great time! It’s pretty good Neutral It kind of bad It’s really bad www.curioresearch.net https://www.flickr.com/photos/47823583@N03/8508102188/ @curio_research
51 QUESTIONS? B I A S M I T I GAT I O N , DATA Q UA L I T Y, D E P LOY M E N T, & A N A LYS I S A R E O N T H E I R WAY Photo by Tim Mossholder on Unsplash www.curioresearch.net
52 Minimize grid usage Relying on grids is cheating www.curioresearch.net @curio_research Photo by Samuel Zeller on Unsplash
53 www.curioresearch.net p://cdn2.content.compendiumblog.com/uploads/user/b8bbc9ab-67b1-4a8e-ac5c-5811daa967bd/1bfbf295-292e-4b39-a40d-a3fed11a5f1e/Image/31dd75016b423826eca4861f3d6f73f1/1grid.jpg @curio_research
54 Bias is real Agree or Disagree? www.curioresearch.net @curio_research Photo by Frida Aguilar Estrada on Unsplash
Respondent Biases 55 Know your enemies Acquiescence Bias Desireability Bias We just want to get along with everyone by being agreeable. Even people we’ve never met. We like appearing better than we actually are. AVOID AVOID Questions involving answers such as: Asking people to recall their own consumption habits. Find actual usage or purchase data instead. Yes/No Agree/Disagree www.curioresearch.net True/False @curio_research
Researcher Biases 56 Know your enemies Sampling Bias Research Bias The sample for the survey doesn’t reflect the population being researched. The point of view of the researcher interferes with the objectivity of the research. USE USE Demographic Quotas Word questions to be as neutral as possible Random Elements: Random number generators Obfuscate your purpose Dice Red Herring answers www.curioresearch.net @curio_research
57 www.curioresearch.net /ba1d87c311ebb69b3704a1d74acb193c—orient-savior.jpg @curio_research
Neutral Question 58 Do you use Gmail? Yes No www.curioresearch.net @curio_research
Neutral Question 59 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email www.curioresearch.net @curio_research
Neutral Question 60 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email www.curioresearch.net @curio_research
Neutral Question 61 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook FantasticMail I have my own email server Other ____________ www.curioresearch.net I don’t use email @curio_research
Neutral Question 62 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook RANDOMIZE FantasticMail I have my own email server Other ____________ www.curioresearch.net I don’t use email ANCHOR @curio_research
Neutral Question 63 Which of these email services do you use? Yahoo! Mail FantasticMail Gmail Plain language Outlook Data quality red herring Hotmail Allows mulSple answers I have my own email server Other ____________ www.curioresearch.net Doesn’t lead I don’t use email Randomized and Anchored It’s not mandatory @curio_research
Neutral Question 64 Which of these email services do you use? Outlook I have my own email server Gmail Yahoo! Mail Hotmail FantasticMail Other ____________ www.curioresearch.net I don’t use email https://www.flickr.com/photos/meme_tn/6878030444 @curio_research
65 QUESTIONS? I N C E N T I V E S A N D A N A LYS I S A R E O N D EC K Photo by Rohit Farmer on Unsplash www.curioresearch.net
66 Incentivize www.curioresearch.net @curio_research Photo by Olga DeLawrence on Unsplash
Incentives 67 Not necessarily difficult or expensive METHODS Online gift cards, account credit, random prize drawing, charitable donations… Get creative! BETTER RESPONSE RATES Mitigates non-response bias for better results. DEMONSTRATES RESPECT Shows you understand the value of respondents’ time. www.curioresearch.net @curio_research
68 Open with gratitude Close with gratitude And a final open end www.curioresearch.net @curio_research Photo by Tim Marshall on Unsplash
/Fig-Mourning-Angel-Sculpture-Gratitude-Consolation-1990440.jpg 69 BE GRACIOUS Thank respondents for their participation at the opening and the closing. BE TRANSPARENT Tell them how the data will be used. Improve a product? Help a non-profit? Inform sponsors? BE GENEROUS Offer to share the results of the study with them when it’s over. BE OPEN Present a final open end so they can share something you might have missed. www.curioresearch.net @curio_research
70 Analyzing the Data www.curioresearch.net 27/9423385629_171671f9c2_b.jpg @curio_research
Have a data analysis plan What are your objectives for the data? Additional data analysis? Conjoint? Regression? Text analytics? www.curioresearch.net 71 1 What variables do you want to use in your cross tabs? 2 3 Will you need special software to run the analysis? 4 @curio_research
Clean Your Data 72 REMOVE • Extreme outliers • Respondents who didn’t answer all of the quesQons • Respondents who used red herring answers • Respondents who straight-lined ASSESS Do you still have the sample size you need? Yes? Go on to the analysis. www.curioresearch.net No? Only remove the worst 20% of the iffy responses or get more respondents. @curio_research
What does standard deviation mean? Low Number High Number Homogeneous Data Heterogeneous Data www.curioresearch.net 73 @curio_research
7 point scale = top 3 boxes It’s really bad It’s kind of bad Neutral It’s pretty good I’m having a great time Responses 1 24 19 59 78 % of Total 0% 13% 11% 33% 43% Top 2 Boxes www.curioresearch.net 76% @curio_research
Top 2 Boxes/Percent Positive 75 Graphing scale data How is the conference going so far? 24% 0% 76% 25% NegaSve /Neutral www.curioresearch.net 50% 75% 100% PosiSve @curio_research
76 QUESTIONS? N E X T A R E K E Y TA K E AWAYS A N D F U RT H E R R E A D I N G Photo by Eunice Lituañas on Unsplash www.curioresearch.net
s/30478819@N08/36996342724/ 77 Key Takeaways www.curioresearch.net @curio_research
Key Takeaways 78 Yes, take a photo of this slide Surveys are a UX problem you can solve. Keep it short. 10 minutes or less. Make them mobile first. Vertical. Concise. Abandon grids. Limit open ends. Make all scales balanced Likert scales. 5s or 7s. Analyze using Top 2 (or 3) boxes. Have a plan for the data before you write the survey. Incentivize. Incentivize. Incentivize. Clean your data before analysis. What are you doing to ensure data quality? www.curioresearch.net @curio_research
https://www.flickr.com/photos/tetsumo/3802078924 79 But Wait, There’s More! www.curioresearch.net @curio_research
Further Reading Internet, Mail, and Mixed-Mode Surveys Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian www.curioresearch.net Questionnaire Design Ian Brace The Complete Guide to Writing Questionnaires 80 People Are Not Robots Annie Pettit, PhD David F. Harris @curio_research
Online Sources 81 NewMR Annie Pettit, PhD Jeffrey Henning Ray Poynter Hashtag #NewMR Twitter @NewMR_News Youtube “NewMR Videos” Twitter @LoveStats Twitter @JHenning Twitter @RayPoynter www.curioresearch.net @curio_research
82 THANK YOU CurioResearch.net @Curio_Research www.curioresearch.net @curio_research Photo by Nick Hillier on Unsplash
Product and service improvement often relies on survey data, but poorly composed surveys yield bad data—and bad data leads to poor decisions.
The good news is that researchers and designers already have the skills to make surveys easier and more enjoyable for the respondents.
Watch to learn:
Basic and advanced questionnaire design tips
How to reduce the respondent cognitive load (making the task more enjoyable)
How to ask better questions in usability testing surveys
Ways to analyze and track the data for better insights
for free. You
can too.