Survey Design Basics for Qual-First UXRs

1 Introduction to Survey Design Lauren Isaacson, MBA Market & UX Research Consultant Curio Research www.curioresearch.net @curio_research Photo by Curtis MacNewton on Unsplash

2 Thank you for being here! www.curioresearch.net @curio_research Photo by Ricky Kharawala on Unsplash

3 www.curioresearch.net https://iwsmt-content-ok2nbdvvyp8jbrhdp.stackpathdns.com/9152012004322nXqEh.jpeg @curio_research

https://c1.staticflickr.com/4/3064/4566278908_c97b3a557a.jpg 4 www.curioresearch.net @curio_research

Photo by Ken Treloar on Unsplash 5 Surveys! What Are They Good For? Absolutely Something www.curioresearch.net @curio_research

6 We love to send surveys www.curioresearch.net @curio_research Photo by Nong Vang on Unsplash

7 If you cannot measure it, you cannot improve it. LO R D K E LV I N www.curioresearch.net M_V76_D622_Lord_kelvin_with_his_compass.png @curio_research

8 easy www.curioresearch.net @curio_research

9 www.curioresearch.net @curio_research

10 We hate to get surveys www.curioresearch.net @curio_research Photo by Andre Hunter on Unsplash

https://i1.wp.com/www.lifecreativelyorganized.com/wp-content/uploads/2018/01/business-3070472_1920.jpg?resize=620%2C247&ssl=1 11 www.curioresearch.net @curio_research

https://cdn.pixabay.com/photo/2012/05/04/10/56/work-47200_960_720.png 12 www.curioresearch.net @curio_research

13 www.curioresearch.net @curio_research

14 www.curioresearch.net 06/5279525928_199804e5c9_b.jpg @curio_research

15 What Makes a Good Survey? www.curioresearch.net dgallery.com/pictures/survey.jpg @curio_research

16 Keep it short www.curioresearch.net @curio_research Photo by Meg Kannan on Unsplash

The shorter the better 17 Have a singular focus Keep the number of research objectives to a minimum. Use quantitative questions Open ended text response questions make the survey longer and more taxing on the respondent. No longer than 10-minutes to complete Fewer Questions = More Responses + More Completes www.curioresearch.net @curio_research https://pixabay.com/p-161539/

18 Mobile First 44% of SurveyMonkey surveys were taken on a mobile device in 2016* www.curioresearch.net *SurveyMonkey European Survey Research Association 2017 Presentation @curio_research Photo by Oliur Rahman on Unsplash

19 Do you really need to ask that? Really? Are you sure? If you knew this information, what would you do? www.curioresearch.net @curio_research a href=”hSps://www.flickr.com/photos/7342411@N02/4865201576/

20 Make all questions tweet* length questions Simply stated. Direct. Clear. Jargon free. www.curioresearch.net *Classic 140, let’s not get started on 280 @curio_research

21 Be friendly Not formal www.curioresearch.net @curio_research Photo by Alex Knight on Unsplash

22 Help respondents give you correct answers www.curioresearch.net @curio_research Photo by Kelly Sikkema on Unsplash

Help Respondents Be Accurate • Make answer opSons exhausSve - • • www.curioresearch.net ccuracy-Dartboard-Sport-Competition-Target-Darts-857092.jpg 23 Think about and do some research to give respondents every possible opQon Consider the ranges you offer carefully - Respondents will interpret the ranges you give as typical of the populaQon

Below average, average, above average Don’t expect respondents to remember anything - Give specifics @curio_research

y.com/p-1955257/ Help Respondents Be Accurate • Offer an opt-out - • Other, None, Not Applicable Let people enter in their own answer - Other _________ • www.curioresearch.net 24 Only use this if you actually plan to use the data @curio_research

25 Do not make questions required www.curioresearch.net @curio_research Photo by Isaiah Rustad on Unsplash

26 www.curioresearch.net /ba1d87c311ebb69b3704a1d74acb193c—orient-savior.jpg @curio_research

Basic Survey Question 27 Do you have a non-human companion? Dog Cat www.curioresearch.net @curio_research

Basic Survey Question 28 What kind of pet do you have? Dog Cat www.curioresearch.net @curio_research

Basic Survey Question 29 What kind of pet do you have? Dog Cat Amphibian Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet @curio_research

Basic Survey Question 30 What kind of pet do you have? Dog Cat Amphibian Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet @curio_research

Basic Survey Question 31 What kind of pet do you have? Dog Cat Amphibian Reptile Rodent RANDOMIZE Fish Bird Other www.curioresearch.net I don’t have a pet ANCHOR @curio_research

Basic Survey Question 32 What kind of pet do you have? Dog Cat Short quesSon Amphibian Plain language Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet ExhausSve answer opSons Allows mulSple answers Randomized and Anchored It’s not mandatory @curio_research

Basic Survey Question 33 Do you have a non-human companion? Dog Cat Amphibian Reptile Rodent Fish Bird Other www.curioresearch.net I don’t have a pet @curio_research https://c1.staticflickr.com/8/7073/6878033654_cf98354007_b.jpg

34 QUESTIONS? S C A L E D E S I G N , B I A S M I T I GAT I O N , DATA Q UA L I T Y, D E P LOY M E N T, & A N A LYS I S A R E CO M I N G U P Photo by Andre Hunter on Unsplash www.curioresearch.net

35 Scales www.curioresearch.net @curio_research hSps://www.flickr.com/photos/40742367@N04/1643536194

Scales Very Dissatisfied Dissatisfied www.curioresearch.net Neutral 36 Satisfied Very Satisfied @curio_research

Scales Very Dissatisfied Dissatisfied Very Dissatisfied Dissatisfied www.curioresearch.net Somewhat Dissatisfied Neutral Neutral 37 Satisfied Somewhat Satisfied Very Satisfied Satisfied Very Satisfied @curio_research

Scales Very Dissatisfied Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Dissatisfied Plus www.curioresearch.net Neutral Neutral Neutral Negative Neutral Positive 38 Satisfied Somewhat Satisfied Somewhat Satisfied Very Satisfied Satisfied Satisfied Satisfied Plus Very Satisfied Very Satisfied @curio_research

Scales Very Dissatisfied Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Very Dissatisfied Dissatisfied Somewhat Dissatisfied Dissatisfied Plus www.curioresearch.net Neutral Neutral Neutral Negative Neutral Positive 39 Satisfied Somewhat Satisfied Somewhat Satisfied Very Satisfied Satisfied Satisfied Satisfied Plus Very Satisfied Very Satisfied @curio_research

40 Neutral is a valid answer www.curioresearch.net @curio_research Photo by Stanley Emrys on Unsplash

41 Good scales are: • Likert/Odd numbered • Balanced - • VerScal - • The same number of posi0ve and nega0ve op0ons with a neutral op0on in the middle Mobile first Labeled www.curioresearch.net @curio_research https://upload.wikimedia.org/wikipedia/commons/9/9e/Scales_of_Justice_%28PSF%29.png

42 www.curioresearch.net /ba1d87c311ebb69b3704a1d74acb193c—orient-savior.jpg @curio_research

Scale Question 43 Please rate your level of satisfaction with this conference: Excellent www.curioresearch.net Good Decent Poor @curio_research

Scale Question 44 How is the conference going so far: Excellent www.curioresearch.net Good Decent Poor @curio_research

Scale Question 45 How is the conference going so far: Poor www.curioresearch.net Decent Good Excellent @curio_research

Scale Question 46 How is the conference going so far: Poor www.curioresearch.net Decent Neutral Good Excellent @curio_research

Scale Question 47 How is the conference going so far: It’s really bad www.curioresearch.net It kind of bad Neutral It’s pretty good I’m having a great time! @curio_research

Scale Question 48 How is the conference going so far: I’m having a great time! It’s pretty good Neutral It kind of bad It’s really bad www.curioresearch.net @curio_research

Scale Question 49 How is the conference going so far: I’m having a great time! It’s pretty good Short quesSon Plain language Neutral opSon Neutral It kind of bad It’s really bad www.curioresearch.net Likert Scale Balanced scale VerScal @curio_research

Scale Question 50 How is the conference going so far: I’m having a great time! It’s pretty good Neutral It kind of bad It’s really bad www.curioresearch.net https://www.flickr.com/photos/47823583@N03/8508102188/ @curio_research

51 QUESTIONS? B I A S M I T I GAT I O N , DATA Q UA L I T Y, D E P LOY M E N T, & A N A LYS I S A R E O N T H E I R WAY Photo by Tim Mossholder on Unsplash www.curioresearch.net

52 Minimize grid usage Relying on grids is cheating www.curioresearch.net @curio_research Photo by Samuel Zeller on Unsplash

53 www.curioresearch.net p://cdn2.content.compendiumblog.com/uploads/user/b8bbc9ab-67b1-4a8e-ac5c-5811daa967bd/1bfbf295-292e-4b39-a40d-a3fed11a5f1e/Image/31dd75016b423826eca4861f3d6f73f1/1grid.jpg @curio_research

54 Bias is real Agree or Disagree? www.curioresearch.net @curio_research Photo by Frida Aguilar Estrada on Unsplash

Respondent Biases 55 Know your enemies Acquiescence Bias Desireability Bias We just want to get along with everyone by being agreeable. Even people we’ve never met. We like appearing better than we actually are. AVOID AVOID Questions involving answers such as: Asking people to recall their own consumption habits. Find actual usage or purchase data instead. Yes/No Agree/Disagree www.curioresearch.net True/False @curio_research

Researcher Biases 56 Know your enemies Sampling Bias Research Bias The sample for the survey doesn’t reflect the population being researched. The point of view of the researcher interferes with the objectivity of the research. USE USE Demographic Quotas Word questions to be as neutral as possible Random Elements: Random number generators Obfuscate your purpose Dice Red Herring answers www.curioresearch.net @curio_research

57 www.curioresearch.net /ba1d87c311ebb69b3704a1d74acb193c—orient-savior.jpg @curio_research

Neutral Question 58 Do you use Gmail? Yes No www.curioresearch.net @curio_research

Neutral Question 59 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email www.curioresearch.net @curio_research

Neutral Question 60 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook I have my own email server Other ____________ I don’t use email www.curioresearch.net @curio_research

Neutral Question 61 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook FantasticMail I have my own email server Other ____________ www.curioresearch.net I don’t use email @curio_research

Neutral Question 62 Which of these email services do you use? Gmail Hotmail Yahoo! Mail Outlook RANDOMIZE FantasticMail I have my own email server Other ____________ www.curioresearch.net I don’t use email ANCHOR @curio_research

Neutral Question 63 Which of these email services do you use? Yahoo! Mail FantasticMail Gmail Plain language Outlook Data quality red herring Hotmail Allows mulSple answers I have my own email server Other ____________ www.curioresearch.net Doesn’t lead I don’t use email Randomized and Anchored It’s not mandatory @curio_research

Neutral Question 64 Which of these email services do you use? Outlook I have my own email server Gmail Yahoo! Mail Hotmail FantasticMail Other ____________ www.curioresearch.net I don’t use email https://www.flickr.com/photos/meme_tn/6878030444 @curio_research

65 QUESTIONS? I N C E N T I V E S A N D A N A LYS I S A R E O N D EC K Photo by Rohit Farmer on Unsplash www.curioresearch.net

66 Incentivize www.curioresearch.net @curio_research Photo by Olga DeLawrence on Unsplash

Incentives 67 Not necessarily difficult or expensive METHODS Online gift cards, account credit, random prize drawing, charitable donations… Get creative! BETTER RESPONSE RATES Mitigates non-response bias for better results. DEMONSTRATES RESPECT Shows you understand the value of respondents’ time. www.curioresearch.net @curio_research

68 Open with gratitude Close with gratitude And a final open end www.curioresearch.net @curio_research Photo by Tim Marshall on Unsplash

/Fig-Mourning-Angel-Sculpture-Gratitude-Consolation-1990440.jpg 69 BE GRACIOUS Thank respondents for their participation at the opening and the closing. BE TRANSPARENT Tell them how the data will be used. Improve a product? Help a non-profit? Inform sponsors? BE GENEROUS Offer to share the results of the study with them when it’s over. BE OPEN Present a final open end so they can share something you might have missed. www.curioresearch.net @curio_research

70 Analyzing the Data www.curioresearch.net 27/9423385629_171671f9c2_b.jpg @curio_research

Have a data analysis plan What are your objectives for the data? Additional data analysis? Conjoint? Regression? Text analytics? www.curioresearch.net 71 1 What variables do you want to use in your cross tabs? 2 3 Will you need special software to run the analysis? 4 @curio_research

Clean Your Data 72 REMOVE • Extreme outliers • Respondents who didn’t answer all of the quesQons • Respondents who used red herring answers • Respondents who straight-lined ASSESS Do you still have the sample size you need? Yes? Go on to the analysis. www.curioresearch.net No? Only remove the worst 20% of the iffy responses or get more respondents. @curio_research

What does standard deviation mean? Low Number High Number Homogeneous Data Heterogeneous Data www.curioresearch.net 73 @curio_research

Top 2 Boxes/Percent Positive 74 Analyzing scale data • When analyzing scale data calculate the percentage of respondents that chose the top 2 boxes - 5 point scale = top 2 boxes

7 point scale = top 3 boxes It’s really bad It’s kind of bad Neutral It’s pretty good I’m having a great time Responses 1 24 19 59 78 % of Total 0% 13% 11% 33% 43% Top 2 Boxes www.curioresearch.net 76% @curio_research

Top 2 Boxes/Percent Positive 75 Graphing scale data How is the conference going so far? 24% 0% 76% 25% NegaSve /Neutral www.curioresearch.net 50% 75% 100% PosiSve @curio_research

76 QUESTIONS? N E X T A R E K E Y TA K E AWAYS A N D F U RT H E R R E A D I N G Photo by Eunice Lituañas on Unsplash www.curioresearch.net

s/30478819@N08/36996342724/ 77 Key Takeaways www.curioresearch.net @curio_research

Key Takeaways 78 Yes, take a photo of this slide Surveys are a UX problem you can solve. Keep it short. 10 minutes or less. Make them mobile first. Vertical. Concise. Abandon grids. Limit open ends. Make all scales balanced Likert scales. 5s or 7s. Analyze using Top 2 (or 3) boxes. Have a plan for the data before you write the survey. Incentivize. Incentivize. Incentivize. Clean your data before analysis. What are you doing to ensure data quality? www.curioresearch.net @curio_research

https://www.flickr.com/photos/tetsumo/3802078924 79 But Wait, There’s More! www.curioresearch.net @curio_research

Further Reading Internet, Mail, and Mixed-Mode Surveys Don A. Dillman, Jolene D. Smyth, and Leah Melani Christian www.curioresearch.net Questionnaire Design Ian Brace The Complete Guide to Writing Questionnaires 80 People Are Not Robots Annie Pettit, PhD David F. Harris @curio_research

Online Sources 81 NewMR Annie Pettit, PhD Jeffrey Henning Ray Poynter Hashtag #NewMR Twitter @NewMR_News Youtube “NewMR Videos” Twitter @LoveStats Twitter @JHenning Twitter @RayPoynter www.curioresearch.net @curio_research

82 THANK YOU CurioResearch.net @Curio_Research www.curioresearch.net @curio_research Photo by Nick Hillier on Unsplash