Experiment-Driven Product Development

<Experiment name> Background Experiment details Question: <What do we want to know?> Measures Why: <Details of the question & background information> What: <What are we going to do e.g. A/B test, UX research, Data analysis> Hypothesis: <What are we going to do? What change are we expecting to see? How will we measure success?> ● Conditions A. B. <A condition - new behaviour> <B condition - existing behaviour> Details ● ● ● ● ● Headline result: <What have we learned?> <Measure, including current rate if applicable> <What is our Minimum Detectable Effect threshold? (e.g. 1%)> <How big a sample does each condition need?> <Therefore, how long will we run the experiment at a minimum? E.g. 24 hours> <What subset of the data will we use? E.g. config.nnn> <Event name/filters> @r4isstatic

Email first on popup experiment Background Experiment details Question: Can we drive more email sign ups by moving the sign up card to the front of the popup? Measures Why: Only ~2% of users that see recommendations see the email sign up card on the popup. We think showing the card earlier would increase sign ups. What: A/B experiment. Hypothesis: Showing the sign up card at the start of the popup (before recommendations) will lead to more email sign ups than having the card at the back, but the verification rate will not change significantly. ● ● ● ● Email subscribers Email sign up rate (subscribes / no. users seeing card) Email verifications Close + no thanks rate and CTR (as checks) Conditions A. B. Email sign-up last card in popup journey Email sign up 1st card in popup journey Details ● ● 24 hours* config.experiment_email_position

‘2017-10-05 14:00:00 - 2017-10-06 14:00:00 Headline result: Putting email signup card first drives far more signups (and most of these still verify) without leading to a large adverse effect on other metrics, although it does massively reduce the recommendations we display (see full writeup for more details) @r4isstatic

Latest Articles sliced by Subject Background Experiment details Question: Would showing latest articles, sliced by Subject, be a more useful feature than the traditional ‘Browse Articles’ box? Measures Why: We want to explore ways that we can usefully slice content, to better show off the range of content within a broad-scope journal What: A/B Hypothesis: Displaying three of the latest articles for each top level subject will lead to a higher CTR to articles than the ‘Browse Articles’ box ● Unique pageviews on homepage ● Unique clicks on any article in each subject box ● Unique clicks to any article in ‘Browse Articles’ box ● Unique pageviews to any SREP article Conditions A. Users visiting the journal homepage are shown the ‘Browse Articles’ box and the ‘Browse Subjects’ box (which just has links to the scoped searches) B. Users visiting the journal homepage are shown four subject boxes, each with a set of the three latest articles from that subject, and a link to the scoped search page Details ● 18-24th October (7 days) Headline result: 5.4% click through to articles, compared to 4.2% with the traditional ‘Browse Articles’ grid of nine. “See All” for each subject received 10% UCTR, compared to 8% from previous design, and 6.7% for ‘See All’ for all SREP