Competition - Was a Website Redesign Successful?

Isaac Awotwe

‌
‌
‌
‌
‌
‌
‌

Reader mode

Beta

Which version of the website should you use?

📖 Background

You work for an early-stage startup in Germany. Your team has been working on a redesign of the landing page. The team believes a new design will increase the number of people who click through and join your site.

They have been testing the changes for a few weeks and now they want to measure the impact of the change and need you to determine if the increase can be due to random chance or if it is statistically significant.

💾 The data

The team assembled the following file:

Redesign test data

"treatment" - "yes" if the user saw the new version of the landing page, no otherwise.
"new_images" - "yes" if the page used a new set of images, no otherwise.
"converted" - 1 if the user joined the site, 0 otherwise.

The control group is those users with "no" in both columns: the old version with the old set of images.

import pandas as pd
df = pd.read_csv('./data/redesign.csv')
df.head()

#Get some information about the data
df.info()

There are 40,484 rows (visits to the websites). All the columns have the expected data types.

1. Summary Statistics

#import relevant packages
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns


#Derive summary  statistics for all columns in the data frame
df.describe(include='all', percentiles=[0.5])

There are four distinct group of website users:

Users who saw the new version of the landing page (treatment=”yes”) that uses new sets of images (new_images=”yes”) - henceforth called Group 1.
Users who saw the new version of the landing page (treatment=”yes”) that did not use the new set of images (new_images=”no”) - henceforth called Group 2.
Users who did not see the new version of the landing page (treatment=”no”) but the version they saw used a new set of images (new_images=”yes”) - henceforth called Group 3.
Users who did not see the new version of the landing page (treatment=”no”) and the version they saw did not use a new set of images (new_images=”no”) – this is the control group.

#Compute the proportion of users who joined the site (conversion rate) within each group

rates_df=df.groupby(["treatment", "new_images"])["converted"].mean().reset_index()
rates_df.columns=["treatment", "new_images", "turnover_rate"]
rates_df

The crosstabulation above shows that:

The control group had the lowest observed conversion rate(10.7%).
Of the group that did not see the new landing page but saw the new set of images, observed conversion rate was about 11.3%
About 11.4% of the group that saw the landing page with a new images of images joined the site.
The group saw the new landing page without the new set of images had the highest observed conversion rate(12%).

1.1 Statistical Significance

These results are inconclusive because they were derived from a sample of users. Will the results change if we used a different sample for the experiment? Are these results just a fluke? In other words are the differences in conversion rate statistically significant?

To answer these questions we will examine what would happen if the experiment was repeated multiple times.

In our first step, we will compare the control group to the other three groups to determine if indeed the new landing page and/or new images yield an improvement.

1.2 Statistical Versus Practical Significance

Before we proceed to compute statistical significance let us talk briefly about practical significance. The observed conversion rate for users who saw the new landing page without the new set of images was 1.3 percentage points more than that of the control group - and this is the highest increase among the three non-control groups. From a business perspective, is this increase substantial enough to warrant adopting the redesign? That is to ask "is 1.3 percentage points increase substantial or practically significant?".

If it is not, there would be no need to proceed with testing for statistical significance and we may need to create another redesign of the landing page and/or images.

We will assume for now that the increase is practically significant so that we continue with testing statistical significance.

2. Bootstrapping

Bootstrapping is a non-parametric way of accounting for uncertainty in sample estimates - non-parametric in the sense that it does not make any assumptions about the underlying distribution of the sample data.

Bootstrapping involves randomly resampling from a given sample with replacement and computing the statistic of interest (referred to as the bootstrap replicate) from the resampled data (bootstrap sample).

If we simulate this process many times, we obtain a distribution of bootstrap replicates which can be used to assess the uncertainty around the observed sample.

In our case, for each of the four groups (including the control), we will generate several bootstrap replicates ("conversion rates") and generate a bee swarm plot from the distribution of boootstrap conversion rates. This will give us a visual representation of how the three treatment groups compare with the control group in repeated experiments.

#create a function that takes a number of bootstrap samples and computes their mean: bootstrap
def bootstrap(data, n_boots):
    return [np.mean(np.random.choice(data, len(data))) for _ in range(n_boots)]

#make a copy of the data frame: df_copy
df_copy=df.copy()

#create a new column that assign each row to their group: group
conditions=[(df_copy["treatment"]=="yes") & (df_copy["new_images"]=="yes"), (df_copy["treatment"]=="yes") & (df_copy["new_images"]=="no"), (df_copy["treatment"]=="no") & (df_copy["new_images"]=="yes"), (df_copy["treatment"]=="no") & (df_copy["new_images"]=="no")]

values=["group1", "group2", "group3", "control"]

df_copy["group"]=np.select(conditions, values)


#Holder DataFrame for bootstrap samples
group_boots=pd.DataFrame()

for group in df_copy.group.unique():
    #Filter to group's converted column
    group_converted=df_copy[df_copy.group==group].converted
    #perform 100 bootstrap samples of group's conversion rate and put in a DataFrame
    cur_boot=pd.DataFrame({"conversion_rate":bootstrap(group_converted, 100), "group":group})
    #Append to other group's bootstrap
    group_boots=pd.concat([group_boots, cur_boot])

#Use beeswarm plot to visualize bootstrap samples
sns.set_style("darkgrid")
sns.set(rc = {'figure.figsize':(8,5)}, font_scale=1.2)
sns.swarmplot(y="group", x="conversion_rate", data=group_boots, color="coral")
sns.despine(left=True, bottom=True)
plt.show()

‌
‌
‌

AI Chat

Code

.mfe-app-workspace-kj242g{position:absolute;top:-8px;}.mfe-app-workspace-11ezf91{display:inline-block;}.mfe-app-workspace-11ezf91:hover .Anchor__copyLink{visibility:visible;}Which version of the website should you use?