August 15, 2019 –
What is a Hypothesis?
An ounce of prevention is worth a pound of cure; a little more preparatory work up front significantly reduces later overhead.
Why is a Hypothesis Critical to Your Optimization Program?
What is being proposed as an improvement. This is the theoretical part of an experiment. Whatever change you think will outperform the current state of the site, big or small: stronger call-to-action text on a button, larger images, reduction of form inputs, redesigned header navigation. You get the picture; a short description of your idea for outperforming the baseline.
The snippets of a response are how you scope an experiment and fine tune the signal-to-noise ratio of the result data.
Looking for an A/B testing tool that lets you get very sophisticated with your targeting and audience building? Check out what Convert Experiences is capable of.
How do you Construct a Split Test Hypothesis?
For example, if the variation is changing the position of the estimated order total of an eCom site, then the targeting might be the cart page and then the audience would only be users who have an item in their cart. You would not want to collect data from users that interact with an empty cart page, since they wouldn’t see the UI elements that display a cart total anyway.
You can see how each snippet is a description of an important piece of data that was either collected from research, proposed via design, or outlines measurement criteria.
Analytics
- Comprehension: We have observed a reduction in multi-item purchases by comparing the last 6 months of purchase data.
- Response: We want to promote paired products with an inline upsell at the cart page on mobile phones for returning users with an item already in their cart.
- Outcome: This should lead to single item purchasers more easily finding and purchasing complementary products, which will be measured by average order value (AOV) and backed up by average order size, multi-item purchase count, order conversion, and revenue.
To make components easy to remember, they follow the CRO acronym that all of us are very familiar with.
Formulating a hypothesis forces us to think about what that goal is, how we should try to achieve it, and what results to look for. A good hypothesis provides structure.
Comprehension
A proposed solution to a problem, whose validity requires evaluation.
Observation
For example, if your primary KPI is revenue from orders and the experiment targets a product page, then the secondary KPIs would be add-to-carts, cart pageviews, checkout pageviews, and order conversions.
Observation Method
Where the experiment will take place on your website; types of pages, URL targeting, device scope and/or screen breakpoints.
Outcome comes from measurement; it is how success is defined and evaluated. The snippets of an outcome are: desired behavioral change, the primary KPI, and secondary KPIs. A poor hypothesis creates confusion; garbage in, garbage out. The hypothesis is the foundation of an experiment, it contains all of the information necessary for building a split test.
- Techniques: surveys, usability testing, product reviews
- Tools: Hotjar, Qualaroo, TurnTo
Response
Collecting each snippet can be a time consuming process, but the payoff is solidly constructed experiments that will provide more overall value in the long run.
Feedback
Variation
The innate desire for improvement is what drives an optimization program, but testing just for the sake of testing is a setup for failure. An experiment needs a primary goal that determines success.
Location
Each component is comprised of two to four short written snippets describing data collected via research, design, and measurement. Assemble the snippets together into sentences and you’ve built a solid hypothesis. Here’s an example, with each data snippet marked in bold:
Audience
Comprehension comes from research; it is identifying and understanding a problem you want to capitalize on. For example: “90% of users are abandoning from the signup form after the 3rd input.” Comprehension is composed of two snippets: the observation, and the observation method.
Who should see the experiment; specific details regarding user’s status in relation to providing meaningful behavioral feedback.
To identify a problem, first you must observe it. Dig into some data and get your hands dirty. Find something about your site that is not performing as well as you want it to. The two main avenues for research are analytics and feedback. Analytics is numeric, quantitative data on a large scale; typically numeric, statistical data that’s passively collected for later aggregation. Feedback is descriptive, anecdotal data that’s actively collected and representative of user behavior.
Response comes from design; it is proposing and targeting the change you want to make. Often, this can be approached by starting with a question, like “what will happen if we increase the size of product images in the category grid display?”
Outcome
The response is the “meat and potatoes” of your experiment; the core that will be configured in the split testing software. The snippets that form a response are: variation, location, and audience.
Desired behavioral change
The metric that determines an experiment’s success or failure. Measurement of a user action that the experiment is trying to affect. An experiment’s raison d’être.
Primary KPI
Metrics that back up the Primary KPI. These help you flesh out the story being told by the primary KPI, and/or check for unexpected user behavior. Secondary KPIs are often conversions at each stage of the funnel between the targeted area and primary KPI endpoint.
Secondary KPIs
There are three main components of a split test hypothesis: Comprehension, Response, and Outcome.
The change in user behavior that you want to be caused by the response; either encouraging a positive action or discouraging a negative action.
Hypothesis Builder
Assembled together, they read as a complete hypothesis. Now, let’s dive into each component.
Over at Corvus CRO we’re all about building little tools to help make the lives of CRO experts easier. To that end, we’ve constructed a hypothesis builder to help simplify and standardize the construction of split test hypotheses. It is a web form that takes text input for each of the necessary snippets and assembles them into a full hypothesis. You can use the tool to proof out and build your own hypotheses. There’s even a “copy to clipboard” button so you can easily grab your constructed hypothesis and paste it into your project management tool of choice. We hope you get as much use out of the tool as we do!
The tool, technique, or process you used to identify the problem. Incorporating the observation method into the hypothesis helps prove that you did your research. Examples of observation methods for each type of research include:
