Lessons from 15 A/B Tests

Seeking to improve the form conversion rates and overall user experience of the broker hand-off form on MoneySuperMarket's Mortgages channel, as part of a small team, we set out to experiment and iterate over the span of two weeks.

Role
UX / UI Design
Timeline & Year
2 weeks
2023
category
FinTech
Mortgages
skills
Usability Research
Rapid Prototyping
Concept / Preference Testing
Visual Design
Copywriting
TOOLS
Figma
HotJar
Optimizely
Google Dashboard

Project Overview

On MoneySuperMarket's Mortgages channel, we embarked on experimenting on the broker hand-off form. This form is necessary once users click out on a product that can only be fulfilled through the broker partners or where the user specifically chooses to go through an intermediary instead of dealing directly with the lender.

What we set out to achieve
This project aimed to increase business metrics and form conversions by enhancing user experience and interface design.

How we approached this project
Throughout the testing process, a total of 15 unique iterations were designed and tested, with valuable insights gathered through surveys, concept testing, observing user comments and behaviour, and digging into analytics. The insights played a crucial role in driving iterative design and experience changes, ensuring that user feedback and analysis were at the forefront of decision-making.

business goals

We set out with the overarching goal of elevating both form completion rates and site interaction, ultimately driving an increase in downstream broker applications. To achieve this, we implemented a variety of activities and tested a range of unique ideas throughout our iterations.

Increase form completion rate
Boost site interaction
Drive downstream broker applications

Activities

The testing process involved collecting user feedback, analysing usability metrics, and implementing iterative design changes based on these findings.

→ Core elements:

Rapid prototyping
Taking "rapid" seriously, I designed over 17 variations during our two-week process, of which 15 got developed and tested at the time.
Expanding the journey
To prevent users going on a dead-end journey in case they choose not the speak to a broker, we created a new page tailored to this segment specifically.
Cross-platform focus
Keeping in mind that desktop and mobile users might have different preferences, we carefully observed user behaviour broken down by device to optimise accordingly.

→ Usability research:

Preference testing

The testing process involved collecting user feedback, analysing usability metrics, and implementing iterative design changes based on these findings.

Comments & recordings

The testing process involved collecting user feedback, analysing usability metrics, and implementing iterative design changes based on these findings.

Analytics

The testing process involved collecting user feedback, analysing usability metrics, and implementing iterative design changes based on these findings.

Preference testing
To gauge what users they preferred, I also conducted testing through HotJar to support (or refute!) what our analytical insights showed.
Comments & recordings
We observed recordings and incoming feedback on HotJar to evaluate further qualitative feedback.
Analytics
Whilst we used Optimisely for our multiple ongoing A/B tests, the number-crunching was done through Google Dashboard.

THEORIES, tests & learnings

Below you can review 10 summarised lessons categorised by our driving theories, along with relevant test screenshots.

Theory: Highlighting process steps cultivates user confidence and increases conversion.

By setting expectations and easing any uncertainty around the process, we expected that offering clear guidance and transparency would foster user trust and confidence.

Lesson 1:

✗ Step boxes performed 25% worse when shown at the beginning of the form.

One insight we gained was that step boxes were found to be less effective at the beginning of the form.

These were subsequently relocated to the confirmation page as concept testing confirmed users preferred to see these at some point in the journey.

Lesson 2:

✓ Showing a progress bar performed 22% better on desktop, but 18% worse on mobile.

Based on these findings, in order to enhance user experience according to the device being used, we've decided to include a progress bar exclusively for desktop devices.

Theory: Showcasing the benefits of using a broker will increase form completion.

Emphasising the unique value proposition was expected to encourage users to perceive the form completion process (and the broker service) as valuable, thereby enhancing the mortgage application journey.

Lesson 3:

✓ Highlighting broker benefits when doing a product switch boosted conversions by 24%.

Current lender-specific pages featuring an integrated benefits carousel outperformed the previous winning variant we tested.

Theory: Users are likely to interact better if we choose their preferred wording.

Mindful of the branding's potential influence and the impact of copywriting on conversions, we set out to establish a unified and consistent flow across all pages within the journey.

Lesson 5:

✓ Using "Mortgage Service" wording performed 9.8% better than "Broker Service."

Early on, one of our tests focused on the service offering's wording to gauge which phrase would sit better with people. The results confirmed that users preferred the term "Mortgage Service."

Lesson 6:

✓ Moving the "your selected mortgage' product card to the bottom of the form and slimming it down also improved performance.

Initially, our form featured a standard product card at the top, potentially causing unnecessary scrolling and disrupting the user flow. To alleviate this, I redesigned a slimmer product card and relocated it to the bottom of the form. Maintaining the card's original functionality, clicking on it still triggered an overlay with additional product details.

Theory: Tailoring content in distinct journeys will enhance overall site engagement.

Anticipating that a personalised approach would resonate with users, our strategy aimed to boost interaction by delivering relevant content within different user journeys.

Lesson 7:

⤤⤦ The first multi-journey page had a drop-out rate of just under 50%.

After surveying our users, 24% indicated readiness to engage with a broker by answering "Yes". They were then directed to the standard form, enabling them to provide their details and connect with a broker.

The remaining 26% clicked on "Not yet." This took them to the second journey page, where they could review their options and browse resources such as guides or calculate how much they might be able to borrow from lenders.

Lesson 8:

✗ The "Not yet" page showed varied interactions, but in terms of broker leads, the multi-journey variant converted 24% less than our winning version at the time.

Compared to a single one-page form, the multi-journey variant performed worse.

However, this did not take into account the possible long-term improvements or Decision in Principle conversions.

Lesson 9:

✓ False door testing resulted in 22.76% clicks.

We included a false door test on the second page to gauge users' interest in a direct Decision in Principle entry point to NatWest.

This part of the page ended up drawing the highest numbers of clicks, so we could confirm there was an appetite for a functional feature.

Lesson 10:

✗ The page with a coloured background performed 28% worse compared to the one with a simple white background.

Our theory for this was that the page may have not appeared consistent with the rest of the broker form journey, which could have potentially raised user concern.

resultS

Leveraging insights gained from our learnings, we integrated the most successful features into a final product.
This optimised variant yielded a remarkable 16% increase in form conversion, directly contributing to a significant improvement in our financial bottom line.

→ Integrated features:

Implemented a progress bar exclusively for desktop users, whilst maintaining an optimal experience without one on mobile devices.

Leveraged the "Mortgage Service" terminology to cater to user preferences.

Based on the findings from our false door testing, we implemented a Decision in Principle click-out.

A multi-journey form was introduced for users exploring new mortgages, complemented by a simplified form tailored specifically for remortgaging users.

Employed a white background across forms to ensure a consistent and cohesive user experience.

Relocated the product card to the bottom and streamlined its design for a more refined and efficient user interface.

Winning variant

You can view a recording of the purchase-specific broker form below: