Chatbot Experiment.
2021
CONTEXT:
The conversational marketing team were experimenting with a group of users, exposing them to a chatbot first instead of immediately having access to human-led support.
We wanted to ensure that this change did not affect the customer experience.
HYPOTHESIS:
We believe customers default to chatting with support when they need help because it is faster than looking for resources to answer their questions independently.
We believe this because previous research found that chat is not seen as a formal support medium and is preferred because of the speed of interaction. Customers often begin to chat and then self-serve simultaneously, rather than attempting to self-serve first.
By focusing on encouraging self-service, we can reduce ticket volume while enabling customers to find solutions, quickly.
RESEARCH APPROACH + IMPACT
-
We wanted to ensure that we understood the impact of introducing the chatbot experience for the customer base.
Customer satisfaction and trust in using the chatbot was paramount as it will not only help the team understand what needs to be improved but to link the customer story to the quantitative data that is coming through from the experiement.
-
We decided to perform a longitudinal study over the course of 2 months.
SENTIMENT REVIEW:
We benchmarked the experience level before the experiment started using a lightweight 'SUS' score to measure the user experience before and after the experiment was released.
INTERVIEWS:
We also interviewed the users involved in the experiment to understand what their experience was like during the experiment.BEHAVIOUR:
The purpose of the experiment was to understand the impact of a chatbot on support ticket volume. Therefore the research also focused on ticket volume and ticket type analysis before and after the experiment was released.
-
The results illustrated that users didn't really notice the difference in their experience.
The results from both the experiment as well as the research showed that the experience was not depreciated with the experience change which gave the team confidence in productising the experience.
Our SUS score illustrated a minimal score change with the overall usability. We saw a -2.3% difference in SUS score after the chat bot friction was added to the help process.
Margin of error was within the 95% range of acceptability which further enabled the confidence in productising the chatbot experience.
E2E Redesign:
2024
Redesiging how we onboard our Free & Starter customer base
CONTEXT:
The overarching aim is to "Make every new B2B business a success story". This research supports the relaunch of the HubSpot low-end experience, which aligns with a new mission and vision for the new ‘low-end‘ persona. The vision is for the HubSpot low-end experience to be the place business builders come to start and scale their business, empowering them to do more with less in less time, and building a thriving content and community ecosystem to help them grow their careers.