AI-moderated user interview vs. online survey

In this post, I’ll be sharing a recent study that I conducted with real respondents that compared Perceptional and Google Forms - looking at multiple dimensions such as quality of responses, time to completion, and end user experience. While this isn’t a scientific study run by a third-party, I share our methodology and results in order to be as transparent as possible about our findings.

I hope this helps product teams and UX researchers choose the best tools for their needs. Ultimately, a new technology serves as a new tool in our user research tool belt that will allow us to improve our research outcomes in certain use cases.

‍

Bottom Line Up Front (BLUF)

Respondents using Perceptional provided up to 4.1x more characters in their response to questions (average 2.5x across all questions).

Respondents user experience with Perceptional was entirely positive (68%) or neutral (32%) compared to Google Forms which leaned towards neutral (40%) or negative (40%).

Even though all participants received the same instructions and incentive, participants using Perceptional spent an average of 2.3x more time completing their responses than participants using Google Forms indicating less likelihood to satisfice.

Methodology

To compare Perceptional and Google Forms, I recruited respondents through Prolific, a third-party platform that allows researchers to easily access vetted research participants. All participants were provided with the same study description, recommended time to completion (7 minutes; 420 seconds), and the same modest incentive for their completion. 

In order to cover a research topic that would be of interest, I decided to explore respondents’ experience with the use of AI in the workplace. I recruited 50 participants in total using Prolific, with 25 using Perceptional and 25 using Google Forms. These participants were working professionals aged 23 to 40 who use AI tools at work more than once a week.

In each platform, participants were asked about the use of AI in their workplace with two additional questions at the end about their experience with the assigned survey platform (Perceptional or Google Forms). This method allowed us to gather both qualitative data on their workplace experiences and direct feedback on their experience with each platform.

Results

When comparing the qualitative data collected, Perceptional clearly stood out over a leading online survey platform in terms of response richness and depth. Here’s a summary of the key metrics:

‍

Based on these results, and by providing the same research brief and asking the same questions, respondents using Perceptional provided up to 4.1x more characters in their response to questions (average 2.5x across all questions). Not only that, but responses in Perceptional were more in-depth than responses in Google Forms for every single question, with the minimum character difference being 150% and the maximum character difference being 410%.

The depth of responses with Perceptional was also richer. Participants engaged more thoroughly with the AI-moderated questions, often sharing context, personal experiences, and nuanced opinions. In contrast, responses from Google Forms tended to be shorter and less detailed - potentially due to respondents’ tendency to satisfice within a static survey format. For example, question 5 asked respondents: “What are your thoughts on the benefits of AI in your professional life?” For this question, Perceptional resulted in 3.2x more characters than Google Forms. The responses with the median character count for this question illustrate this:

• Perceptional: “If I put in the right commands, I could ask AI to help me with my busywork, or correct small grammar spelling mistakes when I'm speeding through writing emails”
• Google Forms: “The main benefits are convenience and time-saving.”

‍

Time

When comparing the time taken to complete, there were clear differences between Perceptional and a leading online survey platform. Here’s a summary of the key metrics:

‍

The data shows that participants using Perceptional spent an average of 2.3x more time completing their responses than participants using Google Forms. This extra completion time is expected, and is directly reflected in the richness of the responses in Perceptional (2.4x more characters).

Keep in mind that all participants were provided with a recommended completion time (7 minutes; 420 seconds) and the same monetary incentive. This may illustrate that respondents using Google Forms are more likely to satisfice in order to complete their survey in a shorter time (average response is 57% of the recommended completion time) compared to respondents using Perceptional (average response is 130% of the recommended completion time).

In summary, Perceptional may take longer to complete, but the trade-off is a significant improvement in the quality of the insights gathered.

User Experience 

To assess the user experience with each survey platform, respondents were asked to 1) provide feedback on their experience with the survey platform; and 2) provide feedback on how the platform compares to others they have experienced. These two questions were presented as required questions at the end of each survey.

‍

The sentiment analysis shows a clear difference in user experience between the two platforms. The sentiment analysis was done by using ChatGPT to categorize each response (direct quote from the survey for questions 8 & 9) into a positive, neutral, or negative category. As can be seen, Perceptional enjoys a much higher percentage of positive feedback at 66.7%, with the remaining 33.3% being neutral and no negative feedback at all. This demonstrates that users generally have a more favorable experience with Perceptional's AI-moderated interviews compared to the more traditional survey format.

While these results are promising, this positive experience may be due to the novelty of AI-moderated user interviews compared to more traditional survey experiences. Our team at Perceptional will have to ensure we continue iterating on our user experience over time to ensure that participants using our platform continue to enjoy their experience while providing rich responses.

With the online survey platform feedback, some sample responses included:

• "Positive. Everything was very clear and went smoothly."
• "This experience has been fine."
• "More interesting questions."

Whereas for the Perceptional, feedback, some sample responses included:

• "It seemed very good and almost human like."
• "It was easy to use. The comments and questions seemed adapted for my responses."
• "Fine. It felt conversational. The tool quoted one of my answers in a way that felt authentic. The flow of questions was useful for organizing my thoughts."

I was happy to see respondents provide such answers without them receiving any prior context about Perceptional (i.e. no access to marketing materials, link to our website, etc.). The responses illustrate Perceptional’s strengths - that an AI-moderated interview could have the context-awareness and adaptability to compare to a human-moderated interview.

These findings are very exciting and promising since we are still early in our journey to develop Perceptional - and it’s only going to get better from here!

Research Findings

While this isn’t related to the purpose of the article, I thought I’d also share some of the interesting findings from respondents about their use of AI in the workplace. And what better way to do it than to share directly from Perceptional’s results analysis page (taking into account only the responses completed using Perceptional).

Below are some snippets from Perceptional’s Key Findings (which is an AI-generated summary of all respondent transcripts):

• AI Usage in the Workplace: "The data reveals a consistent trend where AI tools such as ChatGPT, Bing Chat, Microsoft Copilot, and GitHub Copilot are significantly enhancing work performance across various fields, including coding and content creation. Users credit these tools with streamlining tasks, boosting efficiency, and in some cases, helping overcome learning difficulties like ADHD and dyslexia."
• Common Themes: "One emergent theme is the varying impact AI has on different users. While some report a dramatic improvement in their professional capabilities, others experience a more modest enhancement to their work habits, with concerns about the potential loss of a unique brand voice and over-reliance on AI leading to inconsistency."
• Opportunities and Challenges: "There's a notable interest in further AI integration in professional and personal spheres within the next five years, yet users are expressing a need for more support and training, specifically in AI applications for data analysis. The mention of a desire for customized AI tools tailored to workplace standards underscores the appetite for more personalized AI solutions. Adapting AI tools to individual and organizational needs, coupled with adequate training, could be the next step in fully realizing their potential."

And below are two Perceptional user interview summaries (which are automatically generated AI summaries of each individual transcript):

• User Interview Summary 1: "The user has extensive experience with various AI tools including Gemini Advanced and ChatGPT 3.5 and 4, which are integral to their work in AI development and testing. They find AI tools extremely helpful, particularly in overcoming learning difficulties like ADHD and dyslexia. While the user enjoys the convenience of remote work, they note a sense of loneliness due to less in-person interaction with coworkers. Looking ahead, the user hopes to maintain or advance in their field."
• User Interview Summary 2: "The user utilizes AI tools like ChatGPT and AI-powered photo editors to enhance work performance, particularly for content creation and idea generation. They cited a lack of company support in AI training, specifically expressing interest in AI for data analysis. AI tools streamline brainstorming and scheduling for them but pose a risk to their brand's unique voice. The user foresees a more integrated AI role in their professional and personal life within five years."

Conclusion

Before starting this study, I used to claim that Perceptional was more superior at collecting qualitative data than traditional online survey platforms. I also claimed that Perceptional fits somewhere in between a traditional online survey and a human-moderated user interview. I made those claims based on conversations I had during my validation interviews, customer walkthroughs, and other demos at pitch or startup events. 

Now, I have data to back up my claims! 

By providing participants with the same incentive, study description, questions, and time to completion, respondents using Perceptional provided up to 4.1x more characters in their response to questions (average 2.5x across all questions). While they took 2.3x more time - respondents using Perceptional still had a substantially more positive user experience.

And not only was the character count higher, the responses provided were much more in-depth and actionable. I can’t wait to continue improving the Perceptional platform - giving both our users and their respondents more options for engagement, distribution, and analysis of their studies.

Want to experience Perceptional for your next research project? Visit our homepage to get started with a demo and free trial.