UX Assessment Standardization Pilot
In 2012 I rejoined Intel’s central market research group and was asked with another collegue to build out a formalized UX Research function. Our first task was to develop a repeatable methdology for pre-launch user experience testing based on best practices. We used a request to do some competitive analysis of Google Cloud and Apple iCloud to test out an overall framework for UX assessment, as well as to test specific tools re: physical response testing (e.g., eye tracking, microexpression analysis, emotional response sorts) that we might incorporate.
01 Background and Goals
To meet our goal of developing a methodology which we could standardize around, we needed a project to use as a proof of concept. So we used a request to do some competitive analysis of Google Cloud and Apple iCloud to test out an overall framework for UX assessment, as well as to evaluate some specific tools re: emotional response testing.
After a good deal of literature review, we felt like we wanted a methodology that captured the subjective nature of users’ perception of their experience. Measures like task completion and time to task completion may be objective, but measures that rely on satisfaction (like NPS) deal with not only the reality of the interaction, but the perception as viewed through the lens of the user’s expectations. And we knew that as Intel, we’d need to work that much harder to drive perceptions of satisfaction.
02 Methodology
We recruited 24 representative PC and Mac users who were “cloud novices” into our testing facility in Hillsboro, OR. Half were put in the analytical cell to evaluate iCloud, and the other half were to evaluate Google Cloud.
The initial task was to review a set of materials (eg articles, ads) where the user was to educate themselves on iCloud/Google Cloud, followed by a brief phone interview where we captured their anticipated experience with the service.
This was followed by an in-person evaluation where we tested them in the course of a series of exercises described on the right. Testing included the following:
Standard HCI measures like time to task completion, % completion
Eye tracking
Capture of mental map of cloud service
Overall experience measures, including SUS and NPS
Video capture for later microexpression coding
2 of 8 other emotional response measurement techniques we were evaluating
Within two weeks we recontacted the respondents to capture their recollection of the experience in a brief questionnaire, including second readings of NPS and their perceptions of the experience.
03 Insights
We gained a lot of valuable competitive insight in this study that we were able to put to immediate use in the design of Intel’s own consumer cloud service. Most significantly, in these early days of the consumer cloud, most users struggle to form mental models of these services. iCloud in particular had an overly complex setup process at the time which still largely left users in the dark as to how their personal cloud was configured, when and whether their files were synchronized or where they resided. Users felt a lack of both transparency and control over the process. Also once the service was configured, there was no central place to make changes.
As for the emotional response testing, my Director had high hopes for applying some of the techniques others were having in marketing research to the realm of UX evaluation. However, aside from eye tracking most of the techniques available at the time had technical challenges that made them difficult to use in a lab environment, and microexpression measurement with a human coder proved cost prohibitive.
04 Actionability
Results of this benchmarking went directly into the design of a consumer cloud service we were working on at the time, with an emphasis on onboarding, centralization of controls, and giving users better queues as to the status/availability of synchronized files.
As a UXR team, we were also able to kick the tires on a number of new UX evaluation techniques techniques in order to settle on a template our team could use for future evaluations. And getting everyone involved in the process gave us an opportunity to build a consensus on the methods we would use as a team for the next 4 years.
05 My Learnings
Value of Benchmarking - To our development team, Apple was the 800 lb gorilla of user experience. Doing this testing brought our perceptions of the competition back down to earth and showed real opportunities to develop an experience which was an improvement over what they had delivered.
Inclusion and New Team “forming/storming/norming” - While we were executing this study my co-lead and I were in the process of building up an entirely new UXR team and capability for Intel. Getting everyone involved allowed us to pull in everyone’s experiences as we were developing the norms of how our new team was going to conduct UX evaluations going forward.