A group project focused on user research, needs assessment, and usability testing.
User Experience Researcher / Usability Tester
Heuristic Evaluation / Surveys / Preference Testing / Usability Testing
Jan. - Apr. 2019
Project Summary
I was part of a team of experts who partnered with Autodesk to assess the learning and onboarding materials for their Revit LT software, a program designed for architects to model their designs in three dimensions. They were interested to learn how well supported new Revit LT users are with the learning content available to them, as well as how discoverable and useful it is for users.
The key user groups they were interested in were Revit LT users who don't have any prior CAD experience, primarily undergraduate architectural students, and architects who switch from other CAD software to Revit LT.
Research Process
Below I outlined all the research techniques we used throughout the process, what we learned, and how we applied our findings to later portions of the project.
1. Interaction Map
The team started our research creating an interaction map of the help documentation Autodesk provides to users, both online and in the Revit LT software.
Key Findings
- Most help in the software directs users to online documentation
- Many sources of online help, the large volume might overwhelm new users
2. User Interviews
Next we interviewed architecture software users to build personas that match new Revit LT users. In total we interviewed four people, two students and two professionals, who had a varying levels of Revit LT experience.
I helped recruit users, create the interview protocol, conduct two interviews, and create one of three personas.
Key Findings
- Users’ previous CAD experiences were not translating to proficiency with Revit LT
- Users preferred a walkthrough-style approach to tutorial materials
3. Competitive Evaluation
We researched five products we considered direct or partial competitors of Revit LT to compare the online learning materials each company offers. Our evaluation helped find some methods Revit LT could incorporate.
Key Findings
- Help materials were often synthesized in a combination of visual and text media
- For most products, official forums exist and are used as separate knowledge bases
- For some products, help content is tailored to different BIM proficiency levels
4. Online Survey
Our team followed the competitive evaluation up with a survey we distributed to Revit LT users, both university students and professionals. Our goals was to gather opionions about Autodesk’s online help documentation and onboarding materials from a wide audience, as well as demographic details about users, and more general preferences regarding methods of online learning.
The survey was live for 10 days and we gathered 63 complete responses. I assisted the team with distributing the survey online, and with analyzing our results to draw conclusions to present to Autodesk.
Key Findings
- YouTube videos, especially non-Autodesk created videos, are popular for learning across age groups
- There was a strong preference for documentation with videos and images, and step-by-step, guided help
- Users find Autodesk community forums more helpful than provided documentation
5. Heuristic Evaluation
Using the 10 usability heuristics from Jakob Nielsen we investigated the information structure and usability of Revit LT’s sources of help documentation. I focused on the help documentation within the software, but also evaluated the online resources and aggregated those findings with my teammates.
Key Findings
- The heuristic most critically violated was “Error prevention and recovery”
- The next most critically violated were “Match between system and the real world” and “User control and freedom”
6. Preference Testing
Next, our team conducted preference tests to assess user reactions to changes suggested by our survey and heuristic evaluation findings. The tests asked participants to which of two screens they preferred, one the current state of Revit LT’s help documentation and another a mockup our team created with small changes made.
I distributed the test to architects of varying backgrounds as a Google Form, and anaylzed our 10 responses to draw conclusions for Autodesk.
Key Findings
- Users described a preference for a clear information hierarchy when they’re searching for help online
- Users often preferred all the relevant information to be consolidated to one page
7. Usability Testing
Based on all our previous research we designed and conducted usability tests to gather first-hand data about how users use Revit LT’s help documentation. We met with five University of Michigan architecture students, sophomore to graduate level, for 30 to 60 minutes, and recorded the test audio and video for further analysis.
Key Findings
- Users often missed feature descriptions due to slow response times for hover instructions in the software
- Advanced and intermediate users noted that online documentation weren’t always in line with the models and methods they believed were relevant
- Search functionality was used more than organic browsing through the help sites, regardless of experience level
Overall Findings
At the end of our research we presented our main findings and recommendations to our counterparts at Autodesk. Our main findings were as follows:
- Regardless of users’ experience with Revit or Revit LT, the search functionality was utilized more than organic browsing through the sidebar navigation.
- Users preferred a walkthrough-style approach to tutorial materials.
- Users often overlooked the provided help documentation within Revit LT.
- Usability tests showed users of varying experience had mixed opinions about the suggested workflows and documentation examples.
Given these findings, we recommended they do the following:
- Redesign the help site to focus on searching as the first action, rather than clicking through thumbnails or navigation links.
- Make it more apparent within the software how users can learn more about using Revit LT, such as how Adobe XD helps its new users.
- Examples used to explain the how or why of the workflows should always be relevant and focused on the work that Revit LT users will be conducting.
Lessons Learned
This was a large-scale project and our research group certainly learned a great deal together. We saw recruiting users is difficult to get right for many reasons. It's hard to identify the characteristics you should search for to find research participants, and even more difficult to convince them being a participant is worth their time. Short timelines can also force you to make compromises. For instance, we ran out of time to recruit any professional architects for our usability tests, and had to settle for testing with five college students.
Additionally, this was the first project that asked me to present my UX research and design work to stakeholders with no such background. Each report we submitted needed to speak plainly but directly to our findings. UX designers and researchers never work in a vacuum, and communicating my findings to a diverse group gave me experience I can take forward to future projects.