Lean user testing with Meetup 4
One of the most memorable events from the Brooklyn Beta conference was at the Meetup offices in Manhattan. We were walked through a fascinating overview of their testing process, what worked and what didn’t.
As a designer with no real previous experience of user testing before I joined Netlife, I have since participated in half a dozen all-day user tests comprising 5 people each, either as observer or moderator. These tests have usually occurred towards the end of the design process.
Last year, Meetup did around 400 individual tests. That’s at least one every day on average.
Meetup’s processes and opinions on user testing:
- Meetup has one full time employee in charge of (moderating) all user tests.
- They almost always have at least one test person a day either coming to the office in person or available remotely on GoToMeeting.
- They send invitations to observe on GoToMeeting to anyone who needs it and can’t be there. Designer, developer, project manager etc.
- Anyone at the company can request to have something tested, whenever. “Is someone coming in today? … Drop this in the test”.
- Eyetracking? It’s “a waste of time”. It may look fancy, but do you really need it to tell you a button isn’t working?
- Formal reports are also mostly a waste of time. By all means take notes, but all the people to whom the test is meaningful should be able to observe, and resulting changes be made soon afterwards.
- Test anything – small details, individual buttons or rough experimental concepts the moment they are conceived. Instead of a designer asking themselves “could this work?”, test it! Paper with post-its, mockup, html prototype, whatever works.
- One test person will typically stay for an hour and be presented with 4 or 5 different scenarios to test for roughly 10 minutes each.
- Go low tech and simple. You don’t have to buy an expensive Noldus mobile camera when you can gaffa tape up a webcam on a tripod (as they do). Too expensive to bring people into the office? Use GoToMeeting. Can’t stream video into a separate room? Have an observer sit in the same room.
How I think we can integrate this approach
Of course, a consultancy such as ours cannot be directly compared to a product company such as Meetup, but I think this sort of test-often-and-imperfectly philosophy should be applied to consultants on client projects too.
- We should have a dedicated “Test manager” who can organise and if not necessarily moderate, at least be involved in every test. This will ensure testing becomes regular, consistent, efficient and second-nature.
- Testing should be more spontaneous, and if someone involved such as the client or developer can’t be there physically, do the test anyway and send them a link to follow on GoToMeeting.
- Have a more regular “stream” of user testers coming in. Actively ask employees if they need something tested – anything.
- With regular test persons who are not necessarily booked for one project, we can present them with multiple client projects (2 or 3 for example) per test.
- Eye tracking is great for figuring out why users are having difficulties, but sometimes you just need to quickly test a concept or detail to see whether or not it works and quickly iterate from that.
- A test lab is not a natural environment for users – would it be more natural to sit on a sofa with a tablet? Maybe the lab should have a sofa!
Most importantly: as designers, we need to be more aware of time lost from asking ourselves if a concept will work or not – just get it tested.
Meetup’s “$30” mobile test setup
BoingBoing also blogged about Meetup’s testing philosophy in 2008