Earlier in the year I had the good fortune of presenting to a class from the University of Technology, Sydney’s Interaction Design course. As someone who occasionally hires designers, experience in user testing and a sincere integration of users in the design process is what makes candidates stand out. Why? Because this is what reduces errors, minimizes IT and build change requests and helps ensure users can understand and use our products. It’s a mistake to think something has to be detailed and almost production-ready to be tested. Test ideas, test sketches, test digital, test services. Test early, iteratively and often.
Design Thinking Drinks is an event organised by Deborah Kneeshaw and sponsored by Thoughtworks. It’s on every couple of months and last week’s event attracted a big and curious crowd for Chris Vanstone design co-lead of The Australian Centre for Social Innovation (TACSI) and one of the founders of agency In With For.
Chris Vanstone – previously a founding member of Participle and participant in the UK Design Council’s RED service design program which involved design agencies in social welfare, health and public housing design projects. The RED project provided some fascinating case studies and Chris has brought that experience to Australia. South Australian to be exact whose government provided seed funding for TASCI.
Develop “social start-ups” with design thinking methods:
- Defining the problem/s with evidence based and participatory research. Conceive the solution from the problems evident in the data (from the ground up) as opposed to prescribing them with policy from the “top down”.
- Prototyping and iterative stages to both service delivery and tools used e.g. test what training is required, prototype service interaction to develop service blueprints, test the skills and capabilities required for roles, test and iterate measurement tools
- Follow various stages of incubation to launch fully fledged social services
- Build teams around the solution that can keep iterating and developing the service
That alternatives to government funding will provide resources for incubated services to continue and for the “business models” of the social start-ups to stand-up as self-sustaining and self-funding entities. It will be interesting to see what happens when the South Australian government seed funding ceases.
This was an energising presentation and In With For’s model of mixing the expertise of design with business development and social science demonstrated compelling stories.
My main take-away from the night was to pursue methods of measurement for service design projects as a design challenge in their own right.
Read more about the talk and the case studies presented over at the official event blog (it has much nicer photos too): http://designthinkingsydney.wordpress.com/2012/07/20/correction-chris-vanstone-at-design-thinking-drinks/
I recently finished a project where I conducted user testing to validate the effectiveness of a navigation menu. The project was a collaboration with the client’s project team who were responsible for the prototype and the recruitment. Everyone was confident going in to the user testing on the IA scheme but were open to changes. This may seem a mute point—why do testing if you are not going to change anything? Strangely I have seen people be highly selective of what they wanted to have proven in testing. Luckily this project featured no such hubris and everyone was respectful of the problems encountered by the users.
Elsewhere in the organisation other stakeholders held competing and contrasting views of what needed to be designed in the schema and what labels needed to be used. User testing the IA was seen as a means to streamline and manage the internal decision making process by bringing everyone together on the same page and letting users themselves determine the outcome.
The project was a simple engagement but an important one in an organisation that is seeking to embed UCD and customer centric thinking. The client team invited their colleagues to witness proceedings. Luckily during the testing itself I completely forgot that up to 17 people were watching me! time was scheduled in between the user testing sessions to debrief. I collected the thoughts of the observers; their impressions, implications and design ideas. The debriefing sessions were important. Not only did I benefit from the input of subject matter experts, but members of the organisation got to see the process itself. I was able to field questions about the method, diffuse doubts on the spot and collectively we arrived towards the findings. When it came to the presentation of the recommendations there were few surprises and high engagement.
The testing was not about flogging a dead horse to validate how much something did or didn’t work. Sufficient time was left between sessions for important changes to be made and tested. Stakeholders learnt by observing the process that user testing is not a research exercise. It is a design process. One that is more effective, more pertinent and faster than design by committee.
What are diary studies?
Diary studies, otherwise known as User Research Diaries or “Cultural probes” were pioneered for use in design research by William “Bill” Gaver, Professor at Goldsmiths London. Interestingly he doesn’t analyse diary content, nor does he create scenarios or personas from them instead using them as a base from which to validate other data. He does not create personas, preferring instead to revisit the raw data.
What are diary studies good for?
- Great for understanding the activities undertaken by participants, what they actually do.
- Opportunity to witness subtle behaviour as participant can note activities they may not otherwise recall in a typical research interview, or contextual inquiry
- Better than just a contextual inquiry for understanding the amount of separate actions and activities undertaken
- Great for data on customer tasks and finding out the detail in these sequences
- Great for witnessing all of the channels/touch points a customer encounters — so its perfect for service design, and the gathering of data for customer journeys
Recruit carefully for diary studies
- Allow a little longer than usual for recruitment. It’s more important to recruit well than to recruit on time.
- Pay participants more. More effort is required of them.
- Screen participants yourself. Trust your first impression. If you don’t think they will be suitable, don’t recruit them.
- Note the anticipated schedule of activities of the participant in the screening process i.e. get an idea of what activities they plan to do over the period of your study just in case they are lying and saying yes just for the money!
- It is harder to find participants who are willing to go to the effort of filling out a diary every day – so take the time getting the right people.
- Brief the participant about what effort is involved
- Their information is confidential
- Their identity will not be exposed
- They are free to withdraw from the study at any time
Brief participants about what is involved
- Instruct participants how to use the diaries
- Remind participants what EFFORT is involved, the level of effort expected of them
- Give participants a checklist of the materials they need to return
- Provide written instructions – they will read it, and it saves you explaining at length
- If you can its better to have all participants start on the same day. Makes your job easier later when you have to track their diaries down.
- Don’t be leading about what content you are after of them, written, photographic or otherwise.
Prepare diary materials thoughtfully
When preparing diary packs, prepare a diary for each week of the study, camera (optional), instructions, reply paid envelopes.
You need to make a few decisions upfront — do you want to use structured or unstructured diaries? Unstructured diaries give the participant a blank slate. Structured diaries set some questions or areas for response. I would recommend structured diaries. They give people a sense of what you want, otherwise it can be too hard for them to start. Consider including a survey, it gets people used to filling something in, as well as gathering useful information.
- Label the diaries so you can tell who they came from when you receive them
- Offer participants a choice between a paper diary OR digital diary.
- Identify the participants
Choosing between paper and digital diaries
Paper diaries are good because they’re low fi, low fuss. Paper diaries are bad because:
- You ideally want people to carry them around and record things on the spot, but you can’t really tell if they do or not.
- They have to go to the effort of posting it back to you
- Participants tend to contribute longer answers
- Easy set up – I’ve used microblogging service Posterous in the past (sadly no longer around). A colleague has even given someone a Word document to email back.
Using microblogging and digital diary services:
- Email submission is as good as instant
- Easy for participants to include photos
- Easy to track participants
- Easy for participants to include you in on relevant emails and send you relevant web pages
- You can use comments to ask questions
- It’s not suitable for less technical participants
- It’s harder to set up a structured diary
- Because its unstructured, there can be less detail as to the “why” behind the activities.
Conducting photo diaries and providing cameras
- Photos bring the participant to life, and clients love them
- Great at showcasing the absurd (e.g. paperwork involved in financial processes)
- BUT people don’t know what to photograph. Let them know what you are looking for
- Don’t expect too many photos
- Why don’t people take photos? – people feel like spies, scared they could get in trouble, its an unnatural behaviour.
- Getting people to post back a memory card
What hasn’t worked
- Getting participants to email photos. Uploading is a chore!
Expect difficulty getting materials back
In 2005-06 Australia post delivered 94.9% of letters on time or early. Despite this many* diary participants in studies I have been involved with so far did not post their diaries back on time. Expect most diaries to come 2 days – 1.5 weeks later than they should. Expect that when you phone your participants they will say “it’s in the post”. Participants will lie about getting their homework done, and sending it to you.
- Pay on receipt! But make a good faith payment before the participant starts, on receipt of the diary materials. No diary, no payment.
- Use reply paid EXPRESS envelopes for paper diaries. They arrive quicker, they’re traceable and they are a sign that the diaries need to be returned pronto!
Touch base with participants throughout the study
Debrief entries with participants. If conducting a long study, encourage them to participate more. Keep a log file to track volume of entries, who is contributing, who is not.
BUT remember that people have day jobs. its really hard to get in touch with participants during the day. You kind of get the sense that they may be avoiding you!
Ask them when its best to contact them when you are recruiting them. Be prepared that half of the effort in the project will be getting hold of participants.
- Ask the best time to call
- Interfere as little as possible
- Offer gentle reminders
- Spam your participants. Remember this is meant to be unobtrusive research.
Set some creative tasks
Tasks are fun and people are surprisingly creative with their responses. Tasks are great for seeing what the participant thinks about the process they are undertaking and great for eliciting their attitude, their emotions, and how they see themselves in the process. The results provide powerful metaphors for the whole experience.
Example tasks you can set:
- If this process was an animal what would it be?
- What celebrity best represents you in the process Quote: “I feel like Jennifer Anniston. I’m on an emotional rollercoaster.”
- What object best describes the main person you are dealing with: “My xxxxx is like a stream locomotive. I know they will get there … eventually!”
- Set tasks throughout the study, not all at the end
- Expect everyone to get it, or to comply
End the study with an interview
Use diary studies to GROUND the participant in their actions during the interview or contextual inquiry.
- Expect to yield insights from the diaries alone.
- Prepare your interview questions from the diaries as you receive them
- Pick up the last diary at the interview
- Arrange the final time for the interview upfront
- Ensure the interview is conducted in the most appropriate environment, you still want to see the participant “in situ”
This blog post was taken from a presentation I gave at Different’s Friday Talks. Friday talks are our little learning and sharing fest.
All drawings were done by my lovely boyfriend Colin Stokes. Diary war stories came courtesy of my experiences and those of Matthew Ballesteros, Jason Crane and Vicki Lane at Different.
Recommended reading on diary studies
- http://www.sleepsurvey.net.au/the-sleep-survey/about/ [web archive]
The original presentation has also been published on Slide share.
Correction and updates
* Thanks again to Vicki who pointed out to me that many of her participants did comply and did send diaries back on time. I had previously stated that all participants were late. Maybe I have just been unlucky.
Earlier versions of this article wrote about using micro blogging tool Posterous for digital diary studies. Sadly, what was the best most stable microblogging service was acquired and disappeared soon after. Since then digital diary study software has emerged. Try it out or see if the likes of Tumblr works for you. The tool with the least steps and most minimal learning curve will always be best.
Earlier this week Lauren Tan presented at Service Design Drinks on her university research paper. In it she looked at 2007 DOTT (Design of the Times, internet archive link, may not be complete site) design projects in the public and social space.
“This PhD programme aims to identify and understand how design methodology is used in the public and social sector and the contributions it can make to the broader context of sustainable development.”
— Lauren Tan. Reference: http://northumbria.academia.edu/LaurenTan
Tan researched the methodology of a number of agencies and within the context of service design found the role of the designer to be that of a:
- social entrepreneur
- capability builder
Lauren’s talk gave everyone the opportunity to compare their work with what is being done overseas. Lauren ran us through 2 case studies.
The agency: Thinkpublic
Design role: Designer as co-creator
The problem: The difficulty faced by those recently diagnosed with Alzheimer’s and their support network of learning about the services available to them.
The challenge: Foreseeing the future of dementia services.
The approach: Co-design workshops with service provider stakeholders, researchers, carers and sufferers. These workshops were preceded by an extensive research phase that defined the themes of the design project.
The deliverable: Recommendations for a dementia adviser concierge service. A poster for carers to navigate the service network.
The outcome: Two years after the conclusion of the project the UK government published a strategy document. One recommendation, for a dementia advisor to facilitate easy access to care, support and advice following diagnosis, directly spoke to the research project i.e. the project was written into government policy. The Dementia Adviser Service is now being rolled out in the UK by the Alzheimer’s Society.
(internet archive link, may not be complete site)
The agency: Live|Work
Design role: Designer as provocateur
The problem: Reduce carbon footprints, while not compromising heating needs, and while tackling fuel poverty.
The challenge: By 2016 all homes to have a net zero carbon foot print with all carbon output offset by household activities.
The approach: Design research with residents of Castle Terrace.
The deliverable: A financial product concept articulated through a scenario. Called “Saverbox” it is an interest free energy loan for energy saving measures. Repayments are based on actual energy savings.
The outcome: Several years after the project a government agency released a similar product for small business.
Ultimately what this talk spoke to was the growing scope of design. User experience design and strategy is slowly colonising problems belonging to other fields. It can do so effectively with the inclusion of the user in defining the problem and sometimes even helping to design the solution and through the use of design tools to communicate its outcomes.
- Lauren Tan speaks more on research methodologies (internet archive link).
Matt Hogdson, awsome conference speaker, and writer of things IA and UX, and agile posted a blog about UCD. Intrigued I asked a question in the comments, and got a whole blog post as a response on getting stakeholders and users together in a workshop for co-design magic! Wow!
It’s a good read to boot. If you haven’t been so inclined as to click on any of the other links referenced thus far check it out: http://magia3e.wordpress.com/2011/02/21/ucd-is-getting-users-and-stakeholders-together/