CODY HANSON: A lot to cover -- posting a lot of raw data on http://groups.drupal.org/usability -- How this got started -- He was putting together academic sites to find materials. Went to Barcelona, and noticed that Dries had prioritized usability. After 4th time hearing usability, he realized that they have a state of the art usability lab at the University of Minnesota. Approached Dries, and it got started from there. Excited about the network effect of this usability work.
CODY HANSON: Why would we do formal usability testing? Main reason is because none of us can "unlearn" how to use Drupal or forget what a node settings. People who care about Drupal the most, can't use it again for the first time. Got some volunteers w/ a one-way glass and an eye movement, mouse movement and cameras. They watched them, and it was frustrating to see how hard it was for the evaluators. They would watch what they were doing and saying out loud. The usability group had discussion about what they were going to test and how to test it. Drupal is highly customizable, and so they tested what would be relevant to most users -- CCK + core and the Garland theme. Tasks to get into CCK fields, and deal w/ user roles and permissions, and use taxonomies -- librarians, menu system and blocks. Be careful about tasks -- had to match their mental model of the evaluator. Like saying "Page" not "node." Have to avoid Drupal terminology -- otherwise it becomes a Word-find task. They also talked about "Personas" -- Use four spare different types: 1.) anonymous 2.) Content contrib 3.) site maintainer 4.) Site Admin It was too broad for everyone, and so they settled on the site admin who would deal with site admin and content admin. Finding evaluators, find ppl with experience with similar CMS type software -- movable type, Wordpress, but NOT used Drupal. These are "our" people -- not our mom, or our grandma who you expect to have problems. These are people we want to use Drupal, and could use Drupal
KAREN STEVENSON: What we see. Admin screen looks like a busy picture of Where's Waldo. 'Yowza!' -- open up node page and see all comment settings First task -- Started with a hard task -- could've been easier. First create a form to add a new content type and add new fields. Went to "Site Building" -- 'Content management doesn't sound right, maybe blocks.' Went to over and over again, but nothing on that site building page gave them a clue -- looking for "Form" or "Field" Unfamiliar language like "Content Type" Finally got to the content management panel -- Use "content" over and over again. Can't find what they're looking for, and went from content management panel and back to site building Got all hung up on what was a "Story" Not a single person got there without calling the help desk. One person backed out from the content types panel. The content types panel doesn't have any words that indicate that. They were looking for "previews, and didn't find it. Eye Tracking NEVER saw the Upper Right "FIELDS" screen. Clearly there's some problem with tabs -- theme is probably some issue Want to create forms, but don't know where they live. Completely confused by the word "content type" They were completely page by the Add content Type -- and what it meant. Mostly never found the fields tab, as an aside, it didn't even work when they got there. Were confused, and didn't know what they were doing, and then they started clicking EVERYTHING in site.
ANGIE BYRON: Personally shocked by that 6 of them thought that Content Types were Fields, and they were trying to get them back into the earlier created page.
KAREN STEVENSON: A page, hung onto that. [But they were wrong] Am I creating a page, thing for the page, confused. Confused page vs. story -- they thought they were the only options, and didn't think they could create it. They thought of page as something to create and put blocks and other stuff in. No one knew what story was. After 35 minutes after being pointed to it. Got them to the page to add a field, but they could never find it. They didn't notice the names -- CCK Field type name page was confusing. Saw node reference text and thought it created text. On the Create Node form -- Menu settings has been moved up to be right below the Name, and then they got caught up in it and
He put stuff in Parent item never understood. No one understood the "teaser splitter" and the whole concept entirely. Everyone understood Input Filters, and if didn't get what they wanted to see, then they backed out. Once preview, disappeared. Went to Homepage CHANGED after posting the content -- they didn't know why it was there. They were using that initial page as navigation, and were blown away when it disappeared.
NATHANIEL CATCHPOLE: Massive contrast of going through lots of pain, and they really found the user information quickly. Admin page was fine -- adding roles was easy -- people found the permissions immediately, and they knew exactly what they were looking for. Edit own vs. Edit Any was a bit confusing and giving people more permissions They usually went to access rules before user permissions. They would look alphabetically down before clicking, and access rules comes first. [People still make that mistake]. TASK 3: Classify content with taxonomy. Taxonomy admin page, they immediately got it -- but all of the users were librarians, People would read the help text, and not see where they can do what they read about it -- but what they wanted to do was in the upper right hand of the screen after looking all done. [reading quotes] They got and understood taxonomy very quickly. Vocabulary vs. Term -- only 1 immediately made appropriate vocab, and then terms. But others were putting terms in vocab. Only 3/8 people made it to task 3.
ANGIE BYRON: TASK 4 -- No one actually made it to task 4. I'm trying to get back to the screen with the step-by-step.
The big wall of text of the intro to your site page, they read EVERY SINGLE word, and use it as navigation, and it DISAPPEARS as soon as you post any content. It says that it uses context sensitive help, help was useless. No search in the homepage. Update status, menu, node are Drupal-based -- not task-based. Missing a glossary was removed because it's unmaintained. Wanted to define module and node. What I want to see is a simple HTML form builder People couldn't figure out what a module was. If you enable content page w/o the field modules, it throws you an error to enable one. The yellow box was jarring -- Update status box. People see the help text on the Modules page was too confusing. 'I see a lot of this CCK - what is it' Clicked field set, and they saw the word "Field" and clicked on field group. Saw green "enabled" and thought that they were already enabled. They had a help desk, people will do ANYTHING to avoid calling the help desk. They'll expand every single fieldset, enable every single module. Something in PHP.ini needs fixed "Interesting..." means complicated in MN The Administration panel page was too overwhelming to read all. Need a flash tutorial that shows what Drupal can do, and so they waste a lot of time clicking on the admin page. They focus on the Upper left -- and ABOVE the fold, because that's where people are looking. The little arrows on the side of the menu - and when you click on fieldsets they expand, but on menus they don't Site Building vs. Site Configuration, and they go to one first. 'I didn't expect to feel so people. I don't like feeling tested.' People take it personally 'I need a tutorial' -- is what people say over and over again. 'I already lost the page I just created' -- don't promote to front page. Didn't create a menu page. They had to call help, and tell them admin/content and it's there We take our mastery of the "suck threshold" for granted When stumped, they use brute force. TABS are invisible -- they'd look everywhere else except No one clicked on Content Type Looking for "forms" "fields" or any thing else Lots of work to change from "Node" to "content types" -- but it dilutes it and makes it a bit worse. get away from module-based help topics -- what do users want to do, and then build help around that. B/c site building, then they'd go to the block -- and started typing HTML forms into the blocks. It was easy for them to log in. The permissions page -- they could figure it out -- but looked at access rules first. Content manage User management was easier They taxonomy was shockingly easy for them But that's it -- those are the only easier
Teaser splitter -- they thought it was "cool" -- but they had no idea what a teaser. Need to work on that feature. Title -- menu settings -- body. People thought that it was a required field, and they'd
Asking for "Help" can Kill your data. All of the people were using IE7, which after clicking help, they'd go to a new page. After that people said, I'm not going to ask for help because that'll destroy my data. Password checker -- As you type. You get an error condition as soon as you start typing -- which was frightening to them. Organization of the admin page. Collapsible fieldset, they'll expand them immediately. Usability tests -- give them something easy first to build their confidence Seeing usability testing will change your outlook on Drupal. Every single person found new ways to get stuck, and ways to get out of being stuck. We need to change a lot in Drupal 7, because Drupal 6 is in string freeze. They're talking on groups.drupal.org/usability They're making a wiki page, and lots of issues Need help with the harder issues. Also summer of code is coming up.
NEIL DRUMM: Works at Advomatic BEVAN RUDGE: Civic Actions GREG KNADDISON: Really enlightening to him.
QUESTION: How much did this cost? CODY HANSON: Usability lab is part of the Office of Information Technology and they test enterprise projects. For them it was free, but they do charge. Amazon is trying to get some free or inexpensive lab space.
ANGIE BYRON: They only got a chance to test about 5% of Drupal core. Creating content types, user and taxonomy, and that's it. And there are a lot more things that you could do in various different scenarios.
QUESTION: Is there a document or link of proposed changes.
ANGIE BYRON: On the usability group on g.d.o. there will be a total spreadsheet, and it's more hodgepodge for creating issues. What we need is a mass army to convert that spreadsheet to the issue queue. And there will be a wiki page on the usability group.
CODY HANSON: Did post tasks and ideal path on the site as well.
NEIL DRUMM: A lot of the stuff we can't put directly into issues -- Issue queue is implementing solutions, and we have to discuss the possible solution first.
NEIL DRUMM: There is a heatmaps module as well.
NATHANIEL CATCHPOLE: go to Drupal.org and search for UMN, and you will find a tiny fraction of issues there.
QUESTION: Users weren't used to finding help in a module-way. What if you have a task that has several modules involved, what's the best procedure to have a test around a task.
CODY HANSON: It would have been nice to give people success early, but wanted to give something as functional as possible in creating content types. It's tricky, because real-world tasks span modules.
ANGIE BYRON: We should start a discussion on that because that is a real problem. Simple tasks can involve 3-4 problems, and so it's tricky to have consistent help and consistent UI.
NATHANIEL CATCHPOLE: Experienced users start to type in the correct URL path, but new users have scroll up and down and all over the place. If you're in content types, then there's no cross linking to posting new content. So if it's related, then putting more contextual links could help.
NEIL DRUMM: When you enable a module, it's often tricky to figure out what it does -- and so they do need their own pages. But we need a better way to have multiple module tasks flow together better
BEVAN RUDGE: URL bar -- not one of the evaluators looked at the URL bar for an indication. So thinking that you're hook menu is clearly indicating where they should go is not a safe assumption. They were reading a lot of the help text on admin and at the top of forms -- usually not enough links. Should have more because they're usually task-based.
QUESTION: Did you get a sense of whether to change the interface to work with non-Drupal terminology -- or to create more help text to educate people on the Drupal jargon. Should we rename "content type" again, or have more help text?
CODY HANSON: Answer is probably both. It's not that the vocabulary didn't match -- it's that the meaning didn't match their own mental model. Task-based tutorials is what is really going to help. They'll eventually internalize the vocabulary, and it'll be easier.
ANGIE BYRON: If you ask a web developer on the street, then they'll have no idea about what "content type is" They do know "form" or "web page"
NATHANIEL CATCHPOLE: probably a mixture. About 3 people wanted a video to show them what to do, but it wasn't there. With terminology -- sometimes it was on the admin page, but they just weren't seeing it. Issue is the thing like "story" and "content" are everywhere -- "story" is the most jargony. People knew what they were looking for, but didn't see it.
CODY HANSON: Concerned that Garland was too similar to drupal.org -- but they were able to tell the difference.
QUESTION: Aren't you results biased due to librarians? Roles admin page looks very similar, and they're obviously very familiar with taxonomy.
CODY HANSON: Certainly some bias, but there were some staff, and not all professional, full-time librarians.
QUESTION: (merlinofchaos) Concerned that they started with a really difficult task of creating forms -- problems with all systems and forms. We gave them a really hard task. It's at it's worst. Be cautious. Overreaching to invalid data is what changed "taxonomy" to "category"
CODY HANSON: It was really difficult, and still difficult no matter how familiar you are with Drupal. The kinds of changes that you'll find are very minor. None of us are thinking that we'll have some massive wholesale changes.
NEIL DRUMM: We should always make evolutionary changes, and not revolutionary. And go back and do more testing -- hopefully BEFORE they next release.
QUESTION: Site building is where they did go first. They were looking were they were supposed to. But we don't have enough info there. Need more links.
CODY HANSON: Yes.
QUESTION: Curious to know if after the testing, is this something that they would want to use.
CODY HANSON: Debriefing question: what would you tell a colleague. They realized that they were being put into an artificial situation. They realized the power and flexibility of the system, and used to working with static pages, and so they were helpful.
KAREN STEVENSON: Some people were excited to hear that it was more difficult to use because of the UI of Drupal and not them.
CHAD FENNEL: People were excited about the power, and saw that there was going a lot underneath the hood.
ANGIE BYRON: One participant ironically said that it's more like OSX and not DOS. They wanted more power to get more access to the raw HTML.
KIERAN LAL: One person said, I'm going to work on it over the weekend. Another said I'm going to the bar. And another person said, "This will be really good after it's released." (Laughs)
QUESTION: This is an advanced task that used to require a web programmer to do.
KAREN STEVENSON: Shouldn’t have put first task first. What screwed them up was creating a content type, and figuring out how to get started -- not adding a field. They got that pretty quickly.
QUESTION: Why didn't people see tabs?
NATHANIEL CATCHPOLE: Don't really know. Possibly the colors and the position. They tend to look down and not across.
CODY HANSON: There are no tabs on the default homepage.
KIERAN LAL: Start to reading tons of terminology, and it's done and they're over. And if we're forcing people to look through fishbowl glasses, then that's not good.
CHAD FENNEL: People wanted to walk through things and get a overall context as to where they were at. Some of the complex tasks like creating content types, they want to feel secure. There was also a vertical workflow, and so would not see the tabs.
QUESTION (chx) Drupal is too flexible, and so let's NOT take away features. The methodology. Their eyes didn't read them because the Garland theme isn't tab like.
CODY HANSON: We should test that, because that is very well the case. The one was ONLY one person who didn't see the tabs at all.
ANGIE BYRON: These are our people, and if they didn't see
QUESTION: Permission page was easy to use. When people
who would think to click on blocks -- maybe blocks need to be renamed.
CODY HANSON: That's actually the first person where one person went.
ANGIE BYRON: Probably that it's that block isn't a module.
NEIL DRUMM: User permissions was easy to deal with shows that maybe we should spend more time on content admin or content config.
QUESTION: mostly didn't have problem with users and making a role.
CODY HANSON: Role ended up being an optional step. There was difficult telling the difference between anonymous, authenticated and beyond. People were making insecure permissions.
NEIL DRUMM: People did see the registration options as well.
ANGIE BYRON: They didn't all know that authenticated users were people couldn't create their own account -- they thought that only admins could make users, and did some insecure things.
QUESTION: jstools for admin? Menu block, and ajax call from clicking on the arrow.
NEIL DRUMM: Need to sit down and talk about what happens when you submit a form. That's a research project to figure out what would be best to put into core to make the menu dropdowns more.
QUESTION: This is for a new user. Collapsible fieldsets, advanced users appreciate it. So how do you balance the design decisions for a first-time user vs. advanced user.
BEVAN RUDGE: We don't know. Am working on vertical tabs. Usability finds problems, and doesn't give solutions.
NATHANIEL CATCHPOLE: A few people want wizards to make content types, but no advanced users want to always use it. As people got to the end of the hour, they found stuff a lot quicker. And so they picked it up. Fieldsets are good and they save space, but they also dump stuff in the space, and it's there, and they will still have to look at it anyway.
QUESTION: What will you take and use in other projects beyond Drupal
KIERAN LAL: Use proper terms for the audience -- like say "web pages" and not "content"
BEVAN RUDGE: People don't use the URL bar. Vertical movement of the eyes are the biggest one.
BEVAN RUDGE: User experience goals, and how to repeat this. This is a draft to open up the discussion for how to do this and what should they be. High-level things that we should be aiming to achieve and consider when building UI for Drupal, and for expectations and goals for core. We should make "Where's Waldo" into more order.
1.) Measure the user experience -- gives us data to make the changes 2.) Consistency 3.) Understandable language 4.) Not feel overwhelming 5.) Give them informative feedback to move to the next task.
on http://groups.drupal.org/node/9252
Repeating this testing. We want to see this happening more. The first goal is to measure the user experience, and not guess about it. One of the way we can measure it is with usability testing, but we need labs and people to observer, facilitators to run the lab, and then interviewers to debrief the evaluators, and then resources to get everyone to the lab. Another way to measure is with informal usability testing, which can be as valuable. But it can be guided to be a lot more helpful. Something else we'd like to see is a set of tools to make informal testing more helpful -- the click Heatmap.module. Watch the Usability group for more details!
CODY HANSON: thanks for coming, and hope that the conversation continues throughout the week. Found a lot of issues, but we should be hopeful because the mental models of the evaluators actually matched Drupal's mental model very well.