Aug 09 AT 7:34 PM Dustin Earley 39 Comments

Harvard smartphone usability showdown: Android vs iPhone vs WP7 vs BlackBerry

In a recent study done by a Harvard professor of Human Factors in Information Systems Design, Dennis Galletta, the HTC Thunderbolt, the iPhone “4G” (yes, seriously), the Samsung Focus and the BlackBerry Storm were all put to the test. What kind of test you ask? According to the good professor, a usability test. By putting a bunch of newbies in front of a smartphone and telling them to perform certain tasks, Galletta claims he is able to show the usability of a phone. The results are… embarrassing to say the least.

Since the test was put on by a Harvard professor, you’d think there’d be a little more method to his madness than there appears to be. The test itself was rather straight forward. Take a group of people who had never used the specific model of phone they were being tested on, then get them to make a call, send an SMS message and add a contact. From the looks of it though, some users appeared to be well versed in how to use mobile technology. Others, like anyone who steps up to the Storm, not so much. While the iPhone and Focus tested the best according to the study, there’s clearly not a whole lot of organization and it doesn’t look like anything was taken too seriously. If the test was based on taking the phone out of the box, putting in a SIM card and then performing the tasks, I think the results would be a different story.

If nothing else though, the video is good for a laugh. Maybe someday, we’ll see a test similar to this that we can actually rely on.

Source: PCWorld

Dustin Earley: Tech enthusiast; avid gamer; all around jolly guy.

    Most Tweeted This Week

  • BiGMERF

    I want to call the numbers they were dialing and tell them how foolish this test was and tell then how dumb they re for putting there hone numbers on the web… SMH

    • Dustin Earley

      Make sure to record it!

      • BiGMERF

        LOL………… you guys need to add an edit feature, I misspell so much… HA

        • Interpol91

          +1 for that! I would love an edit feature!!!

          • http://www.nexsoftware.net Justin Shapcott

            I have an edit button.

          • BiGMERF

            @justin.. you da man!

        • Dustin Earley

          I have one. You can hire me to edit your comments.

          • Interpol91

            lol nice. What’s your going rate per letter? haha

          • BiGMERF

            Not fair……….

          • http://www.nexsoftware.net Justin Shapcott

            Whatever Dustin charges, I’ll charge marginally less. I’m like the Walmart of hired comment editors.

  • Interpol91

    Wow. I can’t believe a Harvard professor managed to create the lamest smartphone test ever. Congrats Dennis!

  • Nathan

    I can’t believe that they think the iphone is a “4G” devices! what the Hell!?!?!

    • Interpol91

      Exactly, Just an example of one of many people who think their iPhone is a 4G device. It’s the iPhone 4! Not iPhone 4G!! Come on Dennis

    • Zak Jones!

      Something like 30% of iPhone users think they have a 4G phone, yo. It’s embarassing. Silly hipsters.

      • http://www.infotainmentempire.com pekosROB

        34%

      • http://www.infotainmentempire.com pekosROB

        And 100% of the 34% are idiots!

  • aj

    Obviously it was a woman using the iphone. MAN!!! whats a woman doing using a cellphone. She needs to get off that phone and get back to the kitchen to make all the men dinner!!!!. JK JK. I kid.
    But seriously i think that test was stupid. I could have sworn that android would have been up there next to iphone. Im pretty sure android won more.

  • NamelessTed

    There needs to be way more information available for a proper study. Are they taking a person and having them start with the WP7 and then have them use the next phone right after. By the time they get to the last phone they will have a better idea on how a smart phone works in general. You would need to do this study with hundreds of people and change the orders in which the phones are used to determine if that makes a difference.

    Also, if you are doing a study on cell phones, you should at least know what the fuck the phone is actually called. Calling it the “iPhone 4G” again and again makes me want to not listen to the rest of the data that you collected. That may seem ad hominem but if he can’t get that simple info correct then I feel a lot of other data could have easily been contaminated.

    There is also the question about whether or not this is even pertinent information. Does it REALLY matter how usable a phone is with absolutely no knowledge of the device? I dare you to sit somebody down that has never used a computer and have them go to a website, send an email, and create and save a word document. I think it would be much more important to do an in-depth study on how long it takes a person to learn to use a device until they feel comfortable with it. And also have people use different phones for extended periods of time and test how fast and easily they are able to perform a handful of different tasks.

  • Pedro Ortiz

    Shouldn’t this read “vs Sense” instead of “vs Android”?

  • Kyle

    I don’t see how this got by as an actual Harvard test… The reason for slowness was completely dependent on the person. Some knew to dial a number then try to save it, but because the person driving the android device was a moron, they gave android a 1/5…
    Also, how was the blackberry not a 5/5 on the phone call? There wasn’t one moment of hesitation, and I swear it was completed faster than the android device, which got 5/5… Again with the text messaging, the iPhone was clearly a 5/5, and for how much Android was sped-up, how the hell did it earn a 3/5 and not 1/5?

  • Mighty_O

    I think they need to read the apps better, cause from the looks of things they cant read.

    • AME

      Yeah- The Android person looked at the app tray and looked through all of them and missed “Contacts” in the section with all the apps starting with “C” near the top between the ones starting with “B”s and “D”s.

      Clearly that Harvard student has not taken the class, Advanced Alphabetical Order yet.

  • Jhass

    Ha I had him for an information systems class in grad school

    • AME

      That particular class might be something you should keep to yourself at AndroidandMe now. (Sorry! I had to!)

    • http://www.pitt.edu/~galletta Dennis Galletta

      Really? When were you in my class?

      By the way, please check my response below. I did not do this study!!!

  • kwills88

    After watching this, I am pretty convinced these people are technologically challenged…

    1. Who uses real phone numbers in a video
    2. You have to be really slow In order to take so long to add a contact on a android phone no matter th UI
    3. The iPhone 4G

  • Maxymilian Schou

    LOL! Poor Dennis Galletta did FAIL at the iPhone SMS test :D

    If you look closely @ 08.01 – he did just open up the sms app xD This so called “test” wouldn’t be approved at my University :) At least edit your videos correctly Dennis, or at least show them what to do while the camera is turned off :D …. but then again …. who would belive a test that was executed by the “Professor” who doesn’t know anything about the device he is about to test xD
    But what do we know? maybe the professor did get a iPhone 4G from Apple…. a so called bribe :D

  • Jasonhunterx

    Wow all they had to do was hold search button to do all of that

  • Glenn

    Obviously, not a very scientific study. You can’t have just four different people test four different phones and call the results conclusive.

    And even with just four people, the tasks should be much more controlled–Each person should be told to call the same (fake or public) number; add the same, non-pre-existing contact to the phone; and send the same text message to the same person, whose text message history should be non-existent on the phones.

  • The_Omega_Man

    Odd, Because I meet a lot of newbie smartphone owners (mostly younger female) and they seem to have NO problems with texting or adding my number to their phones. It helps with Android when you can arrange what you want and what you need right were you want and where you need it to be, so you can find it very quickly.

  • tobin

    umm,

    I stopped watching the video after the ridiculous scoring for the “phone” function. Did anyone else notice that the iPhone, Blackberry, and Thunderbolt all had exactly the same steps to make a call (pushing their respective ‘phone’ buttons), but for some reason the Thunderbolt received 5 stars instead of 4? I love Android, but to say that one has advantage over another when it appears that making a phone call is fairly simple on all the devices is kind of absurd. Also, shouldn’t they be testing out smartphone functionality, since the people in the market for smartphones would (in theory) know how to make calls and texts etc, and be looking at upgrading to a smartphone for the other functions it offers?

    • AME

      I noticed that too. I was cheering for Android to win (even though I was pretty sure this was designed to highlight the iPhone), but thought it was unfair that it got 5 stars while the BB and iPhone got 4 for the same performance.

      Nowhere did he explain how he was going to assign the stars which is CRUCIAL in an experiment.

  • http://jwtear.blogspot.com/ cutiyar neriman

    its bulll shiiitttt , in android when u enter the number at once u can add it to contacts if its not exist in ur contact list , he went to enter the number but after he went to all programs and some bull shits , but he comed back to first time when he entered the number to save it.

  • Noel

    Such a useless test…phones come with user manuals and u read them b4 using ur device.

  • ChaosKiller

    Poorly executed usability test. You can’t expect 4 different people to do the same with the same phone. What makes you think you can compare 4 different people with a different phone? All 4 people need to test all 4 phones to make a good comparison.

  • AME

    Once I heard “Yakety Sax” I knew we were dealing with a scientist of the highest caliber.

    I sure hope he paid for this with a grant funded by our tax dollars!

  • Mike

    Did anyone notice that it wasn’t the stock HTC Sense home screen? The message icon and the people icon looked as if they were removed. In addition no one tried to answer the phone. For the 8 hours my dad has a WM7 phone he couldn’t figure out how to answer it!

  • http://www.pitt.edu/~galletta Dennis Galletta

    I’m the professor who you’re all talking about. First and foremost, this was NOT a study! It was provided by my students as a class assignment. If you notice at the end of the video, they put the name of the course and my name on it at the end.

    So, I did NOT do this test. And as an Android fan, I would never call the iPhone 4 a 4G device.

    This was a class project in my Harvard summer course. The students did the entire test and production by themselves and all of this happened before I even knew the video had been done! I asked them to do an objective comparison of the four devices and to make it fun.

    I also wanted them to limit the video to 10 minutes. They said that because of the time limit, they shortened their list of 6 tasks into only 3 tasks. When they completed it, they posted it to YouTube and in 3 hours, there were about 3,000 views (along with plenty of well-deserved criticism)!

    The purpose of the assignment was to become introduced to usability testing. They accomplished that goal, and I provided feedback to my students:

    1. There should be more tasks
    2. There should be more people testing
    3. There should be explicit criteria for scoring
    4. There should be no personal phone numbers in the video

    Thanks for your interest in this class project.

    DG

    • anthill

      Good job DG encouraging your students to post their class projects.

      Love the haters above blaming users for usability problems. Great design insight guys, it’s competition like you that has Apple worth more than Exxon Mobil.