Oct 02 AT 1:43 PM Dima Aryeh 27 Comments

Samsung isn’t the only one boosting benchmark scores

Samsung Galaxy Note 3 2

The most recent controversy in the Android world is Samsung’s cheating in benchmarks. This behavior was reported on the Galaxy S4, and reports are now showing that the Note 3 has the exact same behavior. Depending on the model, the device will ramp up CPU and even the GPU to get better scores in benchmarks, just to appear better than the competition.

The Snapdragon 800 variant of the Note 3 will ramp the CPU up to 100% load to gain a higher score than it would get if it gradually ramped up the CPU. The Exynos powered model will also boost the GPU frequency (by removing thermal throttling) to get better scores in many benchmarks. These apps are detected by name and the behavior is enabled, but not for all benchmarks.

People are raising a stink about Samsung doing all this, but many OEMs actually do this. Samsung is definitely not alone in this behavior; only current Motorola and Nexus devices are immune to it. So if you’re going to raise a stink, do it about everyone!

Benchmark Cheaters

As you can see, many devices have some form of benchmark detection and cheating. Whether it be Snapdragon, Exynos or even Intel chips inside, it’s the manufacturer that implements it into software. Simply renaming the benchmark will actually stop it from optimizing, and you’ll see anywhere from a 3% to a 10% drop in performance for many of these devices.

Obviously, this is a bit of a dirty practice. Inflating benchmark scores provides unrealistic measurements of devices and breaks proper comparisons. All of these manufacturers should really remove any boosting software implemented in their devices, but we can’t see that happening anytime soon. Hopefully it doesn’t get more complicated, because the harder it is to get around, the more resources the company put into that software. And the less work it put into the actual device.

However, as customers, we can really make a difference. Write to these companies and demand that they stop this practice. Write to Google and ask them to add a requirement to their Play services certification. Write to your local government offi….. never mind. Hopefully we can get this practice to end soon enough.

Source: Anandtech

Dima Aryeh is a Russian obsessed with all things tech. He does photography, is an avid phone modder (who uses an AT&T Galaxy Note II), a heavy gamer (both PC and 360), and an aspiring home mechanic. He is also an avid fan of music, especially power metal.

    Most Tweeted This Week

  • ibap

    If they are detected by name, then the benchmark designers and the folks running them for publication need to find a way to rename them to whatever random name they care to use, to thwart this behavior.

    • http://www.androidandme.com Dima Aryeh

      Anandtech is actively changing the names of all of their benchmarks to get accurate results now. That’s a temporary fix.

    • shelie1

      up to I looked at the draft which was of $8326, I didn’t believe that my friends brother had been actualy bringing in money in there spare time on-line.. there moms best frend had bean doing this 4 less than 8 months and at present cleard the mortgage on there villa and bourt a top of the range BMW M3. important link……. http://gg.gg/i7ir

      • TAN5150

        Shelie1, learn to spell if you’re going to spam us! Have some freaking respect…

    • Odie

      Anandtech said that this would simply be a temp fix as they predict that OEM’s will begin coding their software to optimize for certain behaviors; for instance, when a benchmark runs, it’ll behave a certain way that programmers can use to then trigger the CPU and GPU optimization.

      The same thing happened with PC’s, I imagine this will go on until OEM’s stop this MHz race they’re currently running. I really like Apple’s and the Moto X’s approach of making the most efficient use of the hardware inside the device vs. throwing everything at it and brute forcing good results.

  • Adonis K.

    Never bothered with them, never will

  • www.phonewbie.com

    Wassup with Sony?

    • Vlad S

      Sony goes bio-organic and honest. You know their phones (Xperia P and Xperia Mini) that won Green Awards. So is the company. Japanese standards are the ones to follow :).That is why they are 50 years ahead in terms of life (or even more in some situations).

      P.S.
      Maybe they have awesome developers that know what and how to hide it, haha, it doesn’t matter. Either way, I’m going Sony.

      • Alexander drzfr3shboialex

        I’m going Sony is they make next year’s Nexus. Sony’s android UI is horribly hideous.

      • infex.tv

        Glad i bought my Z Ultra

  • Nate B.

    This doesn’t really mean much if they were or not the only ones. It’s still a beast if a device at the end of the day like a few other devices. It’s the same when companies talk about how good their camera is on their phones when announcing it but we all know the pictures shown are in artificial lighting conditions.

  • mike

    Where is thebproof that others are doing this, they have posted code for the Samsung doing this haven’t seen any other codes for other manufacturers yet.

  • Alex

    I don’t think it’s so obvious that this is a bad thing to do. The thing is, the “inflated” benchmark scores are actually more indicative of normal performance for sustained high CPU/GPU demand applications, like games.

    The problem with these benchmarks is that they run for a short time – much shorter than a game. If CPU speed is ramped up only after some period of time, then the scores in a benchmark will be skewed down. However, during actual gameplay, you’d only experience the slower speeds at the beginning, before the boost kicks in. In a level that lasts 10 minutes, the player cares much more about the frame rate that’s achieved for the last 9:55 than they do for the first 5 seconds.

    The worst thing is, from a benchmark results perspective, devices which are tuned to conserve battery power a little longer before switching into high performance mode will be much worse off in the benchmark, even if their full performance matches competitor phones. An extra second delay in the boost could make a large difference in the benchmark score, but be essentially unnoticeable in gameplay where it only happens at the beginning of a level.

    In this sense, it’s really better if ALL phone manufacturers boost their benchmarks so that the shortness of benchmark tests doesn’t give misleading feedback.

    Ideally though, benchmarks would be designed to test these scenarios separately. For sustained use, they should afford some stated period of warm-up time before measuring performance. For intermittent CPU use, we’d want just the opposite. These need to be separate tests.

  • http://droidsamurai.blogspot.com DroidSamurai

    The Moto X didn’t cheat and still it managed to keep up with other flagships like the S4 and HTC One in benchmarks. Now that we know other OEMs has cheated, does that mean the Moto X is even faster than the benchmarks portrayed, when comparing to those phones?

    • clocinnorcal

      That is the power of software optimization. Imagine if all the spec-whore flagship devices put in a little more effort on the software side of things.

  • cj100570

    Or better yet, instead of the manufacturers ceasing this practice, why don’t review cites stop receiving playing into the benchmark BS game as if they are a real indication as to how a given device actually performs in the real world?

  • cj100570

    Or better yet, instead of the manufacturers ceasing this practice, why don’t review cites stop playing the BS benchmark game as if they are a real indication as to how a given device actually performs in the real world?

  • Fulaman

    Where is the Sony Z1 on all this?

  • Mike

    benchmarking app for phones are crap

  • donger

    Still everyone is going to look at Samsung first.

  • Max.Steel

    This is why benchmarks for the most part are useless. They don’t exhibit how the device would perform in the real world especially after months of usage and app installs and uninstalls.

  • Sara Fleischer

    my neighbor’s half-sister makes $88/hr on the laptop. She has been out of work for 8 months but last month her payment was $12592 just working on the laptop for a few hours. More Bonuses http://www.star48.com

  • khaleader

    hi Guys follow us in this blog we have great Pro Android Games and Apps for Free
    you are welcome in http://android-proo.blogspot.com/
    and we also put the feed of this website thanks

  • energizer1389

    So allowing the processors to run a 100% is considered cheating? I think that’s BS. Carmakers alter their cars for competition and that’s the standard, why is it cheating in cellphones? If everyone doesthe ssame thing then this is the std and your still comparing apples to apples. So all of this buzz is a bunch of BS.

    • http://www.androidandme.com Dima Aryeh

      Everyone uses different methods and detects different apps, and the list changes for every manufacturer, so it’s inconsistent at best. That makes it no longer comparing “apples to apples.” And yes, it’s cheating to remove thermal throttling temporarily to get as high a score as possible, when in general use you’ll never be able to get it.

  • da9el

    antutu x can be used to get real results. it’s not recognized as a benchmark app like antutu when launching.

  • kuaisu

    wow, guys, tell you a good news: the most popular 5.0 inch Android 4.2 S4 N9500 Smartphone is big sale now. up to 90% off. now it is only € 99.99. we declare we sell the lowest price and best quality, if someone sell lower than us, we will refund you. so don’t be hesitation!! many good praise. it is the Best-selling products in our shop. now buy it, except 90% off, you will get a free spree. so welcome purchase: http://xrl.us/bpxt3t

  1. If they are detected by name, then the benchmark designers and the folks running them for publication need to find a way to rename them to whatever random name they care to use, to thwart this behavior.

    • Anandtech is actively changing the names of all of their benchmarks to get accurate results now. That’s a temporary fix.

    • up to I looked at the draft which was of $8326, I didn’t believe that my friends brother had been actualy bringing in money in there spare time on-line.. there moms best frend had bean doing this 4 less than 8 months and at present cleard the mortgage on there villa and bourt a top of the range BMW M3. important link……. http://gg.gg/i7ir

      • Shelie1, learn to spell if you’re going to spam us! Have some freaking respect…

    • OdieGuest 2 years ago

      Anandtech said that this would simply be a temp fix as they predict that OEM’s will begin coding their software to optimize for certain behaviors; for instance, when a benchmark runs, it’ll behave a certain way that programmers can use to then trigger the CPU and GPU optimization.

      The same thing happened with PC’s, I imagine this will go on until OEM’s stop this MHz race they’re currently running. I really like Apple’s and the Moto X’s approach of making the most efficient use of the hardware inside the device vs. throwing everything at it and brute forcing good results.

  2. Adonis K.Guest 2 years ago

    Never bothered with them, never will

  3. Wassup with Sony?

    • Vlad SGuest 2 years ago

      Sony goes bio-organic and honest. You know their phones (Xperia P and Xperia Mini) that won Green Awards. So is the company. Japanese standards are the ones to follow :).That is why they are 50 years ahead in terms of life (or even more in some situations).

      P.S.
      Maybe they have awesome developers that know what and how to hide it, haha, it doesn’t matter. Either way, I’m going Sony.

  4. This doesn’t really mean much if they were or not the only ones. It’s still a beast if a device at the end of the day like a few other devices. It’s the same when companies talk about how good their camera is on their phones when announcing it but we all know the pictures shown are in artificial lighting conditions.

  5. mikeGuest 2 years ago

    Where is thebproof that others are doing this, they have posted code for the Samsung doing this haven’t seen any other codes for other manufacturers yet.

  6. AlexGuest 2 years ago

    I don’t think it’s so obvious that this is a bad thing to do. The thing is, the “inflated” benchmark scores are actually more indicative of normal performance for sustained high CPU/GPU demand applications, like games.

    The problem with these benchmarks is that they run for a short time – much shorter than a game. If CPU speed is ramped up only after some period of time, then the scores in a benchmark will be skewed down. However, during actual gameplay, you’d only experience the slower speeds at the beginning, before the boost kicks in. In a level that lasts 10 minutes, the player cares much more about the frame rate that’s achieved for the last 9:55 than they do for the first 5 seconds.

    The worst thing is, from a benchmark results perspective, devices which are tuned to conserve battery power a little longer before switching into high performance mode will be much worse off in the benchmark, even if their full performance matches competitor phones. An extra second delay in the boost could make a large difference in the benchmark score, but be essentially unnoticeable in gameplay where it only happens at the beginning of a level.

    In this sense, it’s really better if ALL phone manufacturers boost their benchmarks so that the shortness of benchmark tests doesn’t give misleading feedback.

    Ideally though, benchmarks would be designed to test these scenarios separately. For sustained use, they should afford some stated period of warm-up time before measuring performance. For intermittent CPU use, we’d want just the opposite. These need to be separate tests.

  7. The Moto X didn’t cheat and still it managed to keep up with other flagships like the S4 and HTC One in benchmarks. Now that we know other OEMs has cheated, does that mean the Moto X is even faster than the benchmarks portrayed, when comparing to those phones?

    • That is the power of software optimization. Imagine if all the spec-whore flagship devices put in a little more effort on the software side of things.

  8. Or better yet, instead of the manufacturers ceasing this practice, why don’t review cites stop receiving playing into the benchmark BS game as if they are a real indication as to how a given device actually performs in the real world?

  9. Or better yet, instead of the manufacturers ceasing this practice, why don’t review cites stop playing the BS benchmark game as if they are a real indication as to how a given device actually performs in the real world?

  10. FulamanGuest 2 years ago

    Where is the Sony Z1 on all this?

  11. MikeGuest 2 years ago

    benchmarking app for phones are crap

  12. Still everyone is going to look at Samsung first.

  13. This is why benchmarks for the most part are useless. They don’t exhibit how the device would perform in the real world especially after months of usage and app installs and uninstalls.

  14. Sara FleischerGuest 2 years ago

    my neighbor’s half-sister makes $88/hr on the laptop. She has been out of work for 8 months but last month her payment was $12592 just working on the laptop for a few hours. More Bonuses http://www.star48.com

  15. khaleaderGuest 2 years ago

    hi Guys follow us in this blog we have great Pro Android Games and Apps for Free
    you are welcome in http://android-proo.blogspot.com/
    and we also put the feed of this website thanks

  16. energizer1389Guest 2 years ago

    So allowing the processors to run a 100% is considered cheating? I think that’s BS. Carmakers alter their cars for competition and that’s the standard, why is it cheating in cellphones? If everyone doesthe ssame thing then this is the std and your still comparing apples to apples. So all of this buzz is a bunch of BS.

    • Everyone uses different methods and detects different apps, and the list changes for every manufacturer, so it’s inconsistent at best. That makes it no longer comparing “apples to apples.” And yes, it’s cheating to remove thermal throttling temporarily to get as high a score as possible, when in general use you’ll never be able to get it.

  17. antutu x can be used to get real results. it’s not recognized as a benchmark app like antutu when launching.

  18. wow, guys, tell you a good news: the most popular 5.0 inch Android 4.2 S4 N9500 Smartphone is big sale now. up to 90% off. now it is only € 99.99. we declare we sell the lowest price and best quality, if someone sell lower than us, we will refund you. so don’t be hesitation!! many good praise. it is the Best-selling products in our shop. now buy it, except 90% off, you will get a free spree. so welcome purchase: http://xrl.us/bpxt3t