[REF] All ICS ROMs Benchmarked

For better fairness, should I...

  • ...continue to benchmark with T132

    Votes: 20 18.9%
  • ...use the kernel supplied with the ROM instead

    Votes: 34 32.1%
  • ...do it both ways

    Votes: 52 49.1%

  • Total voters
    106
Status
Not open for further replies.
Search This thread
B

bedalus

Guest
Results in Google Docs Spreadsheet
All ICS ROMs Benchmarked

Any regular visitors to this thread can't have failed to notice the big changes in the spreadsheet. In particular I've been studying the effect of the kernel the ROM developer includes. If you look at column H, you will see whether I recommend flashing another kernel to improve performance at stock speeds.

REMEMBER: Performance is not everything! Many custom kernels provide extra features!

If you do decide to switch kernels, and only desire stock speeds and a reduced feature set, I recommend two kernels in this post.

If you are new to this stuff read this.

SUMMARY OF RESULTS
#1st place: Pixel ROM 1.72
#2nd place: CyberGR-MOD|NS.NGN ICS v10 HYBRID
#3rd place: MIUI Catalyst 2.4.6
#4th place: AOSP+ N.5
#5th place: [MIUI][2.4.6] Oodie pRiMe rOcket fUeled

Where do I find all these wonderful ROMs?

Their XDA threads are all linked in the table, so click through to discover each ROM's feature set, included apps, theme, and evidence of good karma!

Looking for a Gingerbread ROM? Try this.

Where did all your other benchmarks go?

Kernel Features & Benchmarks: this thread

Battery drain: this thread

CPU Governors and I/O Schedulers: this thread

Power Saving Governors: this thread

Thanks to all the developers.
 
Last edited:
B

bedalus

Guest
Testing Methodology

Each ROM is clean installed, allowed its own default settings, then all benchmark software is installed and run once the system is settled. I also perform one screen off screen on, just out of habbit (this used to resolve a graphics bug in gingerbread, which I'm not even sure exists in ICS). All results are recorded in the spreadsheet. Then I restart in recovery, wipe cache and dalvic (thanks morfic for dalvic wiper) and install morfic's T132-I kernel. Reboot, use nstools to select performance (default I/O scheduler for morfic's kernel is deadline). Reperform all benchmarks, recording them in the spreadsheet. The calculation for the overall CPU RAM I/O and GRAPHICS scores is identical to the one I use in my kernel benchmarking spreadsheet. Please refer to that thread for more detail. However, there is one difference. Quadrant behaves very differently in different ROMs depending on what graphics tweaks they incorporate. As such, no Quadrant scores are used in this ROM benchmarking study. For more details, see under 'Graphics' below.

The final step was to average out the results of both kernel scores, and ranking on that basis. Some would say that they are best ranked by looking at T132 scores only. I would say that is a fair point, and definitely worth consideration.

There are two reasons I incorporate the default kernel score:
1) Overall scores have a worst case scenario accuracy of within 2% of the true mean due to variability in the benchmarks (more detail on how I obtain this figure is available in my kernel benchmarking thread). For normally distributed data (which we can safely assume these benchmarks produce) averaging two sets of results will result in doubling the accuracy to within 1% of the true mean, making the ranking more reliable.
2) Combining the default kernel Scores for ROMs that used a poor performing kernel, or selected bad governor/scheduler combinations by default will suffer as a consequence. These problems can be solved if the user flashes their own preferred kernel and/or adjusts the kernel settings. However, many new users will not perform this step. For that reason, I want to give a score that at least somewhat reflects the performance as the developer intended.

-Graphics

One of the early findings is selecting force GPU rendering in the developer options improves 2D performance, in Quadrant only. Also, and only in Quadrant, 3D performance can be improved by deleting or moving /system/lib/egl/libGLES_android.so using root explorer or equivalent. These tweaks/hacks do not seem to affect other benchmarks in the slightest.

Because of the bizarre effects of these tweaks on Quadrant scores, I have removed them from the formula that calculates the overall GFX score, and this in turn impacts the overall average score. In short, it makes comparing each ROM's score fairer.

You can see evidence of how the graphics tweaks make Quadrant behave differently at the bottom of the spreadsheet, highlighted in light red. The host ROM for this particular comparison was DianXin or DX ROM for short. This is a reference to a post I make in DX ROMs thread, where I first decide that removing Quadrant is the best option: http://xdaforums.com/showpost.php?p=22323688&postcount=194

-Android 4.0.1

These ROMs (at the bottom of the spreadsheet, highlighted in light red) were based on early maguro ports provided by koush. I have left them in, because although they are difficult to compare to 4.0.3 builds with the correct drivers and whatnot, they do compare to each other in one important respect: the kernel. One is stock, the other is built by eugene373 (galaxy nexus/nexus s dev.) Using eugene373's kernel, CPU, I/O, and Graphics were all raised to a high standard that is on a par, or above some of the current ROMs. This goes to show that a customised kernel can offer great benefits, even on a ported ROM.


-Android 4.0.4
As of the 5th of February, my table contains all the ICS ROMs that are listed in the XDA forums. There was only one 4.0.4 version at the time, and this was the best performing in the benchmarks. When the source code is released, I'm sure many ROMs will update. At this point they will benefit from increased RAM scores and javascript performance. If all the ROMs scores improve by the same proportion, the rankings will not be affected. I will attempt to included major releases, but I will not reperform these tests every time there is a new nightly.
 
Last edited:

Ibn Saeed

Senior Member
Nov 3, 2011
397
28
Thanks

I was waiting for this

It might lead to troll wars though

By the way, i dont see Pete ICS rom in there, im using it.
 
  • Like
Reactions: Sigurgrimur

SA-07

Inactive Recognized Developer
Jun 21, 2010
7,566
5,942
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom

But it's up to you ;)

Sent from my Nexus S using XDA App
 

rubyknight

Senior Member
Sep 19, 2010
594
132
Sevierville TN
Personally, benchmarks are very misleading... Some of the most stable and well developed roms don't get the greatest scores because they weren't made with benchmarks in mind.

Sent from my Nexus S 4G using xda premium
 

simms22

Recognized Contributor - R.I.P
Jun 4, 2009
34,053
25,934
BROOKLYN!
www.androidcommunity.com
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom

But it's up to you ;)

Sent from my Nexus S using XDA App

just remember, popular doesnt mean better. but to get angry at the truth(whatever it might be) is just silly. its lies that you should be upset with.
 

uansari1

Senior Member
Jul 27, 2008
3,545
92
Karachi
Where's Peter Alfonso's ROM? Bugless Beast is a mainstream rom, second only to Cyanogenmod in scope and breadth.. The ROM thread is in the General forum, because he links to his page at Rootzwiki, but it's a more prominent ROM than many you've listed.

Please include it in the benchmarking as well. :)

Sent from my Nexus S using Tapatalk
 
  • Like
Reactions: moneybag
B

bedalus

Guest
Omissions fixed!

UPDATE: kwiboo's project removed, was an SDK port. So so tired now, got to sleep. Tomorrow begin again with Brainmaster's tweaked ICS
 
Last edited:

biotecsoul

Member
Nov 18, 2010
47
10
RM
Which kernel is used to make the tests?


-----

Forget It...

"I'll be benchmarking them twice with whatever kernel they come with, and then with morfic's bare boned T132"
 
Last edited:

rrohanjs

Senior Member
Jan 22, 2011
4,262
1,439
37
mumbai
u could also add overnite battery drainage column with sync on or somethin like that, if not overnite then a period of 2 hrs idle .. how much does it drain ..
 

thegtfusion

Senior Member
Jan 24, 2011
4,235
2,219
Utrecht
Holy **** man, what would it be if the NS comunity didnt have you?? :D

Thumbs up!!!! :D

Sent from my Nexus S using xda premium
 
B

bedalus

Guest
Holy **** man, what would it be if the NS comunity didnt have you?? :D

Thumbs up!!!! :D

Sent from my Nexus S using xda premium

Haha, that makes me feel good. Wait, Benchmarks > XDA thread > Positive XDA user feedback > Dopamine release = Addiction to benchmarking? :eek: Time to join crackflashers anonymous...
 

AndroidUser00110001

Senior Member
Aug 16, 2011
3,432
5,739
NYC
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom

But it's up to you ;)

Sent from my Nexus S using XDA App

I doubt it would lead to fights between devs, well hopefully not. I can see users arguing stuff more than devs.

Benchmarks aren't that good anways.

BTW, your kernel sucks!!!!

Just kidding been running smoothly since I started using it the other day :D

-----------

Thanks for the ratings system. Looks nice!
 
Last edited:
Status
Not open for further replies.

Top Liked Posts

  • There are no posts matching your filters.
  • 8
    Thanks, but I've tried these

    #1st place: Pixel Rom 1.6 ICS (Android 4.0.4) moves up to number #1!
    #4th place: CyberGR-MOD|NS.NGN ICS v.9 HYBRID
    #5th place: ICS 4.0.4 IMM26 v1.1 (android 4.0.4) thanks evilisto

    And they are all crap about stability,

    Peter Alfonso Nightlies are the best rom. Stable and smooth with no equals, you can use it everyday

    This is really not very kind of you and i found it a little bit "DISRESPECTFUL"..!!
    Ok it`s your opinion and you can always share it with other users in here but please don`t express them in that way..!!
    Please "RESPECT" every Dev`s work and if you have a problem don`t hesitate to ask them or take a look into every ROM`s thread (Look for a F.A.Q) or anything that may be helpful for you but please don`t say "CRAP" specially if you didn`t even tried to do something of the above!!
    5
    Thanks, but I've tried these

    #1st place: Pixel Rom 1.6 ICS (Android 4.0.4) moves up to number #1!
    #4th place: CyberGR-MOD|NS.NGN ICS v.9 HYBRID
    #5th place: ICS 4.0.4 IMM26 v1.1 (android 4.0.4) thanks evilisto

    And they are all crap about stability,

    Peter Alfonso Nightlies are the best rom. Stable and smooth with no equals, you can use it everyday

    Stability :eek: lol .. all the roms you mentioned are rock solid to me .
    ( if you know how to use a phone by the way )
    Tried all 3 and no stability issues
    4
    Im fully with you. come on guys. the man does it for free!! He works his ass off for us. He could also make the rom and not release it, but make it only for his own use.
    be grateful for what we are getting!!

    Amen to the AWESOME DEVS :D :D
    they make our phone 5 times worth it!!

    My work is very humble (no source) by comparison to the stuff that can be gleaned from AOSP trees - only filling users needs where I can.

    However - whilst turds will throw **** like this at what we've shared with the Nexus S community - I just ask myself -

    "What does ugo999 bring to the Nexus S community - what is he sharing that benefits others in here????"

    I'm trying to spread a little love to the 4G community at the moment. That IS what this is about for sure. Too many jumped up ****s who contribute nothing and profess to know everything.

    As I say - my work is humble; just about delivering the right recipe for the job. That is what I try to contribute.

    If my ROM isn't stable - as Oodie says - this ugo999 possibly doesn't know how to use the phone.
    3
    Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom

    But it's up to you ;)

    Sent from my Nexus S using XDA App
    3
    This is really not very kind of you and i found it a little bit "DISRESPECTFUL"..!!
    Ok it`s your opinion and you can always share it with other users in here but please don`t express them in that way..!!
    Please "RESPECT" every Dev`s work and if you have a problem don`t hesitate to ask them or take a look into every ROM`s thread (Look for a F.A.Q) or anything that may be helpful for you but please don`t say "CRAP" specially if you didn`t even tried to do something of the above!!

    I completely understand this from Cybergr. When someone tries hard to make you a special gift and you say to him "this is crap", Just try ro imagine what he can feel.
    Respect to the developers please and don't forget that all what you find in this forum are kind gifts from them ;)