Attend XDA's Second Annual Developer Conference, XDA:DevCon 2014!
5,768,101 Members 44,000 Now Online
XDA Developers Android and Mobile Development Forum
View Poll Results: For better fairness, should I...
...continue to benchmark with T132 20 18.52%
...use the kernel supplied with the ROM instead 34 31.48%
...do it both ways 54 50.00%
Voters: 108. You may not vote on this poll

[REF] All ICS ROMs Benchmarked

Tip us?
bedalus Old
(Last edited by bedalus; 10th April 2012 at 06:29 PM.)
#1  
Guest
Thanks Meter
Posts: n/a
Default [REF] All ICS ROMs Benchmarked

Results in Google Docs Spreadsheet
All ICS ROMs Benchmarked

Any regular visitors to this thread can't have failed to notice the big changes in the spreadsheet. In particular I've been studying the effect of the kernel the ROM developer includes. If you look at column H, you will see whether I recommend flashing another kernel to improve performance at stock speeds.

REMEMBER: Performance is not everything! Many custom kernels provide extra features!

If you do decide to switch kernels, and only desire stock speeds and a reduced feature set, I recommend two kernels in this post.

If you are new to this stuff read this.

SUMMARY OF RESULTS
#1st place: Pixel ROM 1.72
#2nd place: CyberGR-MOD|NS.NGN ICS v10 HYBRID
#3rd place: MIUI Catalyst 2.4.6
#4th place: AOSP+ N.5
#5th place: [MIUI][2.4.6] Oodie pRiMe rOcket fUeled

Where do I find all these wonderful ROMs?

Their XDA threads are all linked in the table, so click through to discover each ROM's feature set, included apps, theme, and evidence of good karma!

Looking for a Gingerbread ROM? Try this.

Where did all your other benchmarks go?

Kernel Features & Benchmarks: this thread

Battery drain: this thread

CPU Governors and I/O Schedulers: this thread

Power Saving Governors: this thread

Thanks to all the developers.
The Following 103 Users Say Thank You to For This Useful Post: [ Click to Expand ]
bedalus Old
(Last edited by bedalus; 9th February 2012 at 03:55 PM.)
#2  
Guest
Thanks Meter
Posts: n/a
Testing Methodology

Each ROM is clean installed, allowed its own default settings, then all benchmark software is installed and run once the system is settled. I also perform one screen off screen on, just out of habbit (this used to resolve a graphics bug in gingerbread, which I'm not even sure exists in ICS). All results are recorded in the spreadsheet. Then I restart in recovery, wipe cache and dalvic (thanks morfic for dalvic wiper) and install morfic's T132-I kernel. Reboot, use nstools to select performance (default I/O scheduler for morfic's kernel is deadline). Reperform all benchmarks, recording them in the spreadsheet. The calculation for the overall CPU RAM I/O and GRAPHICS scores is identical to the one I use in my kernel benchmarking spreadsheet. Please refer to that thread for more detail. However, there is one difference. Quadrant behaves very differently in different ROMs depending on what graphics tweaks they incorporate. As such, no Quadrant scores are used in this ROM benchmarking study. For more details, see under 'Graphics' below.

The final step was to average out the results of both kernel scores, and ranking on that basis. Some would say that they are best ranked by looking at T132 scores only. I would say that is a fair point, and definitely worth consideration.

There are two reasons I incorporate the default kernel score:
1) Overall scores have a worst case scenario accuracy of within 2% of the true mean due to variability in the benchmarks (more detail on how I obtain this figure is available in my kernel benchmarking thread). For normally distributed data (which we can safely assume these benchmarks produce) averaging two sets of results will result in doubling the accuracy to within 1% of the true mean, making the ranking more reliable.
2) Combining the default kernel Scores for ROMs that used a poor performing kernel, or selected bad governor/scheduler combinations by default will suffer as a consequence. These problems can be solved if the user flashes their own preferred kernel and/or adjusts the kernel settings. However, many new users will not perform this step. For that reason, I want to give a score that at least somewhat reflects the performance as the developer intended.

-Graphics

One of the early findings is selecting force GPU rendering in the developer options improves 2D performance, in Quadrant only. Also, and only in Quadrant, 3D performance can be improved by deleting or moving /system/lib/egl/libGLES_android.so using root explorer or equivalent. These tweaks/hacks do not seem to affect other benchmarks in the slightest.

Because of the bizarre effects of these tweaks on Quadrant scores, I have removed them from the formula that calculates the overall GFX score, and this in turn impacts the overall average score. In short, it makes comparing each ROM's score fairer.

You can see evidence of how the graphics tweaks make Quadrant behave differently at the bottom of the spreadsheet, highlighted in light red. The host ROM for this particular comparison was DianXin or DX ROM for short. This is a reference to a post I make in DX ROMs thread, where I first decide that removing Quadrant is the best option: http://forum.xda-developers.com/show...&postcount=194

-Android 4.0.1

These ROMs (at the bottom of the spreadsheet, highlighted in light red) were based on early maguro ports provided by koush. I have left them in, because although they are difficult to compare to 4.0.3 builds with the correct drivers and whatnot, they do compare to each other in one important respect: the kernel. One is stock, the other is built by eugene373 (galaxy nexus/nexus s dev.) Using eugene373's kernel, CPU, I/O, and Graphics were all raised to a high standard that is on a par, or above some of the current ROMs. This goes to show that a customised kernel can offer great benefits, even on a ported ROM.


-Android 4.0.4
As of the 5th of February, my table contains all the ICS ROMs that are listed in the XDA forums. There was only one 4.0.4 version at the time, and this was the best performing in the benchmarks. When the source code is released, I'm sure many ROMs will update. At this point they will benefit from increased RAM scores and javascript performance. If all the ROMs scores improve by the same proportion, the rankings will not be affected. I will attempt to included major releases, but I will not reperform these tests every time there is a new nightly.
The Following 16 Users Say Thank You to For This Useful Post: [ Click to Expand ]
bedalus Old
(Last edited by bedalus; 14th February 2012 at 01:20 PM.)
#3  
Guest
Thanks Meter
Posts: n/a
Reserved 4
The Following 3 Users Say Thank You to For This Useful Post: [ Click to Expand ]
bedalus Old
#4  
Guest
Thanks Meter
Posts: n/a
Reserved 3
 
Ibn Saeed
Old
#5  
Senior Member
Thanks Meter 36
Posts: 396
Join Date: Nov 2011
Thanks

I was waiting for this

It might lead to troll wars though

By the way, i dont see Pete ICS rom in there, im using it.
The Following 2 Users Say Thank You to Ibn Saeed For This Useful Post: [ Click to Expand ]
bedalus Old
#6  
Guest
Thanks Meter
Posts: n/a
Quote:
Originally Posted by Ibn Saeed View Post
Thanks

I was waiting for this

It might lead to troll wars though

By the way, i dont see Pete ICS rom in there, im using it.
Thanks, and no troll wars please y'all.
 
mathkid95
Old
#7  
mathkid95's Avatar
Recognized Developer
Thanks Meter 5801
Posts: 7,539
Join Date: Jun 2010

 
DONATE TO ME
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom

But it's up to you

Sent from my Nexus S using XDA App
The Following 3 Users Say Thank You to mathkid95 For This Useful Post: [ Click to Expand ]
 
rubyknight
Old
#8  
rubyknight's Avatar
Senior Member
Thanks Meter 133
Posts: 591
Join Date: Sep 2010
Location: Sevierville TN
Personally, benchmarks are very misleading... Some of the most stable and well developed roms don't get the greatest scores because they weren't made with benchmarks in mind.

Sent from my Nexus S 4G using xda premium
Current:

HTC One on Sprint


Previous:

HTC Evo 4G LTE on Sprint


Google Nexus S 4G on Sprint


HTC Hero CDMA on WiFi(Retired/Alarm Clock)


Samsung Galaxy Tab 7.0 Plus on WiFi
 
android_nexus
Old
#9  
android_nexus's Avatar
Senior Member
Thanks Meter 42
Posts: 118
Join Date: Feb 2012
Another great work! Thanks

Sent from my Nexus S using XDA App
 
simms22
Old
#10  
simms22's Avatar
Recognized Contributor
Thanks Meter 21015
Posts: 25,311
Join Date: Jun 2009
Location: BROOKLYN!
Quote:
Originally Posted by mathkid95 View Post
Don't you think "popularity" is going a little too far? It will definitely lead to fights...all the devs are good and I don't think prominence is a good indicator of a good rom

But it's up to you

Sent from my Nexus S using XDA App
just remember, popular doesnt mean better. but to get angry at the truth(whatever it might be) is just silly. its lies that you should be upset with.
Google+ Profile
Twitter Simms22
DerKernel For Trinity kernels and more!

THREAD CLOSED
Subscribe
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes