[KERNELS][ICS][I9000] The ICS Kernel Benchmarking Project -Update: Devil

Search This thread

Pipperox

Senior Member
Nov 11, 2010
134
84
Goal of this little project is to dispel myths and hearsay and trying to assess the elusive performance of custom kernels for our beloved SGS I9000.

So far this has proven quite challenging as there is no single good benchmark on Android (yet):
  • a lot of people have been misled by ridiculous Quadrant scores: ridiculous because, with some small tweaks which do not affect real performance in any way, shape or form, it is possible to boost the Quadrant score by factor 3x.
    You're free to believe that your SGS I9000 which scores 3000+ on Quadrant is faster than a SGS II, but then please leave this thread and move on.
  • some kernels may seem smooth with some games, and get high scores on some synthetic benchmark, yet the UI appears "laggy" and stutters a lot in comparison to other kernels which score lower on the same benchmark
  • some popular benchmarks give results with unacceptably low reproducibility, i.e. if you run them multiple times without changing a thing on your system, you get scores varying by 50% of more, in a completely random fashion
  • most popular benchmarks do not measure or take into account multitasking and CPU contention with other applications, yet on a typical usage one has background tasks such as the media scanner or synchronization which kick in often and unpredictably

So this will be mostly a work in progress, i'm testing several benchmarks and several kernels in multiple combinations, trying to analyze which benchmarks offer certain criteria which make them useful, namely:
  • Reproducibility of results: running the same tests multiple times, should result in a very small variance of the final score
  • Performance separation: benchmarks which are too "synthetic" and show only a dependency on clock speed are not useful to discriminate "fast" kernels from "slow" kernels
  • Performance representation: we all know when a kernel "looks" or "feels" fast or smooth. If a benchmarks shows you that a "laggy" kernel scores higher than a fast and responsive one, it's likely that the benchmark is not well designed

I'll work more on this thread explaining my (current) choice of tests and what they're good for.
But for now i'll just post a link to the summary table, and give a brief recommendation concerning popular ICS kernels; recommendation which i'll explain in the coming days.

Base ROM:
Slim ICS 2.8
(because is fast, smooth and has the least background stuff of all ICS ROMs which i tested)

Test Conditions:
Whenever possible, i tried to overclock the kernels to 1.2GHz which most / all phones should have no trouble achieving.
In case of Semaphore i had to use the bus / live overclock but it wasn't fully stable at 1.2GHz on my phone so i ran most of the tests at 1.14GHz.

Tested Kernels:
Stock Teamhacksung V17
Devil 1.1.6b BFS
Devil 1.1.6b CFS
Icy Glitch V14 b
Semaphore ICS 0.9.5b

Recommendation:
Devil 1.1.6b CFS, Icy Glitch V14b (with SmartassV2 and FIOPS), and Midnight ICS (with a tweaked Conservative) are trading blows for the fastest kernel.
At the time of testing, Midnight is slightly worse in terms of overclocking though, apparently due to different voltages, also it doesn't allow overclocking beyond 1.2GHz.
But what's interesting is that it achieves great performance while using a tweaked conservative governor.

Devil 1.1.6b BFS is good but obviously inferior to its CFS brother.

Semaphore has the lowest cache and memory latency in the multithreaded test, it also has impressive sd card read speed and in general appears super responsive, but it's a bit worse in 3D gaming and especially it lacks "true" overclocking, "live overclocking" changes the bus clock and is way more unstable, in fact on my phone i couldn't run it stable at 1.2GHz.

All kernels are significantly faster than the stock teamhacksung's kernel, so you have no excuses not to upgrade to one of the popular custom kernels!

ICS 4.0.4
Started testing Android ICS 4.0.4 kernels on Slim ICS 3.2.
All tested kernels are "huge mem" versions with 380+MB of available RAM, without breaking video playback or 720p recording.
Summary:
  • the stock kernel from Teamhacksung is now a very respectable performer, unless you plan to overclock probably you don't need to install one of the other kernels
  • Semaphore, Midnight and Devil are all very fast and smooth

Results table:
https://docs.google.com/spreadsheet/ccc?key=0AuBUEB4dGFSSdHIyN2VIeWU4QnhLOFpJejFPWDh5S1E
 
Last edited:

Pipperox

Senior Member
Nov 11, 2010
134
84
Res 1

One request for the kernel developers:
could you please post me what are your preferred / recommended settings in terms of Governor and I/O Scheduler?

Only one configuration per kernel please, as running these tests is rather time consuming.
 
Last edited:

Pipperox

Senior Member
Nov 11, 2010
134
84
Test Settings

So for anybody who wants to follow the same methodology as I used to test kernels, please pay attention that in some tests i didn't use stock settings, to try to improve the reproducibility of the results.

Before all test, i put the phone in flight mode, and disable all synch services.

Antutu: DB I/O and SD Write and Read have poor reproducibility. So i run these tests separately 5 times, and take the best scores.

RealPi: the number of iterations is increased by factor 10x i.e.: 100000 digits

MPAC: lots of customization here. Also be careful as it's not very stable and some settings will make it crash.
All tests: 8 threads (or 8 producer / consumer pairs)
CPU: 10000000 iterations, use case: integer (i'm considering to add logical too)
Memory: stock apart for nr of threads. Repeat the test 5 times and get best numbers
Cache: 40 iterations
 
Last edited:

cba1986

Senior Member
Aug 8, 2010
2,409
621
Somewhere in Buenos Aires
A quick question: did you lock the max freq to eliminate the "governor" variable?
Because each kernel could have governor's tweaks that the other don't.

Based on what you posted here, the differences between Glitch and Devil is practically none.

I tested both and didn't feel any tangible difference, in the end, it comes down to the unique features of each kernel.
 

phzi

Senior Member
Jul 23, 2009
214
37
Overclocking bus vs adding an extra step aren't even slightly comparable. Maybe do tests not overclocked?

Also there is a new glitch build with 100% working bln.

Sent from my GT-I9000 using xda premium
 

belmethos

Senior Member
Jul 30, 2008
576
156
Ankara
+1 for tests without overclock. Majority of us, users do not overclock. Maybe a seperate test for overclocking could be nice , but comparisons should be done with stock speeds imho.

Thanks for the time and effort. We needed this.
 

Pipperox

Senior Member
Nov 11, 2010
134
84
Overclocking bus Vs adding an extra step isn't an apple to apple comparison, I agree.
However my goal was to use each kernel in the best possible way, and if some kernels have the possibility to use higher multipliers / extra frequency steps, that is an advantage for the user, compared to the kernels who only offer live overclock.

Don't get me wrong, i love Semaphore and i've been using it for a long time.
And i have no doubt that some users can get it stable with live overclock to 1.2GHz.
But that is the ceiling, while with other kernels even my phone can reach stable overclocks of 1.5GHz, and that is something to consider.

I chose as the basis for my tests an overclock of 1.2GHz because it's something which practically everybody can use, without massive battery drain, overheat or shortening the life of the device.

I'll try to add measurements at stock speeds for those who don't like to overclock.
 

Pipperox

Senior Member
Nov 11, 2010
134
84
A quick question: did you lock the max freq to eliminate the "governor" variable?
Because each kernel could have governor's tweaks that the other don't.
I didn't want to take the governor variable out.
Because, as you said, each kernel could use (and often does) governor tweaks which make the kernel "special" or different from the others, and that has to be taken into account in evaluating them.

Because nobody will use the phone locked at the maximum frequency.

So for me the governor and its tweaks is part of the user experience of a certain kernel, and a distinctive factor.

At the end, all kernels are coming from almost the same sources, so it's the little things which make the difference.
 

HiKsFiles

Senior Member
Aug 7, 2010
537
158
Québec city
Overclocking bus Vs adding an extra step isn't an apple to apple comparison, I agree.
However my goal was to use each kernel in the best possible way, and if some kernels have the possibility to use higher multipliers / extra frequency steps, that is an advantage for the user, compared to the kernels who only offer live overclock.

Agreed but, then again, benchmarks should be done at original CPU clock IMHO.

Otherwise, results are distorted.
 

zorxd

Senior Member
Jun 30, 2010
546
41
Agreed but, then again, benchmarks should be done at original CPU clock IMHO.

Otherwise, results are distorted.


agreed. Especially since stock team hacksung seems to be clocked at 1GHz
what's the point of the comparison? Really?

As expected, there is no noticeable difference between all 1.2 GHz kernels.
It's not as if there was a real difference between them anyway.
 

Pipperox

Senior Member
Nov 11, 2010
134
84
agreed. Especially since stock team hacksung seems to be clocked at 1GHz
what's the point of the comparison? Really?

As expected, there is no noticeable difference between all 1.2 GHz kernels.
It's not as if there was a real difference between them anyway.

That's not quite true.
If you look closer, you'll see that Devil CFS has quite a distinct advantage over all others in 3D tests.

The point of the comparison between stock hacksung @1.0GHz and the others, who can overclock, is to show what kind of benefit you get from switching to kernels which are overclock friendly.
Especially considering that you can't assume that a 20% clock speed increase will bring a 20% performance speedup across the board.

At last, i'd say that you may have "expected" that the kernels tested at 1.2 GHz don't have such a difference in performance.
But expectations have to be verified.

I tried to answer the questions:
On Devil's kernel, is BFS really better than CFS?
The "popular belief" is that BFS is faster than CFS.
According to my tests, CFS results faster instead.
Another question may be, what kernel gives you the best gaming performance.
If you pay attention to the An3D Bench XL, you'll see that Semaphore 0.9.5b, even overclocked a 1.2GHz, is significantly slower than Devil.
 
Last edited:

belmethos

Senior Member
Jul 30, 2008
576
156
Ankara
If i recall correctly Semaphore Author claimed that some kernel developers overclock GPU, and he didn't. Idk anything about it, but i recall something about it.

Is it possible to overclock only GPU, without overclocking CPU??
 
  • Like
Reactions: stratosk

mialwe

Senior Member
Apr 28, 2011
757
1,565
so you should benchmark them without setting anything. Average users don't overclock and don't change governor or scheduler: they flash the new kernel and stop.
I completely agree on benchmarking every kernel at the same frequency (stock 1Ghz max) but I think there are two different options for further benchmarking:

1) Benchmark kernels configured as similar as possible regarding CPU governor, IO scheduler, readahead -> comparable results for all kernels.

2) Benchmark kernels with default settings (only makes sense if all compared kernels are optimized for similar purpose like performance, does not make sense if a kernel does *not* focus on max. performance and uses e.g. Conservative CPU governor as default setting.

@Pipperox: Would it be possible to check my Mindnight-ICS dev version with your benchmark suite? I'd be really interested in the results as you use the same setup for all kernels (1.2Ghz would not be a problem, does not use LiveOC but old school 1/1.128/1.2Ghz OC).
 

dark_knight35

Senior Member
Jan 15, 2012
1,936
989
Interesting thread... I never used devil's CFS version, always BFS. Will try CFS out now.

@Mialwe Where can we get your ics kernel?
 

Pipperox

Senior Member
Nov 11, 2010
134
84
I completely agree on benchmarking every kernel at the same frequency (stock 1Ghz max) but I think there are two different options for further benchmarking:

1) Benchmark kernels configured as similar as possible regarding CPU governor, IO scheduler, readahead -> comparable results for all kernels.

2) Benchmark kernels with default settings (only makes sense if all compared kernels are optimized for similar purpose like performance, does not make sense if a kernel does *not* focus on max. performance and uses e.g. Conservative CPU governor as default setting.

@Pipperox: Would it be possible to check my Mindnight-ICS dev version with your benchmark suite? I'd be really interested in the results as you use the same setup for all kernels (1.2Ghz would not be a problem, does not use LiveOC but old school 1/1.128/1.2Ghz OC).

Sorry guys, i understand your logic but i do not fully agree with it.
I'm not comparing overclocked kernels with heavy tweaking of voltages and special settings with which they only work.

I did the "poor man"'s overclock, setting to 1.2GHz using NSTools, a setting where 95% of phones should have no problem working.

I think that if some kernels offer you this possibility while others do not, it is fair to use this "advantage" that they have over the other kernels.

Because a lot of users will have the possibility to do the same as i do, without esoteric knowledge and with just a couple of clicks in the menus.

That being said, "due to popular demand" :) i will also try to retest those kernels at 1.0GHz as soon as i get a bit of time.

BUT in my recommendations, i will also consider the overclocking capabilities.

@mialwe: sure, i'll give a run to your kernel as well!
 

Pipperox

Senior Member
Nov 11, 2010
134
84
@Pipperox: Would it be possible to check my Mindnight-ICS dev version with your benchmark suite? I'd be really interested in the results as you use the same setup for all kernels (1.2Ghz would not be a problem, does not use LiveOC but old school 1/1.128/1.2Ghz OC).

Dude, sorry but i don't seem to find your ICS kernel anywhere.. can you provide a link?
 

Top Liked Posts

  • There are no posts matching your filters.
  • 36
    Goal of this little project is to dispel myths and hearsay and trying to assess the elusive performance of custom kernels for our beloved SGS I9000.

    So far this has proven quite challenging as there is no single good benchmark on Android (yet):
    • a lot of people have been misled by ridiculous Quadrant scores: ridiculous because, with some small tweaks which do not affect real performance in any way, shape or form, it is possible to boost the Quadrant score by factor 3x.
      You're free to believe that your SGS I9000 which scores 3000+ on Quadrant is faster than a SGS II, but then please leave this thread and move on.
    • some kernels may seem smooth with some games, and get high scores on some synthetic benchmark, yet the UI appears "laggy" and stutters a lot in comparison to other kernels which score lower on the same benchmark
    • some popular benchmarks give results with unacceptably low reproducibility, i.e. if you run them multiple times without changing a thing on your system, you get scores varying by 50% of more, in a completely random fashion
    • most popular benchmarks do not measure or take into account multitasking and CPU contention with other applications, yet on a typical usage one has background tasks such as the media scanner or synchronization which kick in often and unpredictably

    So this will be mostly a work in progress, i'm testing several benchmarks and several kernels in multiple combinations, trying to analyze which benchmarks offer certain criteria which make them useful, namely:
    • Reproducibility of results: running the same tests multiple times, should result in a very small variance of the final score
    • Performance separation: benchmarks which are too "synthetic" and show only a dependency on clock speed are not useful to discriminate "fast" kernels from "slow" kernels
    • Performance representation: we all know when a kernel "looks" or "feels" fast or smooth. If a benchmarks shows you that a "laggy" kernel scores higher than a fast and responsive one, it's likely that the benchmark is not well designed

    I'll work more on this thread explaining my (current) choice of tests and what they're good for.
    But for now i'll just post a link to the summary table, and give a brief recommendation concerning popular ICS kernels; recommendation which i'll explain in the coming days.

    Base ROM:
    Slim ICS 2.8
    (because is fast, smooth and has the least background stuff of all ICS ROMs which i tested)

    Test Conditions:
    Whenever possible, i tried to overclock the kernels to 1.2GHz which most / all phones should have no trouble achieving.
    In case of Semaphore i had to use the bus / live overclock but it wasn't fully stable at 1.2GHz on my phone so i ran most of the tests at 1.14GHz.

    Tested Kernels:
    Stock Teamhacksung V17
    Devil 1.1.6b BFS
    Devil 1.1.6b CFS
    Icy Glitch V14 b
    Semaphore ICS 0.9.5b

    Recommendation:
    Devil 1.1.6b CFS, Icy Glitch V14b (with SmartassV2 and FIOPS), and Midnight ICS (with a tweaked Conservative) are trading blows for the fastest kernel.
    At the time of testing, Midnight is slightly worse in terms of overclocking though, apparently due to different voltages, also it doesn't allow overclocking beyond 1.2GHz.
    But what's interesting is that it achieves great performance while using a tweaked conservative governor.

    Devil 1.1.6b BFS is good but obviously inferior to its CFS brother.

    Semaphore has the lowest cache and memory latency in the multithreaded test, it also has impressive sd card read speed and in general appears super responsive, but it's a bit worse in 3D gaming and especially it lacks "true" overclocking, "live overclocking" changes the bus clock and is way more unstable, in fact on my phone i couldn't run it stable at 1.2GHz.

    All kernels are significantly faster than the stock teamhacksung's kernel, so you have no excuses not to upgrade to one of the popular custom kernels!

    ICS 4.0.4
    Started testing Android ICS 4.0.4 kernels on Slim ICS 3.2.
    All tested kernels are "huge mem" versions with 380+MB of available RAM, without breaking video playback or 720p recording.
    Summary:
    • the stock kernel from Teamhacksung is now a very respectable performer, unless you plan to overclock probably you don't need to install one of the other kernels
    • Semaphore, Midnight and Devil are all very fast and smooth

    Results table:
    https://docs.google.com/spreadsheet/ccc?key=0AuBUEB4dGFSSdHIyN2VIeWU4QnhLOFpJejFPWDh5S1E
    5
    I completely agree on benchmarking every kernel at the same frequency (stock 1Ghz max) but I think there are two different options for further benchmarking:

    1) Benchmark kernels configured as similar as possible regarding CPU governor, IO scheduler, readahead -> comparable results for all kernels.

    2) Benchmark kernels with default settings (only makes sense if all compared kernels are optimized for similar purpose like performance, does not make sense if a kernel does *not* focus on max. performance and uses e.g. Conservative CPU governor as default setting.

    @Pipperox: Would it be possible to check my Mindnight-ICS dev version with your benchmark suite? I'd be really interested in the results as you use the same setup for all kernels (1.2Ghz would not be a problem, does not use LiveOC but old school 1/1.128/1.2Ghz OC).

    Sorry guys, i understand your logic but i do not fully agree with it.
    I'm not comparing overclocked kernels with heavy tweaking of voltages and special settings with which they only work.

    I did the "poor man"'s overclock, setting to 1.2GHz using NSTools, a setting where 95% of phones should have no problem working.

    I think that if some kernels offer you this possibility while others do not, it is fair to use this "advantage" that they have over the other kernels.

    Because a lot of users will have the possibility to do the same as i do, without esoteric knowledge and with just a couple of clicks in the menus.

    That being said, "due to popular demand" :) i will also try to retest those kernels at 1.0GHz as soon as i get a bit of time.

    BUT in my recommendations, i will also consider the overclocking capabilities.

    @mialwe: sure, i'll give a run to your kernel as well!
    4
    Res 1

    One request for the kernel developers:
    could you please post me what are your preferred / recommended settings in terms of Governor and I/O Scheduler?

    Only one configuration per kernel please, as running these tests is rather time consuming.
    3
    Test Settings

    So for anybody who wants to follow the same methodology as I used to test kernels, please pay attention that in some tests i didn't use stock settings, to try to improve the reproducibility of the results.

    Before all test, i put the phone in flight mode, and disable all synch services.

    Antutu: DB I/O and SD Write and Read have poor reproducibility. So i run these tests separately 5 times, and take the best scores.

    RealPi: the number of iterations is increased by factor 10x i.e.: 100000 digits

    MPAC: lots of customization here. Also be careful as it's not very stable and some settings will make it crash.
    All tests: 8 threads (or 8 producer / consumer pairs)
    CPU: 10000000 iterations, use case: integer (i'm considering to add logical too)
    Memory: stock apart for nr of threads. Repeat the test 5 times and get best numbers
    Cache: 40 iterations
    3
    UPDATE: added Midnight ICS to the kernels tested at stock settings!