Spb Benchmark for (some) Kaiser ROMs

[hch_111]

Hi folks,

I guess no one here will/still remember me.

It's been a couple of years since I was active here cooking ROM for the good old PPC 2003.

Well, to get straight to the point, I am here a bit confuse which ROM should I use for my Kaiser.

So I ended up doing some benchmarking.

It might not include the ROMs that everybody wants since this was initially done for my personal/private research.

The way I do this is by benchmarking fresh ROM without changing anything. Everything are ROM's default.
The only thing I do is install Spb Benchmark in to the memory, soft reset, and run the tests.

The reasons why I only put these ROMs (not many) was because I only need a lite roms.
In this benchmarking, I also put some full roms out of curiosity whether Full is better that Lite.

I also come accross some pagepool thread that tells that you can change pagepool on the fly. So I ended up trying to change the pagepool and compare them with the original pagepool.


The ROMs included in this benchmarking are:

Dutty's v1.1 (original 20mbpp). (now INCLUDED in color rank)
Dutty's SupaLite (original 12mbpp).
Dutty's SupaLite (tweaked 64mbpp).
Dutty's v2 Full (original 32mbpp).
Dutty's v2 Full (tweaked 64mbpp).
Dutty's v2 Med (original 12mbpp).
Dutty's v2 Med (tweaked 64mbpp).
Dutty's v3 (original 24mbpp).
Dutty's v3 (tweaked 64mbpp).
Dutty's v4 (qouted from Dutty's benchmark result). (NEW)
Alex ATT (original 16mbpp).
Alex ATT (tweaked 64mbpp).
Alex HTC (original 16mbpp).
Alex HTC (tweaked 64mbpp).
Jack v1.1a Full (original 32mbpp). (NEW)
Jack v1.1a Full (tweaked 64mbpp). (NEW)
Dejan v1.03WMP (original 16mbpp). (NEW)
Dejan v1.03WMP (tweaked 64mbpp). (NEW)


So after all done, eventhough I do all these just for my personal research, as well as costing me quite a lot of time & energy, I was thinking why not share it with everyone in this our most beloved forum regardless whether readers find it usefull or not.

Last but not least, I would like to say HUGE thanks & salute to all of you ROM makers for creating such a wonderful ROMs, as well as the other gurus that I haven't had a chance to try their roms.

You all are the people that keeps people like me alive & busy poking our dear gadgets in this boring routine life.

ps: My apology to everyone if somehow I did something wrong in this test (like try changing the pagepool, different results from your test, etc).
Again, like I said before, this was initailly for personal use.
I just felt that it would be a waste if I just keep it in my harddisk.


Thank you very much.


FYI : Up until now I still can't decide which one I should use.
Please throw me some suggestions.

[ecltech]

Great comparison. :) Thanks for posting.

[AldaR]

great. ty..


ı need this

edit : v3 64mbpp no cube ? where is it. ı cant find in duttys space ?

[Fared]

This was infinatly helpful, thank you. The only way it could be better was if deviations relative to their peers were calculated; even so, it is very good.

FYI : Up until now I still can't decide which one I should use.
(I was looking at either SupaLite 64mbpp or HTC 64mbpp).
Please throw me some suggestions.

I am going to try the supalite. It is better in the first set of rows, which probably are a better determinant of which is "better".

[duttythroy]

This is my v4 I'm working on just fine tuning v3 with same prgrams same 24 mb pagepool and I feel like this is trully awesome for a full Rom. Have a Look.:D

[duttythroy]

Hi folks,

I guess no one here will/still remember me.

It's been a couple of years since I was active here cooking ROM for the good old PPC 2003.

Well, to get straight to the point, I am here a bit confuse which ROM should I use for my Kaiser.

So I ended up doing some benchmarking.

It might not include the ROMs that everybody wants since this was initially done for my personal/private research.

The way I do this is by benchmarking fresh ROM without changing anything. Everything are ROM's default.
The only thing I do is install Spb Benchmark in to the memory, soft reset, and run the tests.

The reasons why I only put these ROMs (not many) was because I only need a lite roms.
In this benchmarking, I also put full roms (Dutty's) out of curiosity whether Full is better that Lite.

I also come accross some pagepool thread that tells that you can change pagepool on the fly. So I ended up trying to change the pagepool and compare them with the original pagepool.


The ROMs included in this benchmarking are:

Dutty's v1.1 (original 20mbpp).
Dutty's SupaLite (original 12mbpp).
Dutty's SupaLite (tweaked 64mbpp).
Dutty's v2 Full (original 32mbpp).
Dutty's v2 Full (tweaked 64mbpp).
Dutty's v2 Med (original 12mbpp).
Dutty's v2 Med (tweaked 64mbpp).
Dutty's v3 (original 24mbpp).
Dutty's v3 (tweaked 64mbpp).
Alex ATT (original 16mbpp).
Alex ATT (tweaked 64mbpp).
Alex HTC (original 16mbpp).
Alex HTC (tweaked 64mbpp).

So after all done, eventhough I do all these just for my personal research, as well as costing me quite a lot of time & energy, I was thinking why not share it with everyone in this our most beloved forum regardless whether readers find it usefull or not.

Last but not least, I would like to say HUGE thanks & salute to Dutty and Alex for creating such a wonderful ROMs, as well as the other gurus that I haven't had a chance to try their roms.

You all are the people that keeps people like me alive & busy poking our dear gadgets in this boring routine life.

ps: My apology to Dutty and Alex if somehow I did something wrong in this test (like try changing the pagepool, different results from your test, etc).
Again, like I said before, this was initailly for personal use.
I just felt that it would be a waste if I just keep it in my harddisk.
Afterall, you both are MY HEROs.


Thank you very much.


FYI : Up until now I still can't decide which one I should use.
(I was looking at either SupaLite 64mbpp or HTC 64mbpp).
Please throw me some suggestions.

great work, Its hard to compare roms with benchmarks as Roms are equipped with different programs built in but this does allow the chef to look at how he is doing and also try to change to make his rom better. In recent I found the benchmarks really great assets for my next Roms:)

[Bronx31]

This is my v4 I'm working on just fine tuning v3 with same prgrams same 24 mb pagepool and I feel like this is trully awesome for a full Rom. Have a Look.:D

With Dutty telling you and posting this in your thread, to me, He's telling you something :cool:.. GREAT work by the way

[hch_111]

With Dutty telling you and posting this in your thread, to me, He's telling you something :cool:.. GREAT work by the way

From what I can catch, he might want to me to put the unreleased v4 benchmark result in the table, which I already did.

However, like I mentioned before, each person might come up with different result.
Since our man Mr. Dutty hasn't officially release this v4 yet, I just put his benchmark result in.

Once he released it, I'll do it with my benchmark.
Because to be fair, each rom have to be done the same way.
Can't be benchmarked differently, like: install spb benchmark on the memory in 1 ROM, and install on the storage on the other rom in the same table, or else.

[hch_111]

This is my v4 I'm working on just fine tuning v3 with same prgrams same 24 mb pagepool and I feel like this is trully awesome for a full Rom. Have a Look.:D

When will you release it Dutty?

I can't hold my horse any longer.
I'm hungry for your new ROM(s).

Thanks.

[jackleung]

Good stuff! Can you add mine to the table too?

I can provide you w/ the SPB data later.

[hch_111]

Good stuff! Can you add mine to the table too?

I can provide you w/ the SPB data later.

Man, you read my mind.

Is this the link: http://rapidshare.com/files/83243535/FullROM.rar.html?

[sunstorm]

... I also come accross some pagepool thread that tells that you can change pagepool on the fly. So I ended up trying to change the pagepool and compare them with the original pagepool...


Hi hch_111,

what is the procedure or application for changing the pagepool on the fly? I'm assuming this is done without a reflash? Very interesting if no reflashing is involved. hope you can guide me along on this.

[pserbanp]

Great news like always Dutty (V4).
I think with you we can test the maximum nr. of flashing (1.000.000) :D
Keep the good work Sir Dutty (you earn this title for this comunity).and also thanks to everybody who can support this hard work.

Very good Benchmark comparision.

[hch_111]

Hi hch_111,

what is the procedure or application for changing the pagepool on the fly? I'm assuming this is done without a reflash? Very interesting if no reflashing is involved. hope you can guide me along on this.

Yes buddy, it involves reflashing.

I can't pinpoint which particular thread explaining about how to change pagepool because there are so many of them.

This will be the search link: http://forum.xda-developers.com/search.php?searchid=9572410

(can someone help me how to put hyperlink, please? )

[jackleung]

Man, you read my mind.

Is this the link: http://rapidshare.com/files/83243535/FullROM.rar.html?Yes. I also have a non-full variant in http://forum.xda-developers.com/showthread.php?t=357490. If you want, you can take the result I tested for both version.

For pagepool, just use the tool from post#6 in http://forum.xda-developers.com/showthread.php?t=357617 on yr rom before you flash it.

[_Dejan_]

@hch_111: Please can you test My ROM 1.03WMP for comparation with this roms? On my device with 16MBPP I get SPB benchmark index 447.68...
Thanks.

Bye, _Dejan_

[hch_111]

Yes. I also have a non-full variant in http://forum.xda-developers.com/showthread.php?t=357490. If you want, you can take the result I tested for both version.

For pagepool, just use the tool from post#6 in http://forum.xda-developers.com/showthread.php?t=357617 on yr rom before you flash it.

For pagepool, that was one of the thread I've read and done.

For benchmark, your result slightly different from mine (don't know why).
Did you put it on Flight Mode before you soft reset & start the benchmark?

By the way, I've add your Full rom to the updated list.

[hch_111]

@hch_111: Please can you test My ROM 1.03WMP for comparation with this roms? On my device with 16MBPP I get SPB benchmark index 447.68...
Thanks.

Bye, _Dejan_

Sure.
Can you gimme the download link?

Thanks.

[MickyMax]

http://forum.xda-developers.com/showthread.php?t=356892 :)

[_Dejan_]

Sure.
Can you gimme the download link?

Thanks.


Here is download link: rapidshare (http://rapidshare.com/files/82563253/Dejan_v1.03WMP.rar.html)

For aditional info you can look in this thread (http://forum.xda-developers.com/showthread.php?t=356892) in post #4

Thanks.

Bye, _Dejan_

[hch_111]

Got it.

Downloading, ....

[_Dejan_]

Got it.

Downloading, ....

Tnx, If you will have any free time maybe If you can test it with 32MB PagePool.
Realy thanks for yout tests and time.

Bye, _Dejan_

[johanromijn]

Very nice benchmarks!! (Nice spreadsheet!)

Maybe you can benchmark Q-Mobile 1.11 ROM (http://forum.xda-developers.com/showthread.php?t=352418), (or wait a few days for v1.2)
I use that ROM for a few weeks already, and it's very fast and stable.

Tnx for doing this!

[jackleung]

I just found the spreadsheet w/ my result. Thanks a lot for yr time and hardwork.

Maybe I should create a super-lite rom for the competition. ;)

[hch_111]

I just found the spreadsheet w/ my result. Thanks a lot for yr time and hardwork.

Maybe I should create a super-lite rom for the competition. ;)

This is actually not a competition.
The reason is because up until now I can't decide which one I should use.
That's why I keep benchmarking.
(I start to think learning how to cook ROM, but looks complicated).

Instead of competition, let's make this as a tool for ROM creators to help each other creating a dream ROM for all of us.

[hch_111]

Actually, my benchmark result can not be compared with other people's results.
So does the other way around.

IMHO, results can only be compared if the person who test it using EXACTLY the same procedure for all ROMs he/she tests.
Eq.:
- what to turn on or off before benchmarking.
- any Recent Programs on the Start Menu list or not.
- running Spb Benchmark first, exit, soft reset, then run it from Recent Programs dropdown list or straight away running Spb Benchmark after installing it.
- install to memory or storage card.
- etc.

Install & running Spb Benchmark from storage card create slower results.
But I think it doesn't matter because once you decide to install it that way (to Storage card), the other ROMs has to be tested through Storage card as well.

In conclution, regardless who's doing the tests or how he/she tests it, all ROMs has to be tested Apple-to-Apple.
By doing this, it does not matter anymore who got higher results. All might be different.


One thing that might be the same, ... the difference in percentage (maybe).

For example:

Me testing :
ROM A results 1000,
ROM B results 1500,
ROM C results 2000.

Other tests the same ROMs :
ROM A results 400,
ROM B results 600,
ROM C results 800.

Big difference in results, but the percentage are nearly the same. Got the idea?

I could be wrong but this is just my brainstorming.


Thank you.

[jackleung]

Install & running Spb Benchmark from storage card create slower results.
But I think it doesn't matter because once you decide to install it that way (to Storage card), the other ROMs has to be tested through Storage card as well. ic. No wonder some of the result is a little bit off than what I did as I always install SPB on main memory. But that's ok as long as all of them are under the same condition (or nearly the same condition as some may had more preload programs, todays' plugin ..ect). The spreadsheet really helps a lot for rom maker to seek where they can make improvement, and for rom user to decide what rom to use.

As for the improvement, I am willing to share all my tricks and tweaks in case anyone found they are useful. Maybe we should start a thread for this.

[jackleung]

(I start to think learning how to cook ROM, but looks complicated).
It's not as complicate as you think w/ the help of kitchen tools (although I only use and follow 1/3 of it), but it takes a lot of time to compile and test.

[mrnamster]

Maybe I should create a super-lite rom... ;)

Please do! I was curious to flash your roms but I did not want to deal with those extra not-so-freewares. I'm with those who are seeking clean, lite, speedy, and battery-friendly roms, and I still haven't found what I'm looking for.

This spreadsheet comparison is awesome, thanks!

[mrnamster]

It would be great to have Q-Mobile up there too!

[_Dejan_]

@hch_111: Im see you are do test with my ROM's. Thanks :D So is not so bad :D

What is with "Arkaball frames per second"? Is limit 200point or is here something else wrong? maybe device driver limit... I must check why I ger Error in "File Explorer large folder list" maybe because I don't have fexplorer...

Im see something about PagePool. Who use CE V.1933 B 18125 with 64MB loose in speed and who use CE v.1948 B 20000 profite with 64Mb PP :)

[jackleung]

Hi Hch_111,
Thanks for the update. Can you update my rom name as "1.1a Full" as I do have a liter version which should perform a little bit better. And my rom should be the best in "Pocet IE Jpeg load" w/ score 120. Thanks for your time and effort.

[hch_111]

Hi Hch_111,
Thanks for the update. Can you update my rom name as "1.1a Full" as I do have a liter version which should perform a little bit better. And my rom should be the best in "Pocet IE Jpeg load" w/ score 120. Thanks for your time and effort.

Sorry Jack, my bad.
Sometimes looking at numbers for such a long time can make your eyes blurry.
Hahahhahaa.

I've updated it already.

Thanks for the correction.

[hch_111]

@hch_111: Im see you are do test with my ROM's. Thanks :D So is not so bad :D

What is with "Arkaball frames per second"? Is limit 200point or is here something else wrong? maybe device driver limit... I must check why I ger Error in "File Explorer large folder list" maybe because I don't have fexplorer...

Im see something about PagePool. Who use CE V.1933 B 18125 with 64MB loose in speed and who use CE v.1948 B 20000 profite with 64Mb PP :)

Is it?

I didn't notice that far.
In that case, could this be a clue for you all masters?
For me, there's no way I can figure it out why v1933 becomes lower in 64mbpp while v1948 becomes faster.
There must be something in it which I believe only you gurus know.

But if this is a true, then by finding this clue (as well as another clues), we are one step closer to the creation of THE PERFECT ROM.

Thanks for the sharp eye.

[gasman858]

Great work hch,
I don't know how you will keep up with new cooked roms released every few seconds on this site - especially with WM6.1 rom leaks!
Using sbp benchmark, I am noticing that most cooked roms perform only slightly better (~10%, probably less in real-world conditions) than the base roms as long as the registry editing is approximately the same - e.g. if base rom is att 1.61, using same stock rom with regedits for filecache, gdi, etc. yields nearly same results. I haven't been able to test pagepool changes, cuz I am unaware of the on-the-fly change you describe (system seems to revert back to orig. pagepool after SR).
Also, anyone who has reviewed these benchmarks immediately notices the horrible file system performance (which is why an easy "benchmark" is how long it takes to open the /windows dir with file explorer) - this is an area our illustrious cooks should be working on.
For those not familiar with the definitions of the SBP benchmark indices & tests -click and learn below:
http://www.spbsoftwarehouse.com/products/benchmark/indices.html?en
http://www.spbsoftwarehouse.com/products/benchmark/tests.html?en
Cheers -