FORUMS
Remove All Ads from XDA

 View Poll Results: Do you find this benchmark's results helpful while choosing rom for daily use?

Yes, I only use roms that are 'on the top'
 
597 Vote(s)
50.64%
Yes, but I use roms I like, not even if they aren't 'on the top'
 
460 Vote(s)
39.02%
No, I think those results have no impact on daily usage performance
 
63 Vote(s)
5.34%
Sorry, but what are you talking about?
 
59 Vote(s)
5.00%

[PRJ]Rom benchmark by daeron

102 posts
Thanks Meter: 0
 
By daeron.be, Senior Member on 18th May 2009, 05:13 AM
Post Reply Email Thread
Hello guys, I've decided to make a benchmark of latest cooked roms, because nobody else did anything like that for our beloved Blackstone.

It happened again - our spreadsheet grew too big for another online spreadsheet hoster. Therefore, we have to swich different way to distribute our work - via attachments in this topic. To get newest results, just look for a latest post from me, Houms or MAMeingast. Sorry for the inconvenience, but I'm sick and tired of all problems with those services.


To measure performance of each rom I use two free programs:
SPB Benchmark (v1.6.0, build 62)
TestOpenGL by heliosdev (v. 09 May 2009)
Coreplayer (always say what version you used - e.x. v1.3.2a) * optional, it's not a free app!

Matrix trailers:
http://shintak.info/ppc/download/bet...0_1500_128.avi
http://shintak.info/ppc/download/bet...2_1024_128.avi
http://shintak.info/ppc/download/bet...40_512_128.avi

To score "overall rom performance" I'm checking how well is a rom doing in a single benchmark (rom's result/best result or best result/rom's result, it depends on whatever better score is a higher number or a lower) and taking an average of all one rom's results. More % does the rom score, better its overall performance is. If any rom scored 100%, it'd mean that it won all benchmarks.

I've added 'weights' function, It's working and they affect overall score. Give me your feedback about how big they should be for each test. If you export this spreadsheet you can edit them to fit your needs!

What are we doing if rom fails some test? That's easy. We make this test's cell white, and put there an average rom's result.

Test Method:
1. Flash the rom and do a hard reset.
2. Go into airplane mode
3. Disable switching screen off and turning the blackstone off
4. Disable all today screen apps and plugins etc * optional
5. install benchmarking apps
6. reboot (soft-reset)
8. run SPBbenchmark : disable external software tests like word and IE or file explorer (whole category)
9. run TestOpenGL for about 3 cycles of benchmarks (each cycle is 4 benchmarks) and then turn it off
10. run three matrix trailers, note down the results (%). Use default Coreplayer settings and run benchmark immediately after files are loaded.
11. collect results



FAQ:
1. Shouldn't you mention radio version you're using every benchmark? Does radio version affect the results?
I hope it doesn't, because I turn all connections off - I go into the airplane mode

2. Does hard reset after each flashing affect the results?
I have no idea, but somebody said it could happen, so I'm doing it every time I run a benchmark.

3. What are these benchmarks for?
These benchmarks are just for me to help me choose the "best" rom in terms of performance. And I'm a good guy and I let you see these results and even correct me if I make a mistake . And Houms asked me if he can join. Sure! And now when I cannot test rom for myself, you all are welcome to send your own results, as long as you don't cheat and follow the rules how to do it (see the sacred METHOD above)

4. What version of rom (e.x. Miri's) are you testing? Big, small, Premium, Deluxe, Standard, Non-Standard etc?
Well, I try to specify what version I test, but sometimes I forget. And yeah, I always use Standard Miri's roms. But does it make any difference, if they only differ in things such as TF3d or not-TF3d, if I disable it?

5. OM*G! My rom is the best and in your tests it's not! Remove these results immediately, because they confuse people, and don't you dare to test my rom ever again!
Yeah. Sue me.

6. Can I send you my results?
Sure, I'd be glad to add them but you have to follow the Method described and send them via e-mail < daeron [at] [put-my-forum-nick-there] >.

7. Are those benchmarks accurate?
Well, I hope they are . Some roms were tested twice, and differences were always smaller than 1%, but you can say that they might vary by 1% (I'm talking about overall performance difference!). If you aren't sure, you can always check. But do this on a clean rom with the method above.

Big THX goes to houms and MAMeingast for hard work on this project.
 
 
18th May 2009, 05:17 AM |#2  
onkolog's Avatar
Senior Member
Flag Cleveland
Thanks Meter: 9
 
More
Quote:

mario lite 2009.05.01 (I have no idea which rom it is, could anyone give me a hand? it was cooked 2009.05.01, it might be some Onkolog's rom)

Yessss)))it's me)))

You know why my ROM has worse parameters?)))I tell you)))because I don't use any new graphic drivers and all my tweaks was set in average parameters)))
But in my next ROM I'll include new graphic drivers...
The Following User Says Thank You to onkolog For This Useful Post: [ View ] Gift onkolog Ad-Free
18th May 2009, 05:22 AM |#3  
OP Senior Member
Flag Wschowa
Thanks Meter: 0
 
More
Quote:
Originally Posted by onkolog

Yessss)))it's me)))

You know why my ROM has worse parameters?)))I tell you)))because I don't use any new graphic drivers and all my tweaks was set in average parameters)))
But in my next ROM I'll include new graphic drivers...

Which version precisely?
It's hard to say why you scored worse than Dutty... But I guess if you'll study his rom, you might improve yours.
18th May 2009, 05:33 AM |#4  
onkolog's Avatar
Senior Member
Flag Cleveland
Thanks Meter: 9
 
More
)))HAHA)))I won't study his ROM because I have my head on my shoulders...maybe his ROM has better scores,but my ROM is bug freeeee)))Also all benchmark tests always show different results)))If you do twice one test in one device you will get different results 100% sure)))
18th May 2009, 06:07 AM |#5  
KukurikU's Avatar
Senior Member
Flag Rehovot
Thanks Meter: 46
 
More
Quote:
Originally Posted by onkolog

)))HAHA)))I won't study his ROM because I have my head on my shoulders...maybe his ROM has better scores,but my ROM is bug freeeee)))Also all benchmark tests always show different results)))If you do twice one test in one device you will get different results 100% sure)))

sure are bug free...whats left of the original OEM or SYS. what ever could be a problem is out. forget about light, super light and hyper light and try to make a complete one. that's why is called cooking. the chefs are mixing tens of "ingredients", sometimes even different builds to get a good ROM.
chicken and water only don't make a good soup. add some vegetables, salt, noodles...all ingredients.
18th May 2009, 06:43 AM |#6  
onkolog's Avatar
Senior Member
Flag Cleveland
Thanks Meter: 9
 
More
No my friend)))I cannot decide for you or for somebody how to live)))you must decide it for yourself)))that's why I make ROMs only with necessary stuff and you GUYS can decide what you want see in your ROM...I made full ROM for russian users and if you know russian (probably you're not))) you can read feedbacks about my ROM and I give you 100% you won't find any bugs in it...It's same like I gave to XDA users,but it's russian languages
This is it - http://4pda.ru/forum/index.php?showtopic=119115
I'm mah in this forum
18th May 2009, 08:13 AM |#7  
Laurentius26's Avatar
Inactive Recognized Developer
Thanks Meter: 1,154
 
1010
Donate to Me
More
New benchmark tool
Awsom to see these benchmarks, thank you for your work.

Grtz,

Leo
18th May 2009, 08:13 AM |#8  
cyphol's Avatar
Senior Member
Flag Pasir Puteh
Thanks Meter: 94
 
Donate to Me
More
Quote:
Originally Posted by daeron.be

Hello guys, I've decided to make a benchmark of latest cooked roms, because nobody else did anything like that for our beloved Blackstone.

Right now I've tested only few roms, but I'll do my best to test even more roms, so everyone can see which of them is the fastest. I've used free SPB Benchmark (v1.6.0, build 62) and TestOpenGL by heliosdev (v. 09 May 2009). Every benchmark was made with Today screen clean (manila and other stuff disabled).

I've already tested:

Onkolog's lite May 2,2009
PhamQuang V31
miri 6.5 v12.0
duttys 3.3
L26 6.1 2.2


To see detailed results, click the link below:

http://spreadsheets.google.com/ccc?k...YOv7XgbHRVbbmA

Right now, the fastest rom tested is Dutty's V3.3 XT SHIP ROM
See http://www.duttythroy.net/ or http://forum.xda-developers.com/showthread.php?t=507372 to get this piece of art!


Put one more. Topix 1.6.
18th May 2009, 08:31 AM |#9  
Senior Member
Flag Tianjin, China
Thanks Meter: 0
 
More
You I have to say, although I usually run Dutty, and the one time I tried to run Laurentius' ROM it complained to me every 5 seconds that my network wasn't in the database (it's only the largest cell network in the world), these results actually make me want to try Laurentius' ROM again. It looks like he won to me. Close but behind on things that were very short, either close or ahead in some important things, like opening 100 10k files (sounds like Manila), that's where I'd like to shave off a few seconds. 10ms for writing one long file doesn't matter much. I call this a tie for Laurentius and Dutty, at least at first glance. Especially since Dutty's didn't finish every test.

Rob
18th May 2009, 08:51 AM |#10  
KukurikU's Avatar
Senior Member
Flag Rehovot
Thanks Meter: 46
 
More
Quote:
Originally Posted by cyphol

Put one more. Topix 1.6.

there is a 1.6.1 already
port2life
18th May 2009, 09:34 AM |#11  
Guest
Thanks Meter: 0
 
More
nice work
dutty may be the fastest rom, but so buggy

i'm using many dutty's vesion, even dutty V3.3.
just try miri 6.5 v12 1 day ago, not found any bug and feel faster than dutty
Post Reply Subscribe to Thread

Tags
benchmark rom performance

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes