Rom Benchmarking For Topaz |'Official' Benchmarks |User Submitted Benchmarks|Accurate

ThaDeanesta · Jul 8, 2010

This thread is for the submission and viewing of ROM benchmarks. There are two sections of this post: One for the Official ROM Benchmarks, and one for user submitted benchmarks. This will help us get a more accurate representation of the 'Real' benchmark.

'Official' Benchmarks
-----------------------------------------------------------------------------------------------------
These benchmarks are taken by Scabes24. He fallows a more strict set of rules and posts a more exact representation of ROM speeds. Please use this in conjunction with the User Submitted benchmarks to compare ROMs for speed.

This won't be updated constantly,(Will be updated Monthly) so we encourage you to submit your benchmarks in order to keep things fresh around here.

Click here to view the 'Official' benchmarks

User Submitted Benchmarks
-----------------------------------------------------------------------------------------------------

These Benchmarks are submitted by other XDA-Developers users for us to read. These Benchmarks will be up to date (Hopefully), and the rom date will be on the benchmark submission. These spreadsheets will be STRICTLY Moderated so no bogus gets in there. Remember, some of these benchmarks may not be 100% accurate, because it may differ from device to device (but most likely won't). If you need help, just shoot me a PM.

Please Note: The numbers in this spreadsheet are the time in Milliseconds, and not a 'Score'. The lower the number, the better.
Spreadsheet Link (Results)

IF YOU SUBMIT A BENCHMARK, PLEASE "TEST" THE ROM WITHOUT ANY MODS!
Submitting a Benchmark
1. Flash a new ROM and do a hard reset.
2. Go into airplane mode
3. Disable switching screen off and turning the topaz off
4. DO NOT CHANGE ANY SETTINGS!
5. Install SPB Benchmark (Download HERE, and it IS freeware!)
6. Reboot (soft-reset)
7. Run SPB Benchmark : Select the "Main Test" option, and disable external software tests like word and IE or file explorer (whole category)
8. Submit results using this form.

Scabes24 · Jul 9, 2010

Changes for 'Official' Benchmarks

Code:

[SIZE=6][COLOR=red][B]Changelog[/B][/COLOR][/SIZE]

[B][SIZE=4][COLOR=green][U][COLOR=Black]Changes 9/6/10[/COLOR][/U][/COLOR][/SIZE][/B] [B][COLOR=Red]*[/COLOR]Will be updating over the next few days[COLOR=Red]*[/COLOR][/B]

•If anyone would like any [B]stock/shipped ROMs[/B] tested let me know.

[B][SIZE=4][COLOR=Green][U][COLOR=Black]Changes 7/30/10[/COLOR][/U][/COLOR][/SIZE][/B]

•Added [B]Energy CHT[/B] - [B]DinikGlass[/B] 23673/21911 (July 17) - 23121 (July 21)
•Added [B]Energy[/B] [B]Standard Sense[/B] 23673/21911 (July 17)
•Added [B]Energy[/B] [B]Titanium[/B] and [B]Sense 2.1[/B] 23569/21911 July 14
•Added [B]Energy[/B] [B]Standard Reference[/B] 23673/21911 (July 17)
•Added [B]Energy[/B] [B]GTX[/B] 23121 (July 21)
•Added [B]NhatHoa 4.0[/B] 23128 (July 18)
•Added [B]Dutty's HG R15[/B] 23569/21905 (May 31)
•Added [B]Dutty's Nu Era V1 Redemption[/B] 23128 (July 23)
•Added [B]LouMeiYin Yin.09[/B] 23568/21905 (June 11)
•Added [B]QBUS22[/B] 23569/21909 (July 12)
•Added [B]Melissa v0.3A[/B] 21887 (July 8)
•Added [B]MaryOne[/B] 23128/23569/21907 (July 15)
•Added [B]Tess Prime V[/B] 21907 (June 30)
•Added [B]NullPointer Evo v3.5[/B] 23569/21907 (June 28)
•Added [B]Cheetah 4.3r[/B] 21901 (July 20)
•Added [B]Cancer TIR-2[/B] (July 2) and [B]TIR-3[/B] (July 10)
•Added [B]Avatar IV[/B] 23128 (July 19) and 21911 (July 20 - in progress)
•Added [B]TopazNihon[/B] 21911 (July 19) (in progress)
•Added [B]Googy ROM v3.4[/B] 219xx (June 14) (w/ RAM Cab)
•Added [B]Column[/B] - '[B]Average[/B]' of columns H-Z ([B]NOT an overall score[/B])
•Added [B]Column[/B] - '[B]Sum[/B]' of columns H-Z ([B]NOT an overall score[/B])
•Added [B]Column[/B] - '[B]Overall Score[/B]' - (still calculating)
•Added [B]Column[/B] - '[B]Free RAM on first boot[/B]'
•Added [B]Column[/B] - '[B]Free Storage on first boot[/B]'

joooe · Jul 12, 2010

And it would be great to have some kind of spreadsheet formula to calculate the rating of each test (average function, maybe?). Just for us to know which ROM is the fastest, overall.
This initiative is great!

BumAround · Jul 17, 2010

Guys, a benchmark is a benchmark, you must try the ROM for a week and decide for yourself.

Scabes24 · Jul 23, 2010

I think most people that aren't very interested in each specific benchmark test would much rather see some type of 'Overall score' for each ROM, so if someone knows of a simple formula or way to average the benchmark scores and place them on a 1-10ish scale (based on speed and possibly graphics) please PM me and I will add the stats.

bnm7bnm · Jul 24, 2010

Scabes24 said:
I think most people that aren't very interested in each specific benchmark test would much rather see some type of 'Overall score' for each ROM, so if someone knows of a simple formula or way to average the benchmark scores and place them on a 1-10ish scale (based on speed and possibly graphics) please PM me and I will add the stats.

you can try make it like in previous benchmarking thread - relate to the fastest score of each column as 100% and others as "current"*100/fastest but for overall score need to know which of the colums really effects performance feeled by user to give em priority in overall score

Scabes24 · Jul 24, 2010

bnm7bnm said:
you can try make it like in previous benchmarking thread - relate to the fastest score of each column as 100% and others as "current"*100/fastest but for overall score need to know which of the colums really effects performance feeled by user to give em priority in overall score

Yeah I was trying to figure out how to determine what the 'best/max' score would be, going with the current highest/fastest score for each column (or even groups of columns) should work. As far as overall I was thinkin of seperating it into 2 or 3 scores. one for read/write/copy speed and maybe another for graphics.. but you are right, I would need to know which ones effect performance. Also I'm not quite sure how much of an affect the different (21xxx vs 23xxx) builds have on the stats.

I put the average and sum of the columns (except free RAM and storage) for a few energy ROMs, Dutty's Redemption, and Cheetah to see if it they were comparable and accurate to performance but havn't looked at them enough yet, most likely meaningless.

mondilv · Jul 26, 2010

ThaDeanesta said:
I will do this. First, however, we need benchmarks to make the score. I will handle everything =].

Not think, thats right way: all b/m must be at the same device, this the same condition (like disabled manila, sound, phone, phone canvas, the same time after flash/boot etc.).
Devices use different nand sometimes, with diff FS speed. So for compare roms, you need to use the same device.
It's creazy work.

Pietrucci did use it before. Ask him, how right to interpret results.
(the every test element have the own "weight" in the test. Need the special formula for calculate).

ljankok · Jul 26, 2010

mondilv said:
Not think, thats right way: all b/m must be at the same device, this the same condition (like disabled manila, sound, phone, phone canvas, the same time after flash/boot etc.).
Devices use different nand sometimes, with diff FS speed. So for compare roms, you need to use the same device.
It's creazy work.
Pietrucci did use it before. Ask him, how right to interpret results.
(the every test element have the own "weight" in the test. Need the special formula for calculate).

Yes, this is true. There is a special formula for this. Piertrucci knows this, so it is a good idea to ask him instead of doing the same work twice.

ThaDeanesta · Aug 5, 2010

i have scabes24 doing everything on the same device

LLKS · Sep 2, 2010

Hello!

When you do a benchmark with SPB Benchmark this generates an XML file that can be uploaded on the website of SPB and compare with other devices. It also lets us do a custom table of devices and gives the% of one over the other in each test.

So if you still have all generated XML can be loaded into SPB under section Benchmark>Visualice Test Results(on the left column) and will be much easier to see which is better.

I'm Spanish my good english mercy to Google

lmemperador · Sep 2, 2010

LLKS said:
Hello!

When you do a benchmark with SPB Benchmark this generates an XML file that can be uploaded on the website of SPB and compare with other devices. It also lets us do a custom table of devices and gives the% of one over the other in each test.

So if you still have all generated XML can be loaded into SPB under section Benchmark>Visualice Test Results(on the left column) and will be much easier to see which is better.

I'm Spanish my good english mercy to Google

good idea.

Scabes24 · Sep 3, 2010

I actually do have most of the XMLs, ill take a look. I was also thinkin about that a while back.

LLKS · Sep 3, 2010

Hi!

I was lookin how the sistem rates and I think that the results all are in milisecond. Then the lower of all sums is the speedy ROM.

BUT I think som results have more weight in the result.

For example Arkball test measures ms between. An speedy ROM has a resul of 3 and a slow one has 12. But this is nothing wen compared to the time spend in list 2000 files (2-3 thousand)

If we want to compare the scores with a formula i think we have to determine how make som values weigth in more than others.

Sorry for my english

lmemperador · Sep 3, 2010

LLKS said:
Hi!

I was lookin how the sistem rates and I think that the results all are in milisecond. Then the lower of all sums is the speedy ROM.

BUT I think som results have more weight in the result.

For example Arkball test measures ms between. An speedy ROM has a resul of 3 and a slow one has 12. But this is nothing wen compared to the time spend in list 2000 files (2-3 thousand)

If we want to compare the scores with a formula i think we have to determine how make som values weigth in more than others.

Sorry for my english

yes, the user of xda, pietrucci had the formula, but i think it is possible to make a new one, with only logic and ponderation of all the factors, or the most important at least.

Scabes24 · Sep 3, 2010

Yeah I tried contacting pietrucci a while ago with no response.

We discussed this before, the simple way would be to mark the highest and lowest scores for each column and place them on a grading scale from 1-100 (1-X). Could be done without the XML docs. Then graph it ..or just grade each ROM with a number.

But yeah we would have to figure out which tests are relevant to speed. I personally don't know enough about each specific test to figure that out. Maybe ThaDeanesta would know a bit more.

lmemperador · Sep 3, 2010

Scabes24 said:
Yeah I tried contacting pietrucci a while ago with no response.

We discussed this before, the simple way would be to mark the highest and lowest scores for each column and place them on a grading scale from 1-100 (1-X). Could be done without the XML docs. Then graph it ..or just grade each ROM with a number.

But yeah we would have to figure out which tests are relevant to speed. I personally don't know enough about each specific test to figure that out. Maybe ThaDeanesta would know a bit more.

yes of course.

well i think one of the importants are the "copy files" times? for example...

i hope all the people post what they think is relevant to try t make a simple formula that helps in a single view of the numbers to decide. maybe not accurate, but more simple than now.

i know all the data are important but it is kaos ....

well, i have decided my rom for long time ago so, it is basically for helping people here

Scabes24 · Sep 5, 2010

lmemperador said:
yes of course.

well i think one of the importants are the "copy files" times? for example...

i hope all the people post what they think is relevant to try t make a simple formula that helps in a single view of the numbers to decide. maybe not accurate, but more simple than now.

i know all the data are important but it is kaos ....

well, i have decided my rom for long time ago so, it is basically for helping people here

Yeah other users input would be helpful. I'm going to finish updating the latest releases and then I'll mark the highs and lows. Then we can move foward from there

lmemperador · Sep 5, 2010

Scabes24 said:
Yeah other users input would be helpful. I'm going to finish updating the latest releases and then I'll mark the highs and lows. Then we can move foward from there

yes of course. the people may be invloved in this
*****************************************
i have an idea, can we make a survey to let the people vote for the prefered rom, like this user did in this thread to decide the build?¿?

http://xdaforums.com/showthread.php?t=722602

at the top (view poll results)

also it can be done by google forms

you know, all the peolple said that numbers are numbers but the overall experience with a rom is a fact. so.... they could vote for this.

as we try to improve the benchmarks view.

hope this helps

LLKS · Sep 6, 2010

Things i think are important and why:

Free RAM: If you try to surf the web without memory will go slow. Also Sense will work slow

Directory list of 2000 files: Because if you enter explorer with a slow ROM in this test and you want enter Windows directory it freeze the sistem. Is a shame for that ROM.

File Explorer large Folder List: Same as above.

Arkaball frames per second: For thos who want gaming and may be ¿video?. It depends on the tech used in the game (hardware acceleration¿?)

Things to determine:

Read-Write-Copy: To determine how important it must first determine where copying, whether in RAM or SD.

CPU Test: I think these aren't important because results are very pretty the same.

These are my thoughts.... may be wrong

Rom Benchmarking For Topaz |'Official' Benchmarks |User Submitted Benchmarks|Accurate

Retired Forum Moderator

Inactive Recognized Themer

Senior Member

Senior Member

Inactive Recognized Themer

Senior Member

Inactive Recognized Themer

Senior Member

Retired Recognized Developer

Retired Forum Moderator

Member

Senior Member

Inactive Recognized Themer

Member

Senior Member

Inactive Recognized Themer

Senior Member

Inactive Recognized Themer

Senior Member

Member

Similar threads

Top Liked Posts