FORUMS

Analysis & Opinion

Top Forum Discussions

Rom Benchmarking For Topaz |'Official' Benchmarks |User Submitted Benchmarks|Accurate

413 posts
Thanks Meter: 162
 
By ThaDeanesta, Retired Forum Moderator on 8th July 2010, 02:23 AM
Post Reply Subscribe to Thread Email Thread
This thread is for the submission and viewing of ROM benchmarks. There are two sections of this post: One for the Official ROM Benchmarks, and one for user submitted benchmarks. This will help us get a more accurate representation of the 'Real' benchmark.


'Official' Benchmarks
-----------------------------------------------------------------------------------------------------
These benchmarks are taken by Scabes24. He fallows a more strict set of rules and posts a more exact representation of ROM speeds. Please use this in conjunction with the User Submitted benchmarks to compare ROMs for speed.

This won't be updated constantly,(Will be updated Monthly) so we encourage you to submit your benchmarks in order to keep things fresh around here.

Click here to view the 'Official' benchmarks

User Submitted Benchmarks
-----------------------------------------------------------------------------------------------------

These Benchmarks are submitted by other XDA-Developers users for us to read. These Benchmarks will be up to date (Hopefully), and the rom date will be on the benchmark submission. These spreadsheets will be STRICTLY Moderated so no bogus gets in there. Remember, some of these benchmarks may not be 100% accurate, because it may differ from device to device (but most likely won't). If you need help, just shoot me a PM.


Please Note: The numbers in this spreadsheet are the time in Milliseconds, and not a 'Score'. The lower the number, the better.
Spreadsheet Link (Results)


IF YOU SUBMIT A BENCHMARK, PLEASE "TEST" THE ROM WITHOUT ANY MODS!
Submitting a Benchmark
1. Flash a new ROM and do a hard reset.
2. Go into airplane mode
3. Disable switching screen off and turning the topaz off
4. DO NOT CHANGE ANY SETTINGS!
5. Install SPB Benchmark (Download HERE, and it IS freeware!)
6. Reboot (soft-reset)
7. Run SPB Benchmark : Select the "Main Test" option, and disable external software tests like word and IE or file explorer (whole category)
8. Submit results using this form.
Last edited by ThaDeanesta; 9th October 2010 at 07:43 PM.
The Following User Says Thank You to ThaDeanesta For This Useful Post: [ View ]
 
 
9th July 2010, 04:33 AM |#2  
Scabes24's Avatar
Recognized Themer
Flag Seattle, WA
Thanks Meter: 803
 
Donate to Me
More
Changes for 'Official' Benchmarks
Code:
Changelog

Changes 9/6/10 *Will be updating over the next few days*

•If anyone would like any stock/shipped ROMs tested let me know.

Changes 7/30/10

•Added Energy CHT - DinikGlass 23673/21911 (July 17) - 23121 (July 21)
•Added Energy Standard Sense 23673/21911 (July 17)
•Added Energy Titanium and Sense 2.1 23569/21911 July 14
•Added Energy Standard Reference 23673/21911 (July 17)
•Added Energy GTX 23121 (July 21)
•Added NhatHoa 4.0 23128 (July 18)
•Added Dutty's HG R15 23569/21905 (May 31)
•Added Dutty's Nu Era V1 Redemption 23128 (July 23)
•Added LouMeiYin Yin.09 23568/21905 (June 11)
•Added QBUS22 23569/21909 (July 12)
•Added Melissa v0.3A 21887 (July 8)
•Added MaryOne 23128/23569/21907 (July 15)
•Added Tess Prime V 21907 (June 30)
•Added NullPointer Evo v3.5 23569/21907 (June 28)
•Added Cheetah 4.3r 21901 (July 20)
•Added Cancer TIR-2 (July 2) and TIR-3 (July 10)
•Added Avatar IV 23128 (July 19) and 21911 (July 20 - in progress)
•Added TopazNihon 21911 (July 19) (in progress)
•Added Googy ROM v3.4 219xx (June 14) (w/ RAM Cab)
•Added Column - 'Average' of columns H-Z (NOT an overall score)
•Added Column - 'Sum' of columns H-Z (NOT an overall score)
•Added Column - 'Overall Score' - (still calculating)
•Added Column - 'Free RAM on first boot'
•Added Column - 'Free Storage on first boot'
Last edited by Scabes24; 7th September 2010 at 03:44 AM. Reason: For Scabes24
12th July 2010, 03:15 PM |#3  
joooe's Avatar
Senior Member
Thanks Meter: 105
 
More
And it would be great to have some kind of spreadsheet formula to calculate the rating of each test (average function, maybe?). Just for us to know which ROM is the fastest, overall.
This initiative is great!
17th July 2010, 06:31 AM |#4  
Senior Member
Thanks Meter: 59
 
More
Guys, a benchmark is a benchmark, you must try the ROM for a week and decide for yourself.
23rd July 2010, 04:04 AM |#5  
Scabes24's Avatar
Recognized Themer
Flag Seattle, WA
Thanks Meter: 803
 
Donate to Me
More
I think most people that aren't very interested in each specific benchmark test would much rather see some type of 'Overall score' for each ROM, so if someone knows of a simple formula or way to average the benchmark scores and place them on a 1-10ish scale (based on speed and possibly graphics) please PM me and I will add the stats.
24th July 2010, 02:56 AM |#6  
bnm7bnm's Avatar
Senior Member
Thanks Meter: 6
 
More
Arrow
Quote:
Originally Posted by Scabes24

I think most people that aren't very interested in each specific benchmark test would much rather see some type of 'Overall score' for each ROM, so if someone knows of a simple formula or way to average the benchmark scores and place them on a 1-10ish scale (based on speed and possibly graphics) please PM me and I will add the stats.

you can try make it like in previous benchmarking thread - relate to the fastest score of each column as 100% and others as "current"*100/fastest but for overall score need to know which of the colums really effects performance feeled by user to give em priority in overall score
24th July 2010, 05:09 AM |#7  
Scabes24's Avatar
Recognized Themer
Flag Seattle, WA
Thanks Meter: 803
 
Donate to Me
More
Quote:
Originally Posted by bnm7bnm

you can try make it like in previous benchmarking thread - relate to the fastest score of each column as 100% and others as "current"*100/fastest but for overall score need to know which of the colums really effects performance feeled by user to give em priority in overall score

Yeah I was trying to figure out how to determine what the 'best/max' score would be, going with the current highest/fastest score for each column (or even groups of columns) should work. As far as overall I was thinkin of seperating it into 2 or 3 scores. one for read/write/copy speed and maybe another for graphics.. but you are right, I would need to know which ones effect performance. Also I'm not quite sure how much of an affect the different (21xxx vs 23xxx) builds have on the stats.

I put the average and sum of the columns (except free RAM and storage) for a few energy ROMs, Dutty's Redemption, and Cheetah to see if it they were comparable and accurate to performance but havn't looked at them enough yet, most likely meaningless.
26th July 2010, 12:40 PM |#8  
mondilv's Avatar
Senior Member
Flag Riga
Thanks Meter: 13
 
More
Quote:
Originally Posted by ThaDeanesta

I will do this. First, however, we need benchmarks to make the score. I will handle everything =].

Not think, thats right way: all b/m must be at the same device, this the same condition (like disabled manila, sound, phone, phone canvas, the same time after flash/boot etc.).
Devices use different nand sometimes, with diff FS speed. So for compare roms, you need to use the same device.
It's creazy work.
Pietrucci did use it before. Ask him, how right to interpret results.
(the every test element have the own "weight" in the test. Need the special formula for calculate).
26th July 2010, 04:23 PM |#9  
ljankok's Avatar
Retired Recognized Developer
Flag Berkel en Rodenrijs
Thanks Meter: 6
 
Donate to Me
More
Quote:
Originally Posted by mondilv

Not think, thats right way: all b/m must be at the same device, this the same condition (like disabled manila, sound, phone, phone canvas, the same time after flash/boot etc.).
Devices use different nand sometimes, with diff FS speed. So for compare roms, you need to use the same device.
It's creazy work.
Pietrucci did use it before. Ask him, how right to interpret results.
(the every test element have the own "weight" in the test. Need the special formula for calculate).

Yes, this is true. There is a special formula for this. Piertrucci knows this, so it is a good idea to ask him instead of doing the same work twice.
5th August 2010, 03:33 PM |#10  
ThaDeanesta's Avatar
OP Retired Forum Moderator
Thanks Meter: 162
 
Donate to Me
More
i have scabes24 doing everything on the same device
2nd September 2010, 04:08 PM |#11  
LLKS's Avatar
Member
Thanks Meter: 1
 
More
Hello!

When you do a benchmark with SPB Benchmark this generates an XML file that can be uploaded on the website of SPB and compare with other devices. It also lets us do a custom table of devices and gives the% of one over the other in each test.

So if you still have all generated XML can be loaded into SPB under section Benchmark>Visualice Test Results(on the left column) and will be much easier to see which is better.

I'm Spanish my good english mercy to Google

Read More
Post Reply Subscribe to Thread
Previous Thread Next Thread
Thread Tools
Display Modes