[UPDATED January 29] Which is the best WM6 ROM? Benchmarks of 52 ROMs + links

[KarhU]

Ladies and Gentlemen!

After an extended period of flashing I am pleased to present you with the result of my effort :D

I took 52 ROMs (all except for one based on the official releases) and run them through benchmarks in SPB Benchmark 1.6.

ROM listing with links to their threads:
Black Satin: Light, Medium, Normal (http://forum.xda-developers.com/showthread.php?t=316755)
Schap's 3.30c Full (http://forum.xda-developers.com/showthread.php?t=312604)
Schap's 3.57a: Lite, Full (http://forum.xda-developers.com/showthread.php?t=318849)
vp3g's v3.0 Official (http://forum.xda-developers.com/showthread.php?t=317952)
Sleuth's Developer ROM v3.1 (http://forum.xda-developers.com/showthread.php?t=319171)
Cacahuate's v2.0 (http://forum.xda-developers.com/showthread.php?t=318338)
Elise .12 (http://forum.xda-developers.com/showthread.php?t=316889)
Pandora v3.0 beta2 (http://forum.xda-developers.com/showthread.php?t=316881)
Dopod official release ROM (http://forum.xda-developers.com/showthread.php?t=316723)
Dopod Lite ROM (http://forum.xda-developers.com/showthread.php?t=320010)
Faria's Super Clean v1 (3rd update) ROM (http://forum.xda-developers.com/showthread.php?t=309590)
Faria's HTC Hermes Customized v1 (http://forum.xda-developers.com/showthread.php?t=308987)
WM6 Samurai Final (http://forum.xda-developers.com/showthread.php?t=311854)
Schap's WM6 Pro 4.00 Full (http://forum.xda-developers.com/showthread.php?t=342168)
Schap's WM6 Pro 4.00 Lite (http://forum.xda-developers.com/showthread.php?t=342168)
vp3G's WM6 v3.62.3 AT&T ROM (http://forum.xda-developers.com/showthread.php?t=334946)
HyperCore Lite 3.62SE (http://forum.xda-developers.com/showthread.php?t=339529)
Shamanix`s Grass v1.1 (http://forum.xda-developers.com/showthread.php?t=342246)
Starbase64 3.62/7 WWE Rom (http://forum.xda-developers.com/showthread.php?t=328468)
TNT.1921 Series Professional (http://forum.xda-developers.com/showthread.php?t=336908)
Pandora Rom Halo 3 Edition (http://forum.xda-developers.com/showthread.php?t=341529)
Schap's WM6 Pro 4.01 Full (http://forum.xda-developers.com/showthread.php?t=343942)
CUSTEL Im Back No PTT - 6MB Page Pool (http://forum.xda-developers.com/showthread.php?t=344151)
Pandora Naked 2 ROM (http://forum.xda-developers.com/showthread.php?t=348996)
S64 3.62/HTC WWE Rom (http://forum.xda-developers.com/showthread.php?t=347081)
HCLite 3.62 Final (http://forum.xda-developers.com/showthread.php?t=339529)
Austinsnyc's AT&T 3.62 Clean ROM (http://forum.xda-developers.com/showthread.php?t=335257)
Schap's WM6 Pro Lite 4.20 ROM (http://forum.xda-developers.com/showthread.php?t=347806)
K's Rom v1.21 and 1.30a Lite (http://forum.xda-developers.com/showthread.php?t=350695)
Pandora Naked 5 ROM (http://forum.xda-developers.com/showthread.php?t=351274)
Pandora Naked 6 ROM (http://forum.xda-developers.com/showthread.php?t=351274)
BlackDragon WM6 ROM (http://forum.xda-developers.com/showthread.php?t=352031)
Pays WM6 WWE v1.22 and 1.23 Big Storage ROMs (http://forum.xda-developers.com/showthread.php?t=349338)
Schap's WM6 Pro 4.22 Full and Lite ROMs (http://forum.xda-developers.com/showthread.php?t=350690)
Shamanix`s Grass v2.2 (http://forum.xda-developers.com/showthread.php?t=351973)
ROM v3.62.07.08 CHO HTC TyTN PDA VIET (http://forum.xda-developers.com/showthread.php?t=352443)
mERCURY v2.1 Base AT&T 3.62 (http://forum.xda-developers.com/showthread.php?t=351127)
TNT.SE "Special Edition" Official and Professional ROMs (http://forum.xda-developers.com/showthread.php?t=352914)
Telstra imate RUU_Hermes_DL_UK_3.62.305.5_6275_1.54.30.10_108 Official ROM (http://forum.xda-developers.com/showthread.php?t=348436)
Pays WM6 WWE v1.30 Big Storage Lite ROM (http://forum.xda-developers.com/showthread.php?t=349338)
K's Rom v1.31 Full (http://forum.xda-developers.com/showthread.php?t=355722)
CRCs PHKv11 Release 3 Complete ROM (http://forum.xda-developers.com/showthread.php?t=355439)
BlackDragon WM6 ROM Updated (http://forum.xda-developers.com/showthread.php?t=352031)
Schap's WM6 Pro 4.30 Full ROM (http://forum.xda-developers.com/showthread.php?t=355774)
PDA Viet WM 6.1 19716.1.1.0 PV (http://forum.xda-developers.com/showpost.php?p=1860583&postcount=45)

ROMs were tested on my HERM 100 (Orange M3100) with 1 GB Kingston MicroSD. Radio version is 1.54.30.0 and during the testing phone part of the device was off (flight mode). Device was attached to a charger. All ROMs were tested immediately after the flash. SPB benchmark was installed into the main memory. After that I soft resetted the device, and stopped all running programs using settings>>memory. All ROMs were tested with their original settings - i.e. no today plug-ins were disabled. Results are device specific - some tests use the MicroSD card - so if you do the tests your results will probably differ.

Findings:

1) Cooked ROMs do make a difference: increased performance is obvious, additional tweaks and enhanced functionality of cooked ROMs make official ROMS quite useless.
2) Lite ROMs are more snappy and as they get more bloated their speed diminishes. Install only programs you need.
3) Black ROMs seem to be the fastest so far.
4) Faria is really a great cook (his ROM proved worthy it's almost legendary status :D )
5) There is more than just benchmark results that makes a ROM great. Read their respective threads about their advantages and disadvantages.
6) Cooked ROMs have made a noticeable progress since january. Older benchmark results can be found here (http://forum.xda-developers.com/attachment.php?attachmentid=35518&d=1174067937) and here (http://forum.xda-developers.com/showthread.php?t=303849). The performace and usability gain is quite impressive.
7) Older ROMs scored lower than the newer ones. Keeping your device updated is good for it's health ;)
8) Actual differences between latest ROMs are quite small. Choose according to your taste.
9) Out of new (3.62.0-3) ROMs Schap's ROMs are outstanding. He is like a good (French) wine - gets better with age, The only ROMs with similar results were Black ROMs which are out of development now
10) Newest ROMs are very fast - including official one from Telstra - it seems that the video playback problem older official ROMs had is gone and official ROM now can compete with cooked ROMs in terms of speed
11) kin0kin has made a straight run for the elite cookers with his K's Rom v1.21. It is one of the fastest available.
12) Shamanix's Grass 2.2 is officially the second fastest ROM I have tested (R.I.P. Black ROMs)
13) New Pandora 6 Naked ROM also broke records in some categories...

Notes:

- I will not run tests on following ROMs:
- Sajakati's HypercoreLite froze while benchmarking (only HR solved the situation) and though I run less tests than usually it did not finish properly - benchmarking not recommended due to extreme instability while doing so) However new Black Dragon Sajakati's ROM was running very smoothly and I had absolutely no problems benchmarking it.
- Please note that benchmarking is a stress testing, so eveni if the benchmarking does not work on a ROM it does not mean it will brick your device someday.
- It seems that NETCF version has no impact on overall device performance (tested the ROM with either NETCF 2.0 or 3.5) compared to the ROM without NETCF or between the NETCF versions. That however might not be true for using programs that use these frameworks.

To do:
- Include more ROMs
- Include tabular comparison of ROMs features
- Include benchmarks from other programs
- Try to persuade mods to do this thread sticky....
- Any wishes, comments?

Updates:
29.01,2008 - Added PDA Viet's WM 6.1 ROM
02.01.2008 - Added 5 New ROM Benchmarks
22.12.2007 - Added K's Rom v1.30a Lite benchmark results
21.12.2007 - Added PaYs 1.23 ROM benchmark results
19.12.2007 - Added 13 New ROM benchmarks
07.12.2007 - Added 5 New ROM Benchmarks, added excellent ROM comparison and ranking by Niteline (kudos buddy!) in the zip package, FROZE THE PANES IN .XLS - REJOICE!
16.11.2007 - Added 2 New ROM Benchmarks - Shap's 4.01 Full and CUSTEL Im Back No PTT - 6MB Page Pool
10.11.2007 - Added 8 new ROM benchmarks, added RAM and ROM after flash stats, revived the thread :p
31.07.2007 - Added links to the ROM threads, added two ROM benchmarks, added results in xls (MS Office 2003) , xml and mht formats - everybody can format the results to his/her liking now.
31.07.2007 - Added Schap's 3.30c Full ROM benchmark

Thank yous:
All of the chefs that make this flashing fiesta possible...
THANK YOU ALL FOR YOUR SUPPORT

Please help me to make the benchmarks better. More info here (http://forum.xda-developers.com/showthread.php?t=353347).

[KarhU]

Performance tweaks that will improve responsiveness of your ROM no matter which you choose (Kudos Cyphol):

1. Improve performance/response time:
- Start > Settings > Phone => Keypad = Off
- Start > Settings > Sounds & Notifications; Under Sound Tab => Disable/Untick Events (warings, system events) - this tweak will BRUTALLY increase the responsivenes of Start menu and Quick Menu 2.7 (because system will no more have to search for sound files in the ROM everytime you tap the screen)

2. Your Windows Mobile device is just like your Windows PC. So, soft reset it before going to sleep (or whenever appropriate) once a day to make it runs good at all time (because of the memory leak when you close the program not all the memory is restored - sof reset restores RAM lost due to Memory leak).

3. Make sure you install the today plugin apps such as Resco Today Plugin etc. on the internal/device memory. Some today plugin can be installed on the storage card but it'll slow down your system performance a bit.

4. Do not install too many apps on the internal/device memory. Leave as much free space as you can on it. Check whether it can be install on the storage card or not. Normally, don't let it become as low as 8MB free but 10MB is enough already.

5. Know what you're doing such as tampering your system registry. Make sure it can be done and you're doing it the right way. If not, then leave it as is.

6. Never hide the WirelessManager Tray Icon at the bottom.

Don't believe me?

Try it and feel the difference.

This is the key:

[HKEY_LOCAL_MACHINE\Services\WirelessMgr]
"Keep"=dword:00000000 ;<--- (dword:00000000 to hide and dword:00000001 to show)

7. Never let your device go to sleep by itself. Do it manually by pressing the power button everytime when you've finished using your device.

8. Make sure you shutdown the 'Activesync' completely after activesyncing your device with your PC. You can use task manager (X-Button), Magic Button etc or you can use memory applet in setting > System > memory; Running Programs

[raymondu999]

YES!!! Finally some solid numbers to depend upon for comparison! Btw, how long did it take you to do this frenzy of HR, flash, benchmark, HR flash, benchmark?

[KarhU]

YES!!! Finally some solid numbers to depend upon for comparison! Btw, how long did it take you to do this frenzy of HR, flash, benchmark, HR flash, benchmark?

Including waiting on free slot on rapidshare it was cca one full day ;).

[bapski]

would a day of benchmarking suffice for conclusive results? although this would help would be FLASHERS to decide which ROM they can try first instead of starting another "WHICH IS THE BEST ROM" thread.

[raymondu999]

would a day of benchmarking suffice for conclusive results?

The thing is, SpB Benchmark is a program, as in lines of code. Hence, these numbers might not mean much for stability, but for speed, these numbers tell no lies.

[KarhU]

would a day of benchmarking suffice for conclusive results?

As I have wrote before - benchmarks are just numbers. Actual experiences vary. Decisions about which ROM to use should not be solely based on benchmarks. Important factors are also support and or newer versions in the making, feel of the ROM (they do not look all the same) and for example bonus features also differ. Choose according to your needs.

[typester]

some of the top scores were not in blue?? and ones that are tied should be in another color like yellow to show equality. i know you have spent lots of time on this and i don't mean to harp on you, so these are only suggestions.

thanks for the hard work!! this was a great idea. i will look forward to you adding new ROM's.

[gerald_s]

Guess I missed something. Where did you find Farias super clean 3 rom? I only can find super clean v1.

BTW: Great idea, I hope somebody will add this list to the wiki

[Bensat]

Great work!!!

I've asked myself this question over and over again........... Hahaha but now I see these results I can only say to the COOKIES that they've done a GREAT job for all of us.
That is what matters, our guys even listen and they will sometimes ad some special flavors into a ROM. This development and research goes so fast I even wonder if the developers of HTC come here to see what is going on over here!

Keep up the good work!

[peter7]

Hi there - this really helps to decide which ROM one should take - but what is the cooked rom - I understood that I can no I have to choose whichsoftware I will install - and my understanding is that this takes even more time for the whole process - I liked it for example withe VP3 to simply follow the guide line in the forum and it was done ! Or is the installation provess with the cooked rom also simple ?

[speckledpig]

Cooked as opposed to straight from the manufacturer ("official").

[ratcom]

This is one of the most usefull post's Ive seen in a long time, thanks for your hard work :) Oh and its crazy but "Faria's Super Clean 3 ROM" is still in the running after all this time! I think that says alot about Faria's work ;)

[MaRaHoX]

Some small tests also here: http://www.ce4you.cz/forums/view_topic.asp?t=23151&fi=0 - this is our czech internal community testing:).
About 30 latest ROMs.

In czech language, but main values are universal.

[neil@domino.org]

intereting and very useful. Whats more interesting is that I'm using the DOPOD rom as it seems to be the most stable for me. No BSOD or having to reboot all the time to get data connection working again.

[KarhU]

Guess I missed something. Where did you find Farias super clean 3 rom? I only can find super clean v1.

BTW: Great idea, I hope somebody will add this list to the wiki

Of course it is Faria's v1, number taken from package name (third update).

@ neil@domino.org

That is what benchmarks can not show. And stability really is important "feature". I personally have never had SOD with any of the ROMs.

[deadeyed]

Great Work!! Thank you for quantify this. I've been using Satin medium for over a week now, very stable & not a single lockup with this ROM so far. And it is good to know I'm using one of the fastest around. Thanks again.

[Lordsmiff]

Well done - this is excellent :)

Using this, with my thread here (http://forum.xda-developers.com/showthread.php?t=317024), anyone should be able to decide which ROM they are going to run on their phone, without relying on the eye-candy screenshots that don't really tell you anything :)

Kudos to you, dude :)

Smiffy.

[Chiefcooker]

i think such a test says nothing. all results are nearly the same, and 119 frames instead of 111, i ont think that i can see that BIG impovement.
:p

mfg chief

[WDawn]

Very usefull info thnx!

[crazyC]

Nice to see that statistics, but I wonder how much this reflects the real world feel of the ROMs. For example, look back at the Satin vs Schaps thread. The general feel seemed to be on this forum (until these results came out) that Schaps actually felt faster than Satin.

Whilst I have always been a big Black fan, and thought Majik and Satin were great, I have tried Schaps, and yes, it does actually seem (maybe just my perception) that little bit quicker.

[esseff]

No offence man, but that is the crapest layout of information I have ever seen. After all the work you put into doing the tests it seems a shame you made no effort in the presentation of the results.

[db77]

Nice to see that statistics, but I wonder how much this reflects the real world feel of the ROMs. For example, look back at the Satin vs Schaps thread. The general feel seemed to be on this forum (until these results came out) that Schaps actually felt faster than Satin.

Whilst I have always been a big Black fan, and thought Majik and Satin were great, I have tried Schaps, and yes, it does actually seem (maybe just my perception) that little bit quicker.
I think also that. (Je pense la même chose).
It must see on use (Il faut voir à l'usage).
All tests are made without soft installed (tous les tests sont fait sans logiciel installé)

[Quist]

I'm testing Schapps now after having Satin for about a week. In some respects it "feels" faster, but there's some things I definitely prefer with Satin, like the file explorer and the streaming media skin.

I also wonder if Schapps is better for those of us in 3G networks that want a video dialer. After installing Satin one of the first things we have to do is install the other phone dialer for the same functionality as is already in Schapps. The tweaks or adjustments needed to get the right mix might be slowing Schapps down less than they do Satin.

Still, they're both excellent ROMs and a quantum leap over what we had just 1-2 months ago. Keep up the good work folks!

[KarhU]

I'm testing Schapps now after having Satin for about a week. In some respects it "feels" faster, but there's some things I definitely prefer with Satin, like the file explorer and the streaming media skin.

I also wonder if Schapps is better for those of us in 3G networks that want a video dialer. After installing Satin one of the first things we have to do is install the other phone dialer for the same functionality as is already in Schapps. The tweaks or adjustments needed to get the right mix might be slowing Schapps down less than they do Satin.

Still, they're both excellent ROMs and a quantum leap over what we had just 1-2 months ago. Keep up the good work folks!

I think some tweaks may have not be visible in improving GUI performance. These however can be seen in benchmarks. I had the same feel about Schap's and Satin. With latest generation of ROMs the results are pretty close. Nice things about ROMs with the same base is that you can see who of the cooks has the skills.

[chunkymonkey]

I tink this is an excellent bit of research.

As you say, it doesn't give a 'best ROM' result but is a guide only. The main guide is one's own experience with one's own setup.

As to layout, F that. It's perfectly legible and readable.

[kipe]

useful posts.. makes u want to try em all back once again... :D ..it certainly got me... on faria's super clean now... man, suddenly the itch to flash comes back.. :D

[alancstan]

Very informative benchmark review.... Thanks!

[zim2dive]

Still, they're both excellent ROMs and a quantum leap over what we had just 1-2 months ago. Keep up the good work folks!

Aye.. what a wonderful problem we have.. so many good ROMs to pick from! :)

My compliments to the many chefs!

Mike

[edhaas]

Nice bit of research. I was pleased my rom was in the pack. I would like to caution folks that "your mileage may vary". I have done some work on my own with these benchmarks, and have found that some of them can be increased very significantly with tweaking, (for example the sd card reported speeds), but the result does not reflect in the "real world", and in fact can cause a lag in response. Most of these benchmarks are "machine" tests, vs OS response tests. For example the cpu benchmarks should all be the same, since the code is internal to the program itself, and the cpu is always the same. (Although if there were lots of interrupts running in the background it may slow the reported speed.)
As others have noted, the "snappiness" of the rom will not necessarily be reflected in the benchmark, and even that can be a trade off. For example, reducing the page pool will cause an increase in responsiveness, but multitasking will suffer.
So, I would agree with those who suggest that the differences you are seeing in the benchmarks are small, and may not be reflective of "real world" experience. I think we still need to try them ourselves before making a judgment.

Best wishes,

[Voldemort]

i always went for black roms till satin (which i didnt really like as the previous ones) so i tried schaps and i must admit it s simply better. the millisecond difference isnt a big deal, the bunch of extra features is :p

on a side note: black's explorer is way better than the standard ones (used on schap's too) but resco explorer is once again a step forward - so far it s the only program that can browse shared folders on a wifi network i ve found :cool:

[Meteo]

mmmm. delicious numbers... *munch munch*

[washi]

i can confirm that schaps rom feels faster than satin.
used black roms until satin. didnt like it as the previous ones so i tried out schaps and im happy with it.
even more when you consider that there will be no more black roms and i think not much updates for the black roms since jj will buy a diffrent phone.

[batt]

Gup, gup, gup...

[ckraus000]

what about free memory after installation?? would be also nice to know^^

[thefunkygibbon]

any updates on this?

[KarhU]

any updates on this?

Today I will hopefully add three more ROMs - Grass 1.1, Schaps 4.0 Lite and vp3Gs 3.62.3. If wondering which one is fastest - Schaps 4.0 Lite is :D

[gunigugu]

Today I will hopefully add three more ROMs - Grass 1.1, Schaps 4.0 Lite and vp3Gs 3.62.3. If wondering which one is fastest - Schaps 4.0 Lite is :D

Sweet.. can't wait.. first time i stumbled on this thread..

actually in the process about to flash my phone.. only really do it every few months as i'm quite dependent on it and don't get a chance to back-up everything i need... and really found this thread useful. Lots of good info in there.. Although to be honest I usually go with the "full" rom for conveniences as it has alot of things in there i normally use...

Would it be possible to give schaps v4. full rom a quick test too as it's probably the most suitable for me and I imagine a few others too...

[Shamanix]

Today I will hopefully add three more ROMs

Great... thanks man!

I am looking forward to see some numbers..

[WDawn]

I think Hypercore LiteSE is a good competitor also. I'm amazed with the speed, even loaded with programs. It feels faster then Schaps and Black Satin.

[aaronsmckee]

Could you put starbase64 3.62v7 in there to, I know it probably won't be the fastest (lots and lots of built in apps) but it would be great to see what the actually speed differential is vs some of the less feature rich roms out there.

[mlalahoi]

First, great bit of work and you have big brass ones to flash your phone 16 times in one day with no fear of bricking, kudos.
Regarding some of the more negative posts, I agree that we need to do some more practical testing of these ROM's. I suggest that a few of us get together and do the following:
1.agree on a set of "most common" applications that most of the people around here are using, and all install them in the same way

2. agree on a set of useage guidlines, i.e. push mail, VoIP calling, syncing schedule, etc. so we are all using the ROM's in appx the same way

3. agree on what parameters to test. Boot times, load new SMS with XX programs running, etc. track number of soft resets, problems, etc.


Then we take the time to flash our phones with all the ROM's running the same software being used in roughly the same way for a definite period of time, say one week. Then we can compile the results and get some more "real world" test data.
We could divide the ROM's between say 15 of us willing to dedicate our phones, and test each ROM for one week. If we have 3 people test each ROM for one week then test another two the following two weeks we could gather some pretty solid data on all 16 ROM's in under a month.
What do you think? I can vouch for two Hermes 100's to commit for a month of testing. Of course we would need to put some real thought into the benchmarks and apps to use in the process, but it would be fun and I think very helpful. Anyone interested??

[lino8419]

This is an awesome list!!! thanks a bunch!!

[galaxys]

Could you include TNT V2.0 Professional in there? It is one of the fastest with lots of apps built in!

[pozzwali]

Thank you very much for this!

:) :)

[ermenzegna]

great job !!!

[KarhU]

Major update: 8 new ROMs added. Please check the first post.

[Pumpiron579]

Thanks, Very informative. I don't see the official AT&T rom on the list. I would like to add that I ran Satin Normal with radio 1.50.00.00 and added the Kaiser Home screen and my usual stuff since Satin was released and it was fast. Fantastic picking up the 3G network and push was working great.I Reset Satin probably once every other day. I decided to try the official AT&T rom since I am on the AT&T network and I added Live, plus all the same programs I had on Satin and I found the AT&T rom is as fast and with radio 1.54.07.00, I pickup 3G better then I did with Satin as radio 1.54.07.00 sucked with Satin. Just my 2 cents worth

[WDawn]

Thnx! This was probably hell of job! Thank you for including Hypercore, it confirms that it is indeed pretty fast. And it pinpoints that there aint much free RAM to play with.

[alancstan]

Please test along Edhaas T-Lite and D-Lite Rom. Superb and stable ROM with minimal problem encountered!

rgds

[diezoo]

Thank you KarhU for this amazing work!! :)

[funky_Saggi]

You, my man, are a LEGEND for posting these numbers!

[venelar]

nevermind!

[EightEight]

Thank you KarhU for this amazing work, now I have numbers to show my firends that the Dopod Official WM6 ROM is trash, so they will move to a Cook Rom. :)

[DaVince]

holy sh*t :eek:

This is hard work. I bow down to your perseverance, dedication, commitment.. whatever.. I just can't find the right words to describe your hard work. Thank you.

[KarhU]

Hello everyone;

I added two new ROMs to the benchmarks, enjoy the number-crunching!

[CUSTEL]

Hello everyone;

I added two new ROMs to the benchmarks, enjoy the number-crunching!
I'm pleased with the results, thanks for adding mine to your test :)

[PAPPL]

Hi karHu,
the Hypercore HC3.62 ROMs (70 & 80MB) are now updated by sakajati in first post.
They have another cache settings now and other modification.
Could you please update Hypercore 3.62 Full ROM 80MB (not SE) in possible future update too. I think this ROM is a burner now. :)

[irockash]

Question, why manually put it to sleep as opposed to it being timed?

PS.. to shut the presentation critcs up (since you ignored them, I doubt you care), freeze the panes so the tests stay in the same place on the spreadsheets.

[KarhU]

Question, why manually put it to sleep as opposed to it being timed?

PS.. to shut the presentation critcs up (since you ignored them, I doubt you care), freeze the panes so the tests stay in the same place on the spreadsheets.

About your first point: I do not know :D I took these from Cyphol, he is the right person to ask - the best from these tweaks is the number one, the others are less important or obvious. I am not sure about importance of that particular tweak/advice too.

To the second point: You are right, I will change it with next release - I got too obsessed with adding new ROMs that it become really messy. Thanks for comments.

[hjlow]

mm just wondering, how does the official HTC ROM compare to these?

[irockash]

To the second point: You are right, I will change it with next release - I got too obsessed with adding new ROMs that it become really messy. Thanks for comments.

I totally understand, its an easy thing to do manually so I didn't mind... I just didn't see the point in people posting in the thread about the layout of the thing. If they're that obsessed with it, they can fix it themselves.

[larsuck]

To do:
- Include more ROMs
- Include tabular comparison of ROMs features
- Include benchmarks from other programs (need advice)
- Try to persuade mods to do this thread sticky....
- Any wishes, comments?



How about some benchmark connected to ROM stability? For example, you could count the unresolved problems in each thread and divide by the number of total downloads... It wouldn't be scientific really, but it might lead to some interesting discoveries about what makes for a stable ROM and what doesn't.

[niteline]

Thank you so much for your dedicated work, KarhU!

Great job! :cool:

I was interested in how the ROMs rank relative to each other.
So I added a bit to your XL file... I hope that is OK with you.

My file has a whole new section with relative numbers in addition to the absolute numbers already present in the original file.

Also, i calculate an overall relative benchmark score, which is then used to rank all the MODs.

Have fun playing around with it. :)

[KarhU]

Thank you so much for your dedicated work, KarhU!

Great job! :cool:

I was interested in how the ROMs rank relative to each other.
So I added a bit to your XL file... I hope that is OK with you.

My file has a whole new section with relative numbers in addition to the absolute numbers already present in the original file.

Also, i calculate an overall relative benchmark score, which is then used to rank all the MODs.

Have fun playing around with it. :)

Thanks for your work (late but better than never ;) )

I have added five new ROM benchmarks to the first post, froze the panes in the xls and also added your excellent ROM comparison to the benchmarks zip.

Thank you!

[Chatty]

Karhu, you've been doing great so far. But you really should consider to use niteline's modifications on the sheet.

most important:
cell formats - this way you can use the numbers for charts

Even better: transpose the table and put the scale into the header. This way columns can be even smaller and more info would fit onto a page.

Next thing: use conditional formats for best/lowest score thus you don't have to look for that each time you add a ROM.

If you need any help, don't hesitate and ask.

[thanosc]

Wow.. great thread and files....
It would be great if we knew on which devices these roms run well or not at all...
Arty out...

[SaturnusDJ]

mm just wondering, how does the official HTC ROM compare to these?
I agree with that.

[kazuya_gonzalez]

waiting for benchmark updates :p

[KarhU]

waiting for benchmark updates :p

Benchmarks will be updated this evening. Stay tuned...

[its right]

great work bro

ofcourse this is very useful for most of users who can't test every rom

really hard thing to update your device with every rom then install the app then get the results then compare with this orginized table :)

keep it up.

[KarhU]

Added 13 new ROM benchmarks. Every major ROM should be included now.

[spastico]

Have you noticed any speed or stability issues w/ Roms where the Extended Rom has been merged versus the ones where the Extended Rom is left intact?

Just curious.

Thanks,
Berd

[KarhU]

Have you noticed any speed or stability issues w/ Roms where the Extended Rom has been merged versus the ones where the Extended Rom is left intact?

Just curious.

Thanks,
Berd

It depends - some are Ok while others are not. With your ROM and older sajakati's one I had freezes during testing that led to bricking on the soft reset (hung at Windows screen, HR to fix). That only happens with ROMs with merged extROM. It is not a widespread issue though. Speed of booting (even after HR) of these ROMs is fantastic - but I can not tell you otherwise because I do not use these ROMs daily - I prefer "light" to "ultralight".

[MaRaHoX]

Oh, must take hat off to you and your sedulity! This is really amazing work! Looks like, Pandora is the best;) and Grass 2.2 is really very responsive. For me is for now best TNT, but this is my own personal view;)

[PAPPL]

Thx for the update!
Seems like every cooked ROM is a winner, the new ROMs are very close together in speed.
6MB Pagepool seems to be waste, 4MB is the same speed (plus 2MB more free RAM).

[spastico]

It depends - some are Ok while others are not. With your ROM and older sajakati's one I had freezes during testing that led to bricking on the soft reset (hung at Windows screen, HR to fix). That only happens with ROMs with merged extROM. It is not a widespread issue though. Speed of booting (even after HR) of these ROMs is fantastic - but I can not tell you otherwise because I do not use these ROMs daily - I prefer "light" to "ultralight".

Interesting.
Thanks for testing mine also.
I guess as long as it works for me...
I just hate to see that 10+ mb's just sitting there.
I know there's other ways to use the ExtRom, I just figured merging the partitions made the most sense. oh well. :rolleyes:
But as I said in my thread: "I ran Schaps 3.54c w/o Ext Rom for almost 2 months and never had any issues.
It ran just as good as the one with the Ext Rom.
So I did the same for the 4.01's"

Thanks Again,
Berd

[PaY87]

Great! Thank you very much for testing the roms! :o

[KarhU]

Interesting.
Thanks for testing mine also.
I guess as long as it works for me...
I just hate to see that 10+ mb's just sitting there.
I know there's other ways to use the ExtRom, I just figured merging the partitions made the most sense. oh well. :rolleyes:
But as I said in my thread: "I ran Schaps 3.54c w/o Ext Rom for almost 2 months and never had any issues.
It ran just as good as the one with the Ext Rom.
So I did the same for the 4.01's"

Thanks Again,
Berd

Benchmark is not the daily use - it is just testing according to some pattern - and this one collides with SuperStorage ROMs. I have used superstorage before it was fixed and had to HR several times. I have no need to use another 10 MB because I put everything I can on the SD card (because I flash very often - I can just copy .lnks back to the start menu and go - fast restore). If your ROM works for you then it is perfectly Ok to use. I do not believe that there are people willing to change their ROM just because of benchmark results.

[KarhU]

Thx for the update!
Seems like every cooked ROM is a winner, the new ROMs are very close together in speed.
6MB Pagepool seems to be waste, 4MB is the same speed (plus 2MB more free RAM).

6MB Pagepool may be a waste for benchmarking, but some programs just work better with it (Matt keeps 6MB pagepool for a reason - I think RAM intensive apps like it, also 6MB pagepool os the default on official ROMs so it should be the best tradeoff between speed and stability).

[RanDomIze]

Hi.
Thank you very much for testing the roms, including mine.

Regards

[KarhU]

Added one new benchmark - PaY's 1.23 Big Storage ROM.

[WDawn]

Thnx for your work! Looking at the results and the excellent reactions, i'm gonna flash K's ROM tonight

[kin0kin]

Thnx for your work! Looking at the results and the excellent reactions, i'm gonna flash K's ROM tonight

hold your horses...unless you want to flash 1.30a lite, as I will be delivering Christmas present shortly! :D

[WDawn]

hold your horses...unless you want to flash 1.30a lite, as I will be delivering Christmas present shortly! :D

Dude you're too good for this world. And thank you for mentioning as i would hate to flash tonight and see that you release a better version this weekend :)

[pietrucci]

Thnx for your work! Looking at the results and the excellent reactions, i'm gonna flash K's ROM tonightK's is super (1.21 and 1.30, didn't try 1.30a yet) and according to my SPB Benchmark tests it's even the fastest ROM around. Don't know if KarhU finds the same results...

[pietrucci]

hi KarhU,

I notice a slight difference between your in testresults and my own.
Most of the time my results are a littlebit slower than yours.
When I repeat a test, just to make sure, the result are almost exactly the same as the first time. Any idea about our differences?

I use the most recent SPB Benchmark on a TyTn. I never make changes in settings before testing.

[KarhU]

hi KarhU,

I notice a slight difference between your in testresults and my own.
Most of the time my results are a littlebit slower than yours.
When I repeat a test, just to make sure, the result are almost exactly the same as the first time. Any idea about our differences?

I use the most recent SPB Benchmark on a TyTn. I never make changes in settings before testing.

I always use charger while testing (I do not want to kill the battery right away) - this adds around 40 points to the overall benchmark results compared to batt only tests. I usually test the ROMs more than once because results vary and publish the best ones. I do not really know what cause the variability of the results, but I try to keep a methodology so the results are comparable. Maybe there are slight differences between our devices too (different memory chips batch and so on).

Regards,

KarhU

[pietrucci]

I always use charger while testing (I do not want to kill the battery right away) - this adds around 40 points to the overall benchmark results compared to batt only tests. I usually test the ROMs more than once because results vary and publish the best ones. I do not really know what cause the variability of the results, but I try to keep a methodology so the results are comparable. Maybe there are slight differences between our devices too (different memory chips batch and so on).

Regards,

KarhUOk, sounds reasonable. I also test with power charger.

[Pumpiron579]

Great job, this is really usefull in deciding what rom to flash. Just wondering if you are going to ever test the new AT&T rom. AT&T RUU_Hermes_CINGULAR_WWE_3.62.502.3_6275_1.54.07.00 _108_UOOS_SVN05_Ship

[KarhU]

Added K's Rom v1.30a Lite benchmark. Please help me to improve the benchmarks - read about my benchmarking app idea here (http://forum.xda-developers.com/showthread.php?t=353347).

[PaY87]

Added K's Rom v1.30a Lite benchmark. Please help me to improve the benchmarks - read about my benchmarking app idea here (http://forum.xda-developers.com/showthread.php?t=353347).

Donate to the guy he is doing a great hard job for us! ;)

[burkay]

Hey all,

I did some statistics on KarhU's numbers and obtained a simpler ranking for the existing ROMs. The following list is only a simplified version of KarhU's statistics, so that if you do not want to get lost in numbers :) you can choose a rom from the following list:

Shamanix`s Grass v2.2 100.00
Pandora Naked ROM 6/3G Dialer 99.84
Schap's WM6 Pro 4.22 Lite 98.21
Schap's WM6 Pro 4.20 Lite ROM 96.48
mERCURY v2.1 Base AT&T 3.62 96.36
Pandora Naked 5/ 4MB PagePool 94.24
Schap's WM6 Pro 4.00 Lite 94.19
Pandora Naked ROM 5 94.13
Schap's 4.00 Full 93.94
Austinsnyc's AT&T 3.62 Clean ROM 93.85
K's Rom v1.30a 93.28
K's Rom v1.21/6mpp 92.95
Black Satin Light 92.38
Shamanix`s Grass v1.1 4m PP 91.42
Black Satin Medium 90.52
BlackDragon 89.98
Telstra Imate 3.62.305.5 Official R 89.50
ROM v3.62.07.08 CHO PDA VIET 88.25
TNT.1921 Professional 87.05
HyperCore Lite 3.62SE 86.70
TNT.SE "Special Edition" Profession 86.63
Pays WM6 WWE v1.23 Big Storage ROM 86.04
Schap's WM6 Pro 4.22 Full 85.95
Pandora v3.0 Beta 3 85.43
Schap's 4.01 Full 85.22
Pandora Naked ROM 2 with NETCF 3.5 84.94
Pandora Naked ROM 2 84.68
Pandora Naked ROM 2 with NETCF 2.0 83.14
Schaps 3.57a Lite 82.07
Dopod Lite 1.0 81.97
TNT.SE "Special Edition" Official 81.20
Black Satin Normal 81.15
vp3g WM6 v3.0 Official 80.45
Faria's Super Clean3 ROM * 79.80
Pays WM6 WWE v1.22 Big Storage ROM 79.03
S64 3.62/HTC WWE Rom 78.74
Cacahuate's 2.0 WM6 78.64
Elise .12 78.17
vp3G's 3.62.3 AT&T WM6 ROM 77.25
CUSTEL Im Back No PTT - 6MB PP 74.80
Sleuth's v3.1 Developer ROM 73.54
Pandora Halo 3 3.62 69.37
Faria's Custom V1 67.91
Schaps 3.57a Full 60.42
Last Samurai 58.37
Schap's 3.30c Full 55.56
S64 3.62/7 WWE Rom 54.98
Dopod Official WM6 ROM 0.00

[kipe]

awesome burkay.. that's more like it.. this is far more better than the US Billboard Top 20.. good job! :) kudos to u & of course da man karhu...

[KarhU]

Added 5 new ROMs Benchmark results. This should cover all of new Hermes ROMs available.

[spastico]

...clip...
- I will not run tests on following ROMs:
- Sajakati's HypercoreLite and Berdinkeldickle's Schap's Stripped 4.01 Full with Bepe's Big Storage froze while benchmarking (only HR solved the situation) and though I run less tests than usually it did not finish properly - benchmarking not recommended due to extreme instability while doing so)
...clip...


I was hoping you would have updated the above comments by now.

After you first mentioned the issue you had with my rom, I immediately went back and fixed it (matts suggested fix).
I have ran three sbp tests on it, and have had no issues.
(freezes, crashes, or otherwise)

I hope you may find time to give it another test.
I will be glad to send a donation for your time.

Thanks,
Berd


http://forum.xda-developers.com/showpost.php?p=1752540&postcount=26

[spastico]

Nevermind,

After doing some more research, it seems that some devices just crash easier than others.

So I guess it's better you don't even risk it.

I will go and delete the roms.
I would hate to see someone ruin their device, just to try and gain a measly 10mb. :o

Thanks Again.

[KarhU]

Nevermind,

After doing some more research, it seems that some devices just crash easier than others.

So I guess it's better you don't even risk it.

I will go and delete the roms.
I would hate to see someone ruin their device, just to try and gain a measly 10mb. :o

Thanks Again.

Hello Berd!

I will erase the sentence as it is nonsensical now. It should not crash on other devices if it works on one - that would indicate hardware failure/change (i.e. bad blocks in NAND). Maybe I should date the ROMs so they can see how old the version of the ROM was. Nobody can RUIN a device by running your ROM - Benchmarking is a stress testing - something normal users will never do - and as you said it worked flawlessly for you ever since you used your ROM. I do not wish any ROMs to be pulled just because of this thread/bencmark results. I think I should write more about the Benchmark limitations.

Regards,

KarhU

[laules]

I've intalled the 4.30 in French and started to test it : GREAT JOB Schaps !

I've noticed a small issue regarding the Keyboards. I've got a French one (azerty) and the 'a' and 'q' keys are swapped with the blue associated keys as well.

Is anybody have a tip to re-configure my Hermes on a proper way ?

Thanks in advance

[KarhU]

I've intalled the 4.30 in French and started to test it : GREAT JOB Schaps !

I've noticed a small issue regarding the Keyboards. I've got a French one (azerty) and the 'a' and 'q' keys are swapped with the blue associated keys as well.

Is anybody have a tip to re-configure my Hermes on a proper way ?

Thanks in advance

This is not a proper thread where to ask but anyway:

You can change keyboard layout using Schap's Advanced Config (you will find it in settings or in Programs>System). Just select your layout and press done, then restart your PDA and you should be set.

[frustin]

thanks for the tips in the 2nd (or 3rd) post on page 1, never knew them. is there anymore to get the most out of the tytn?

[Mikeapollo]

This all really needs putting into the wiki under the technical stuff. As I've been doing a bit of editing recently if Karhu and others agree I'm happy to script up a page somewhere over the weekend (unless someone beats me to it!)

[aaronwms]

Excellent research and work! Now, I have a better picture about the ROMs. Thank you so much for your generous contribution to the community.

[alkapone02]

Great job brother

[rotaflex]

Waiting for the results of the new ones these two weeks....

[larsuck]

Just thought I would give this thread a bump. I am wondering how these new 3.62.3 ROMs with big storage are going to stack up.

KarhU, where arhU?

[KarhU]

Hello guys, I am sorry for not feeding you numbers in your time of need :) I was working on my other projects (I localized three apps into Slovak and Czech languages) and also dealt with some off-line (real?) life issues. You can expect the benchmarks updated in next few days. However I think I will wait a little, because with the emergence of legit WM 6.1 all of the major cooks will upgrade their ROMs (features like threaded SMS are too cool to be left out). So I expect Schap's, kin0kin, Matt, Pay, Randomize, shamanix, Berd and others to provide us with some high quality stuff soon. Later today I will post the results for the new PDAViet 6.1 ROM So stay tuned!

[KarhU]

New WM 6.1 PDA Viet benchmark results added. Rom outperformed all the older ROMs in "File Explorer large folder list" by quite a margin (25% faster than previous fastest ROM in this area). Other than that its results are average.*

[corepda]

Great index bro, thanx...Really very helpful...

[kc_hubbard]

Any chance of getting one (or more) of the new 3.62 super storage ROMs tested? I know there are probably a lot of people like me who feel that these are pretty fast...just curious.

[Skabsy]

Hey all,

I did some statistics on KarhU's numbers and obtained a simpler ranking for the existing ROMs. The following list is only a simplified version of KarhU's statistics, so that if you do not want to get lost in numbers :) you can choose a rom from the following list:

Shamanix`s Grass v2.2 100.00
Pandora Naked ROM 6/3G Dialer 99.84
Schap's WM6 Pro 4.22 Lite 98.21
Schap's WM6 Pro 4.20 Lite ROM 96.48
mERCURY v2.1 Base AT&T 3.62 96.36
Pandora Naked 5/ 4MB PagePool 94.24
Schap's WM6 Pro 4.00 Lite 94.19
Pandora Naked ROM 5 94.13
Schap's 4.00 Full 93.94
Austinsnyc's AT&T 3.62 Clean ROM 93.85
K's Rom v1.30a 93.28
K's Rom v1.21/6mpp 92.95
Black Satin Light 92.38
Shamanix`s Grass v1.1 4m PP 91.42
Black Satin Medium 90.52
BlackDragon 89.98
Telstra Imate 3.62.305.5 Official R 89.50
ROM v3.62.07.08 CHO PDA VIET 88.25
TNT.1921 Professional 87.05
HyperCore Lite 3.62SE 86.70
TNT.SE "Special Edition" Profession 86.63
Pays WM6 WWE v1.23 Big Storage ROM 86.04
Schap's WM6 Pro 4.22 Full 85.95
Pandora v3.0 Beta 3 85.43
Schap's 4.01 Full 85.22
Pandora Naked ROM 2 with NETCF 3.5 84.94
Pandora Naked ROM 2 84.68
Pandora Naked ROM 2 with NETCF 2.0 83.14
Schaps 3.57a Lite 82.07
Dopod Lite 1.0 81.97
TNT.SE "Special Edition" Official 81.20
Black Satin Normal 81.15
vp3g WM6 v3.0 Official 80.45
Faria's Super Clean3 ROM * 79.80
Pays WM6 WWE v1.22 Big Storage ROM 79.03
S64 3.62/HTC WWE Rom 78.74
Cacahuate's 2.0 WM6 78.64
Elise .12 78.17
vp3G's 3.62.3 AT&T WM6 ROM 77.25
CUSTEL Im Back No PTT - 6MB PP 74.80
Sleuth's v3.1 Developer ROM 73.54
Pandora Halo 3 3.62 69.37
Faria's Custom V1 67.91
Schaps 3.57a Full 60.42
Last Samurai 58.37
Schap's 3.30c Full 55.56
S64 3.62/7 WWE Rom 54.98
Dopod Official WM6 ROM 0.00


Hi and excuse my niaveity as new to this. According to your results here, are you saying that with the Schap's roms that I tend to use, the lite versions are better than the full?

Thanx and big shoutout to KarhU for starting this off.

[KarhU]

Guys, I did some benchmarking with newer ROMs and I got unusually low results. I will further investigate into it, but can please someone test the Pandora 8.3 beta and Shamanix' Ultimate 2.1 using the methodic on the first page of this thread? I got results of 340 and 346 respectively in the overall benchmark result after multiple benchmarking of each ROM (as these ROMs are lite ROMs the results should be better - around 360, but maybe the culprit is somewhere else - such as ROM base or cooking tools). I want to eliminate possibility that the results are flawed due to my device as some people base their ROM selection on these results. Thank you for your support.

[neva dinova]

Guys, I did some benchmarking with newer ROMs and I got unusually low results. I will further investigate into it, but can please someone test the Pandora 8.3 beta and Shamanix' Ultimate 2.1 using the methodic on the first page of this thread? I got results of 340 and 346 respectively in the overall benchmark result after multiple benchmarking of each ROM (as these ROMs are lite ROMs the results should be better - around 360, but maybe the culprit is somewhere else - such as ROM base or cooking tools). I want to eliminate possibility that the results are flawed due to my device as some people base their ROM selection on these results. Thank you for your support.

i might be wrong but aren't most of the big storage roms a little slower than their regular storage counter parts?

[KarhU]

It depends - some superstorage ROMs tend to be slower. But the fastest ROMs are the "Lite" version almost invariably. (but then again the biggest difference is made by the amount of available RAM - the less today plugins activated, the better benchmark result). That is why I want to improve the benchmarks - see the link in the first post.

[elivne]

really interesting to see

[CRCinAU]

Hi KarhU,

Can you please run a benchmark on my latest ROM... It's based on WM6.1 - it seems lightening quick - I'd like to see if things have changed since WM6.0 benchmark wise.

Thanks :)

[kazuya_gonzalez]

hmm karhu will u update these benchmark?

[magix]

please update the base. with all those fast roms it would be nice to konow how they compare

[adry1988]

i was just given a spv m3100 (herm100) by a friend of mine. and i want to upgrade the rom. i have never done it before can you please help me upgrade or send me a link with the intructions.

[joshkoss]

i was just given a spv m3100 (herm100) by a friend of mine. and i want to upgrade the rom. i have never done it before can you please help me upgrade or send me a link with the intructions.
Wow dude, why post on this thread, is'nt it very old.. plus not your topic.. dude
look here (http://forum.xda-developers.com/showthread.php?t=299462)and http://forum.xda-developers.com/showthread.php?t=419158

[Derik.cloete]

Great work!!!

I really think we all appreciate your days work to make our lives a little easier vinding good roms. I did not check to see if you tried TNT .19199 aswell, allthough i am using a HERM 200 it runs great with HomeScreen app built in. Its a must try. Sorry, I dont have the link right now. Someone help please

[D_NazGUL]

Just wanted to know if there are going to be some new benchmarks in the near future or not? :)

[adamkhanmian]

I flashed maaany in a couple of days... Was just checking which one suited me....I stuck to Schaps full version for longest...its great but was a lil slow with all my stuff in it. Im sure this prob wouldve been solved if i had flashed the lite one but it had no windows live and i NEED windows lite.. nice work schaps!!

[slaki85]

Nice work man..

[slaki85]

Well,I tested really many roms.Faster roms are roms who haved 31 mb of ram after install...If you use a lot surfing on the Internet,more ram is very important.Also,clean lite roms are so far best.And much stable.Many cooked roms have some "stupid" addons who take ram memory..After many test I can confirm that is Grass 2.2 much faster (and Black roms) bcs they are very naked and very fast.Applications like Igo 8,or Opera browser are much faster in this roms.Full roms with many preinstalled softwares,who comed with 22-25 mb of available rams after few programs running are very slow....very...useless...Clean roms are best for users.I used few roms with Manilla and some other Today craps...They use large amount of rams.And look only good..But slow and useless for any work...Sry if my eng is very bad...My advice Grass 2.2 with 31.8 mb free ram (amazing fast)