[UPDATED January 29] Which is the best WM6 ROM? Benchmarks of 52 ROMs + links

[KarhU]

Ladies and Gentlemen!

After an extended period of flashing I am pleased to present you with the result of my effort

I took 52 ROMs (all except for one based on the official releases) and run them through benchmarks in SPB Benchmark 1.6.

ROM listing with links to their threads:
Black Satin: Light, Medium, Normal
Schap's 3.30c Full
Schap's 3.57a: Lite, Full
vp3g's v3.0 Official
Sleuth's Developer ROM v3.1
Cacahuate's v2.0
Elise .12
Pandora v3.0 beta2
Dopod official release ROM
Dopod Lite ROM
Faria's Super Clean v1 (3rd update) ROM
Faria's HTC Hermes Customized v1
WM6 Samurai Final
Schap's WM6 Pro 4.00 Full
Schap's WM6 Pro 4.00 Lite
vp3G's WM6 v3.62.3 AT&T ROM
HyperCore Lite 3.62SE
Shamanix`s Grass v1.1
Starbase64 3.62/7 WWE Rom
TNT.1921 Series Professional
Pandora Rom Halo 3 Edition
Schap's WM6 Pro 4.01 Full
CUSTEL Im Back No PTT - 6MB Page Pool
Pandora Naked 2 ROM
S64 3.62/HTC WWE Rom
HCLite 3.62 Final
Austinsnyc's AT&T 3.62 Clean ROM
Schap's WM6 Pro Lite 4.20 ROM
K's Rom v1.21 and 1.30a Lite
Pandora Naked 5 ROM
Pandora Naked 6 ROM
BlackDragon WM6 ROM
Pays WM6 WWE v1.22 and 1.23 Big Storage ROMs
Schap's WM6 Pro 4.22 Full and Lite ROMs
Shamanix`s Grass v2.2
ROM v3.62.07.08 CHO HTC TyTN PDA VIET
mERCURY v2.1 Base AT&T 3.62
TNT.SE "Special Edition" Official and Professional ROMs
Telstra imate RUU_Hermes_DL_UK_3.62.305.5_6275_1.54.30.10_108 Official ROM
Pays WM6 WWE v1.30 Big Storage Lite ROM
K's Rom v1.31 Full
CRCs PHKv11 Release 3 Complete ROM
BlackDragon WM6 ROM Updated
Schap's WM6 Pro 4.30 Full ROM
PDA Viet WM 6.1 19716.1.1.0 PV

ROMs were tested on my HERM 100 (Orange M3100) with 1 GB Kingston MicroSD. Radio version is 1.54.30.0 and during the testing phone part of the device was off (flight mode). Device was attached to a charger. All ROMs were tested immediately after the flash. SPB benchmark was installed into the main memory. After that I soft resetted the device, and stopped all running programs using settings>>memory. All ROMs were tested with their original settings - i.e. no today plug-ins were disabled. Results are device specific - some tests use the MicroSD card - so if you do the tests your results will probably differ.

Findings:

1) Cooked ROMs do make a difference: increased performance is obvious, additional tweaks and enhanced functionality of cooked ROMs make official ROMS quite useless.
2) Lite ROMs are more snappy and as they get more bloated their speed diminishes. Install only programs you need.
3) Black ROMs seem to be the fastest so far.
4) Faria is really a great cook (his ROM proved worthy it's almost legendary status )
5) There is more than just benchmark results that makes a ROM great. Read their respective threads about their advantages and disadvantages.
6) Cooked ROMs have made a noticeable progress since january. Older benchmark results can be found here and here. The performace and usability gain is quite impressive.
7) Older ROMs scored lower than the newer ones. Keeping your device updated is good for it's health
8) Actual differences between latest ROMs are quite small. Choose according to your taste.
9) Out of new (3.62.0-3) ROMs Schap's ROMs are outstanding. He is like a good (French) wine - gets better with age, The only ROMs with similar results were Black ROMs which are out of development now
10) Newest ROMs are very fast - including official one from Telstra - it seems that the video playback problem older official ROMs had is gone and official ROM now can compete with cooked ROMs in terms of speed
11) kin0kin has made a straight run for the elite cookers with his K's Rom v1.21. It is one of the fastest available.
12) Shamanix's Grass 2.2 is officially the second fastest ROM I have tested (R.I.P. Black ROMs)
13) New Pandora 6 Naked ROM also broke records in some categories...

Notes:

- I will not run tests on following ROMs:
- Sajakati's HypercoreLite froze while benchmarking (only HR solved the situation) and though I run less tests than usually it did not finish properly - benchmarking not recommended due to extreme instability while doing so) However new Black Dragon Sajakati's ROM was running very smoothly and I had absolutely no problems benchmarking it.
- Please note that benchmarking is a stress testing, so eveni if the benchmarking does not work on a ROM it does not mean it will brick your device someday.
- It seems that NETCF version has no impact on overall device performance (tested the ROM with either NETCF 2.0 or 3.5) compared to the ROM without NETCF or between the NETCF versions. That however might not be true for using programs that use these frameworks.

To do:
- Include more ROMs
- Include tabular comparison of ROMs features
- Include benchmarks from other programs
- Try to persuade mods to do this thread sticky....
- Any wishes, comments?

Updates:
29.01,2008 - Added PDA Viet's WM 6.1 ROM
02.01.2008 - Added 5 New ROM Benchmarks
22.12.2007 - Added K's Rom v1.30a Lite benchmark results
21.12.2007 - Added PaYs 1.23 ROM benchmark results
19.12.2007 - Added 13 New ROM benchmarks
07.12.2007 - Added 5 New ROM Benchmarks, added excellent ROM comparison and ranking by Niteline (kudos buddy!) in the zip package, FROZE THE PANES IN .XLS - REJOICE!
16.11.2007 - Added 2 New ROM Benchmarks - Shap's 4.01 Full and CUSTEL Im Back No PTT - 6MB Page Pool
10.11.2007 - Added 8 new ROM benchmarks, added RAM and ROM after flash stats, revived the thread
31.07.2007 - Added links to the ROM threads, added two ROM benchmarks, added results in xls (MS Office 2003) , xml and mht formats - everybody can format the results to his/her liking now.
31.07.2007 - Added Schap's 3.30c Full ROM benchmark

Thank yous:
All of the chefs that make this flashing fiesta possible...
THANK YOU ALL FOR YOUR SUPPORT

Please help me to make the benchmarks better. More info here.

[KarhU]

Performance tweaks that will improve responsiveness of your ROM no matter which you choose (Kudos Cyphol):

1. Improve performance/response time:
- Start > Settings > Phone => Keypad = Off
- Start > Settings > Sounds & Notifications; Under Sound Tab => Disable/Untick Events (warings, system events) - this tweak will BRUTALLY increase the responsivenes of Start menu and Quick Menu 2.7 (because system will no more have to search for sound files in the ROM everytime you tap the screen)

2. Your Windows Mobile device is just like your Windows PC. So, soft reset it before going to sleep (or whenever appropriate) once a day to make it runs good at all time (because of the memory leak when you close the program not all the memory is restored - sof reset restores RAM lost due to Memory leak).

3. Make sure you install the today plugin apps such as Resco Today Plugin etc. on the internal/device memory. Some today plugin can be installed on the storage card but it'll slow down your system performance a bit.

4. Do not install too many apps on the internal/device memory. Leave as much free space as you can on it. Check whether it can be install on the storage card or not. Normally, don't let it become as low as 8MB free but 10MB is enough already.

5. Know what you're doing such as tampering your system registry. Make sure it can be done and you're doing it the right way. If not, then leave it as is.

6. Never hide the WirelessManager Tray Icon at the bottom.

Don't believe me?

Try it and feel the difference.

This is the key:

[HKEY_LOCAL_MACHINE\Services\WirelessMgr]
"Keep"=dword:00000000 ;<--- (dword:00000000 to hide and dword:00000001 to show)

7. Never let your device go to sleep by itself. Do it manually by pressing the power button everytime when you've finished using your device.

8. Make sure you shutdown the 'Activesync' completely after activesyncing your device with your PC. You can use task manager (X-Button), Magic Button etc or you can use memory applet in setting > System > memory; Running Programs

[raymondu999]

YES!!! Finally some solid numbers to depend upon for comparison! Btw, how long did it take you to do this frenzy of HR, flash, benchmark, HR flash, benchmark?

[KarhU]

Originally Posted by raymondu999 YES!!! Finally some solid numbers to depend upon for comparison! Btw, how long did it take you to do this frenzy of HR, flash, benchmark, HR flash, benchmark?

Including waiting on free slot on rapidshare it was cca one full day .

[bapski]

would a day of benchmarking suffice for conclusive results? although this would help would be FLASHERS to decide which ROM they can try first instead of starting another "WHICH IS THE BEST ROM" thread.

[raymondu999]

Originally Posted by bapski would a day of benchmarking suffice for conclusive results?

The thing is, SpB Benchmark is a program, as in lines of code. Hence, these numbers might not mean much for stability, but for speed, these numbers tell no lies.

[KarhU]

Originally Posted by bapski would a day of benchmarking suffice for conclusive results?

As I have wrote before - benchmarks are just numbers. Actual experiences vary. Decisions about which ROM to use should not be solely based on benchmarks. Important factors are also support and or newer versions in the making, feel of the ROM (they do not look all the same) and for example bonus features also differ. Choose according to your needs.

[typester]

some of the top scores were not in blue?? and ones that are tied should be in another color like yellow to show equality. i know you have spent lots of time on this and i don't mean to harp on you, so these are only suggestions.

thanks for the hard work!! this was a great idea. i will look forward to you adding new ROM's.

[gerald_s]

Guess I missed something. Where did you find Farias super clean 3 rom? I only can find super clean v1.

BTW: Great idea, I hope somebody will add this list to the wiki

[Bensat]

Great work!!!

I've asked myself this question over and over again........... Hahaha but now I see these results I can only say to the COOKIES that they've done a GREAT job for all of us.
That is what matters, our guys even listen and they will sometimes ad some special flavors into a ROM. This development and research goes so fast I even wonder if the developers of HTC come here to see what is going on over here!

Keep up the good work!