[KNOWLEDGE][H81x|G4] The AMAZING and INCREDIBLE twrp BENCHMARK

[steadfasterX]

The AMAZING and INCREDIBLE twrp BENCHMARK


About

I just came to the point where I wanted to use the best performance settings while backing up with TWRP.
That said I looked around and there are no recommendations on I/O scheduler, CPU governor, read_ahead_kb or/and filesystem.
So I decided to benchmark on my own to provide the best speed in TWRP.
Well and maybe someone cares so I share that here for the public
Besides that I will create a new TWRP image containing the best settings based on the test I made.

Hardware

Device: LG G4 - H815 international (all these can be adapted to all other G4 variants of course)

• SDCard #1: BRAND NEW bought - SanDisk Ultra Android 64 GB microSDXC Memory Card, FFP up to 80 MB/s, Class 10, UHS-1
  

     PC benchmark with GNOME disk (gnome-disk):
          measurement size 50,0 MiB (52.428.800 Bytes)
          average read: 89 MB/s (100 measuring points)
          average write: 30,6 MB/s (100 measuring points)
          average access time: 0,64 ms (1000 measuring points)
      
      4 measuring points directly on the device (within twrp):
        ~ # dd if=/dev/zero of=/external_sd/testfile bs=1G count=1
          1073741824 bytes (1.0GB) copied, 26.909159 seconds, 38.1MB/s
          1073741824 bytes (1.0GB) copied, 28.349294 seconds, 36.1MB/s
          1073741824 bytes (1.0GB) copied, 28.204614 seconds, 36.3MB/s
          1073741824 bytes (1.0GB) copied, 27.088227 seconds, 37.8MB/s

  That means in theory it would be possible to get a write speed between: ~30,6 MB/s and 38.1MB/s with this card

• SDCard #2: BRAND NEW bought - Sandisk Extreme 128GB SDXC Memory Card with SD Adaptor up to 90 MB/s, Class 10, UHS-3
  

     PC benchmark with GNOME disk (gnome-disk):
          measurement size 50,0 MiB (52.428.800 Bytes)
          average read: 87 MB/s (100 measuring points)
          average write: 51 MB/s (100 measuring points)
          average access time: 0,40 ms (1000 measuring points)
      
      4 measuring points directly on the device (within twrp):
         ~ # dd if=/dev/zero of=/external_sd/testfile bs=1G count=1
           1073741824 bytes (1.0GB) copied, 19.063505 seconds, 53.7MB/s
           1073741824 bytes (1.0GB) copied, 19.155900 seconds, 53.5MB/s
           1073741824 bytes (1.0GB) copied, 19.517557 seconds, 52.5MB/s
           1073741824 bytes (1.0GB) copied, 18.673294 seconds, 54.8MB/s

  That means in theory it would be possible to get a write speed between: ~51MB/s and ~54.8MB/s with this card

Those benches above are just for getting the max possible values. They are not that much meaningful on their own because I want to know the performance of a twrp backup itself of course.

Backup setup

All test-set runs are made with the following backup sources:

Images:
Well those are very small so it is not a very good test - the "Files" section is really more important but I use it nevertheless.

• Recovery (40 MB)
• Boot (40 MB)

Files:

• /system (1520 MB - ROM) - formatted as EXT4
• /data - formatted as EXT4
  - 1x 500 MB file (created once with dd if=/dev/urandom ...)
  - 1x 1GB file (created once with dd if=/dev/urandom ...)
  - 1x 3 GB file (created once with dd if=/dev/urandom ...)
  - normal /data content (~500 MB random data)
  --> decrypted /data ! means we have another layer between

How


A test-set is a combination of scheduler, governor and readahead size and those are the candidates:

• CPU governors: intelliactive, performance
• I/O schedulers: fiops, deadline, sioplus
• read_ahead_kb: 1024, 2048
• filesystem (sdcard): EXT4, F2FS
• compression: on, off (compress the backup or not)

So to be more detailed good: means already finished)

EXT4 (Sandisk Extreme 128GB - Class 10, UHS-3)

TBD

F2FS (Sandisk Extreme 128GB - Class 10, UHS-3)

TBD


EXT4 (SanDisk Ultra Android 64 GB - Class 10, UHS-1)

• :good: intelliactive + fiops + rh 1024
• :good: intelliactive + deadline + rh 1024
• :good: intelliactive + sioplus + rh 1024
• :good: intelliactive + fiops + rh 2048
• :good: intelliactive + deadline + rh 2048
• :good: intelliactive + sioplus + rh 2048
• :good: performance + fiops + rh 1024
• :good: performance + deadline + rh 1024
• :good: performance + sioplus + rh 1024
• :good: performance + fiops + rh 2048
• :good: performance + deadline + rh 2048
• :good: performance + sioplus + rh 2048
  
• :good: intelliactive + fiops + rh 1024 + compression
• :good: intelliactive + deadline + rh 1024 + compression
• :good: intelliactive + sioplus + rh 1024 + compression
• :good: intelliactive + fiops + rh 2048 + compression
• :good: intelliactive + deadline + rh 2048 + compression
• :good: intelliactive + sioplus + rh 2048 + compression
• :good: performance + fiops + rh 1024 + compression
• :good: performance + deadline + rh 1024 + compression
• :good: performance + sioplus + rh 1024 + compression
• :good: performance + fiops + rh 2048 + compression
• :good: performance + deadline + rh 2048 + compression
• :good: performance + sioplus + rh 2048 + compression

F2FS (SanDisk Ultra Android 64 GB - Class 10, UHS-1)

• :good: intelliactive + fiops + rh 1024
• :good: intelliactive + deadline + rh 1024
• :good: intelliactive + sioplus + rh 1024
• :good: intelliactive + fiops + rh 2048
• :good: intelliactive + deadline + rh 2048
• :good: intelliactive + sioplus + rh 2048
• :good: performance + fiops + rh 1024
• :good: performance + deadline + rh 1024
• :good: performance + sioplus + rh 1024
• :good: performance + fiops + rh 2048
• :good: performance + deadline + rh 2048
• :good: performance + sioplus + rh 2048
  
• :good: intelliactive + fiops + rh 1024 + compression
• :good: intelliactive + deadline + rh 1024 + compression
• :good: intelliactive + sioplus + rh 1024 + compression
• :good: intelliactive + fiops + rh 2048 + compression
• :good: intelliactive + deadline + rh 2048 + compression
• :good: intelliactive + sioplus + rh 2048 + compression
• :good: performance + fiops + rh 1024 + compression
• :good: performance + deadline + rh 1024 + compression
• :good: performance + sioplus + rh 1024 + compression
• :good: performance + fiops + rh 2048 + compression
• :good: performance + deadline + rh 2048 + compression
• :good: performance + sioplus + rh 2048 + compression

In general the following has to be mentioned as well:

• every single test-set will be done 3 (three!) f*** times and based on that an average will be calculated
• after each single run TWRP will be rebooted to ensure a clean start
• all tests are made in twrp v3.0.2-1

Status

Just check the above :good: marks .. that means (X * 3) = Y single backup runs :silly:
To be able to test that without much effort I've created a benchmark script which autostarts a twrp backup with given options via adb (will be provided later as well).
I document all that stuff in LibreOffice Calc + nice graphs. Everyone likes graphs (and tbo its the only way to see the diffs later).

You see I still need to run dozens of tests!!!
.. but for the moment it seems that sioplus and f2fs are a good choice for backing up fast but we will see the real winner at the end first..

Not surprisingly the sdcard itself is crucial for speed so I will test another sdcard which is UHS-3(!!!) at the end as well. At least for the most promising of the above test-sets.


Credits
@marn3us : yea you are the reason behind doing all this


Famous last words

Don't trust any benchmark not made by yourself. They are all telling lies.


.

 

[steadfasterX]

Results

The incredible result graphs

But first of all: HOW TO READ THOSE GRAPHS?

I made 4 graphs for each filesystem and split them further by compression or not, time and speed.

Speed:
This bar chart shows the average (over 3 runs) MB/s for each test-set. You see a line overlay as well which shows the fastest speed.
The higher the better!

Time:
This bar chart shows the average (over 3 runs) time taken in seconds for each test-set. The line overlay shows the fastest test-set.
The lower the better!


EXT4 - UHS-1

• compression - SPEED
  
  
• compression - TIME
  
  
• without compression - SPEED
  
  
• without compression - TIME
  

F2FS - UHS-1

• compression - SPEED
  
  
• compression - TIME
  
  
• without compression - SPEED
  
  
• without compression - TIME
  

Conclusion

What does that all means?

1. There is not the one and only (even this is not that surprisingly no?).
2. It matters which filesystem you use on your sdcard.
3. It matters if you use compression or not.
4. There is no specific setting (like the readahead size) which is best for all test-sets. The combination matters!
5. No rule without exceptions: The scheduler sioplus seems to be a good choice in most cases but it depends on the filesystem.

So when it comes to have the best working test-set for all of this there is no real winner at all.
Well but one can say there are loosers e.g. look at:

• F2FS+Compression - TIME chart and find the longest bar (501 secs).
  or
• EXT4+Compression - TIME chart: find the longest bar there (666 secs).

So one could say:
The most people may use compression to keep the backup small and the most people use EXT4 or vFAT on the external SD card.

In that case (compression + EXT4) the winner is (with very close competitors):
EXT4, governor=performance, scheduler=sioplus, rh=1024

If you want the best speed for that case (compression on) it seems this is the combo of your choice:
F2FS, governor=intelliactive, scheduler=sioplus, rh=1024
(difference to the best EXT4 = 24 secs in time and 0,67 MB/s in speed)

If you say that you have a big sdcard and don't care about the backup size the best combo seems to be (without compression):
EXT4, governor=intelliactive, scheduler=deadline, rh1024

Again the differences between the followers of the winner are so small (e.g. 1 sec) that one can say they are in measure tolerance.
If we assume a measure tolerance of 5 secs there are 7 winners for EXT4 + compression and 8 with F2FS + compression.
You see this all just let one conclusion: choose whatever you like but avoid the "bad" ones:

on F2FS avoid (without or with compression): governor=intelliactive + scheduler=deadline + rh 2048
on EXT4 avoid (without or with compression): governor=performance + scheduler=deadline + rh 1024

Keep in mind that all those values are for taking a TWRP backup only.
I haven't measured a restore and it may has nothing to do with your daily usage in Android.
It's just a very specific benchmark for a specific use case.


Please find the ODS libreoffice calc sheet with really all the amazing details attached (only ZIP allowed so you have to extract it first )




.

 

[obvious]

Love it ?

 

[marn3us]

Thanks a lot @steadfasterX, this is a truly amazing work!!

But now I feel guilty for having suggested this idea that is taking so much of your time

Keep it up mate!!

 

[steadfasterX]

Thanks a lot @steadfasterX, this is a truly amazing work!!

But now I feel guilty for having suggested this idea that is taking so much of your time

Keep it up mate!!

Oh yes you are right shame on you
I updated the OP (look at credits) :angel:

But seriously I just want to make things right :highfive:

 

[marn3us]

Oh yes you are right shame on you
I updated the OP (look at credits) :angel:

But seriously I just want to make things right :highfive:

Haha now I feel even more guilty!

Keep up the great job mate, looking forward to seeing your results!

 

[steadfasterX]

Haha now I feel even more guilty!

hahaha good

one more testing marathon day is over - a lot tests done - a lot still to do.. updated the test-set list in OP..



.

 

[marn3us]

hahaha good

one more testing marathon day is over - a lot tests done - a lot still to do.. updated the test-set list in OP..



.

Thanks again!

I also really hope F2FS support will be introduced in AOSP roms as well because it looks like it makes quite a bit of difference!

 

[steadfasterX]

Thanks again!

I also really hope F2FS support will be introduced in AOSP roms as well because it looks like it makes quite a bit of difference!

You just need a kernel which is supporting f2fs (like llama kernel)

 

[marn3us]

You just need a kernel which is supporting f2fs (like llama kernel)

Wow, are you sure that we just need support on recovery and kernel side?

Because I"m running llama sweet but I feared 7.1 Nougat wasn't ready for f2fs for some reason

 

[steadfasterX]

Wow, are you sure that we just need support on recovery and kernel side?

Because I"m running llama sweet but I feared 7.1 Nougat wasn't ready for f2fs for some reason

Let's see what genessix will tell us I have absolutely no experience with nougat yet so there could be things I do not know ..

 

[steadfasterX]

....oooook so I'm finished with testing and documentation..

I just need to make meaningful graphs out of that now.
What I can say is that there is 1 test-set looser but the other test-sets are very close to each-other.. We will see how this will be reflected in an overview.
I will take the time in seconds as the most important measurement point as MB/s alone is not enough.
I will not graph the "Images" backup as they are too small to be a good test object.

The other thing is my UHS-3 card arrived yesterday and I added the first general benchmark of it to the OP as well.

When I have the graphs ready I decide which tests I will re-run on the UHS-3 card, too.

So at the end what I can say for SURE is: if your sdcard is UHS-1 or UHS-3 matters more then any test-set combination.
Not that surprisingly right? But well I just want to add this here as a note.

stay tuned


UPDATE:

It's time for rock'n'roll:
http://forum.xda-developers.com/showpost.php?p=69852331&postcount=2





.

 

[Vipery]

What about best governor/sheduler/rh combo for performance/battery? I'm running stock with Samsung 64Gb Ultra Red micro sd, pretty fast card.

 

[marn3us]

Very interesting results!! Thanks again for your hard work!!

What combination is your default 3.0.2-1 version using? Will it receive a small update/tweak after these tests?

Ps: by looking better at the data and the graphs, SioPlus+IntelliActive looks like a great combo!! It is almost always the best or very close to the best!

 

[steadfasterX]

What about best governor/sheduler/rh combo for performance/battery? I'm running stock with Samsung 64Gb Ultra Red micro sd, pretty fast card.

First of all: The class + UHS number are important so if you have a class 10 card it does not mean you will have the best speed. The UHS class makes the difference. (https://www.sdcard.org/developers/overview/speed_class/)

You mean for TWRP? All those results and tests are made for a TWRP use case not for Android. So for Android it depends heavily on your usage.
I haven't tested for battery as in TWRP all I want is speed.

My personal experience had shown that for Android zzmoove is a good battery saver but for scheduler and rh it depends. So the only thing I can recommend here is to test them on your own because every user has different usage scenarios..

Very interesting results!! Thanks again for your hard work!!

What combination is your default 3.0.2-1 version using? Will it receive a small update/tweak after these tests?

Ps: by looking better at the data and the graphs, SioPlus+IntelliActive looks like a great combo!! It is almost always the best or very close to the best!

Thx (and take a look at my sig again )
Well .. I believe it was performance + rh=512 + deadline but tbo I'm not sure (and cannot take a look on it atm)

Yes sioplus + intelliactive are my favorites. it seems to make no diff if we use 1024 or 2048 rh so i may test that again with 512 just to see what happens..

Yes the next TWRP version will be released with the winners (see also https://github.com/steadfasterX/kernel_lge_llamasweet/issues/1)


.

 

[marn3us]

First of all: The class + UHS number are important so if you have a class 10 card it does not mean you will have the best speed. The UHS class makes the difference. (https://www.sdcard.org/developers/overview/speed_class/)

You mean for TWRP? All those results and tests are made for a TWRP use case not for Android. So for Android it depends heavily on your usage.
I haven't tested for battery as in TWRP all I want is speed.

My personal experience had shown that for Android zzmoove is a good battery saver but for scheduler and rh it depends. So the only thing I can recommend here is to test them on your own because every user has different usage scenarios..



Thx (and take a look at my sig again )
Well .. I believe it was performance + rh=512 + deadline but tbo I'm not sure (and cannot take a look on it atm)

Yes sioplus + intelliactive are my favorites. it seems to make no diff if we use 1024 or 2048 rh so i may test that again with 512 just to see what happens..

Yes the next TWRP version will be released with the winners (see also https://github.com/steadfasterX/kernel_lge_llamasweet/issues/1)


.

That is great! Yeah if I remember correctly, the read-ahead especially helps in sequential situations so it may not provide large boosts to twrp backup. If you try 512kb please let me know, I'm curious

Ps: when I had said time ago that in my test best was 2048Kb+Zen I had meant for android usage in general, because I had no idea how to benchmark TWRP :angel: thanks for finding a way to do it!