FORUMS
Remove All Ads from XDA

[TOOL] Lanchon REPIT: The Data-Sparing Repartitioning Tool For Android

2,672 posts
Thanks Meter: 4,447
 
By Lanchon, Senior Member on 13th April 2016, 06:50 PM
Post Reply Email Thread
21st April 2016, 02:52 AM |#11  
Junior Member
Thanks Meter: 0
 
More
Pls help
##################################
Lanchon REPIT
A Data-Sparing Repartitioning Tool
Version: 2016-04-15
Device: i9100
Copyright 2016, Lanchon (GPLv3)
####################################

===== PRELIMINARY CHECKS =====
info: valid package names: <prefix>[-(system|data|sdcard|preload)=<conf>]...<suffix>
info: valid partition <conf> values: [<size-in-GiB>|same|min|max][+[keep|wipe][+[ext4|vfat|f2fs|swap|raw]]]

----- DEFAULTS -----
system = size:same + content:keep + fs:ext4
data = size:same + content:keep + fs:ext4
sdcard = size:same + content:keep + fs:vfat
preload = size:same + content:keep + fs:ext4

info: parsing package name

----- CONFIGURATION -----
system = size:1.0 + content:keep + fs:ext4
data = size:6 + content:keep + fs:ext4
sdcard = size:max + content:keep + fs:vfat
preload = size:min + content:keep + fs:ext4

info: disabling swap
BusyBox v1.22.1 bionic (2016-01-10 12:49 +0530) multi-call binary.

Usage: dirname FILENAME

Strip non-directory suffix from FILENAME

info: copying package to '/tmp'
cp: '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' and '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' are the same file

[ERROR 1]

Whats wrong
21st April 2016, 12:52 PM |#12  
Senior Member
Flag the wild south
Thanks Meter: 239
 
More
just change your current directory to /tmp and try to flash it again from there. (all within the recovery)
21st April 2016, 03:53 PM |#13  
Senior Member
Thanks Meter: 123
 
More
Quote:
Originally Posted by Lanchon

ok. then i would format in ext4 and trim. you can use repit for doing both in a single shot, but you need a trimming kernel. take a dd snapshot of one or a couple of short pieces of the partition, both before and after the operation. compare them and post here, i'd surely like to know .

but... keep in mind that there are several different emmcs out there in the S2s, so you'd need to validate for each one. which reminds me, also post your device emmc IDs! use the eMMC brickbug check app for instance.

regarding the /dev/urandom vs /dev/zero thing, i'm not religious, i don't hold unwarranted beliefs about what i don't know. i investigate, i have no gut feelings on stuff

bauner produced a new n7000 IsoRec with TWRP + OTG which I wanted, so while I was installing I tested a few things.

I generated a block binary file with each of the 1kB blocks annotated for the 200MB n7000 cache partition, and dd'd the file to the cache partition prior to each mkfs.

Code:
$ cat blocknull.pl
#!/usr/bin/perl
use strict;

my $count = $ARGV[0];
$count = 1 if $count == 0;
$count *= 1024 if $ARGV[0] =~ /M$/;
$count *= 1024 * 1024 if $ARGV[0] =~ /G$/;

for (my $i = 1; $i <= $count; $i++)
{
	printf "\nBLOCK %10d kB %6d MB %2d GB BLOCK\n", $i, $i / 1024, $i / 1024 / 1024;
	print chr(0) x (1024-43);
}
$
Code:
BLOCK          1 kB      0 MB  0 GB BLOCK
...
BLOCK     204800 kB    200 MB  0 GB BLOCK
Then operated on both vfat and ext4 file systems to determine which of the 204800 block signatures where remaining after various operations.

Code:
# tr -d '\0' < /dev/block/mmcblk0p7 | grep BLOCK.*BLOCK > cache.x.y.desc
...
# wc -l cache.?.?.*
   204800 cache.1.1.dd
   201632 cache.1.2.mkfs.fat32
   197041 cache.1.3.copyfile
   204800 cache.2.1.dd
        0 cache.2.2.mkfs.ext4
   204800 cache.3.1.dd
   197380 cache.3.2.mkfs.ext4.twrp
       60 cache.3.3.fstrim
       60 cache.3.4.copyfile
#
So as expected vfat format doesn't change the underlying data returned, it just gets changed as it is used, as show by the signature counts on cache.1.*

ext4 is more interesting, as I tried a manual mke2fs -t ext4 for cache.2.* and TWRP wipe ext4 for cache.3.*. The TWRP wipe is not using secure discard as can be seen from recovery.log, but things are largely back on track after the first fstrim except for a hole from 131081 kB to 131125 kB.

Code:
recovery.log:
warning: wipe_block_device: Wipe via secure discard suppressed due to bug in EMMC firmware
Maybe more interesting is that even though it seems manual mke2fs -t ext4 does a wipe, when I do a fstrim is still says 180MB trimmed, which I don't expect, some inconsistency in fs metadata setup in mkfs maybe? Doesn't matter to me, 1 or 2 wipes, but seems I will be using mke2fs -t ext4 via terminal and then using TWRP wipe/format as required.

FYI, EMMC IDs PNG attached.
Attached Thumbnails
Click image for larger version

Name:	Screenshot_20160422-000001.png
Views:	494
Size:	94.9 KB
ID:	3726029  
22nd April 2016, 10:56 AM |#14  
Lanchon's Avatar
OP Senior Member
Thanks Meter: 4,447
 
Donate to Me
More
Quote:
Originally Posted by samarium

bauner produced a new n7000 IsoRec with TWRP + OTG which I wanted, so while I was installing I tested a few things.

I generated a block binary file with each of the 1kB blocks annotated for the 200MB n7000 cache partition, and dd'd the file to the cache partition prior to each mkfs.

Code:
$ cat blocknull.pl
#!/usr/bin/perl
use strict;

my $count = $ARGV[0];
$count = 1 if $count == 0;
$count *= 1024 if $ARGV[0] =~ /M$/;
$count *= 1024 * 1024 if $ARGV[0] =~ /G$/;

for (my $i = 1; $i <= $count; $i++)
{
	printf "\nBLOCK %10d kB %6d MB %2d GB BLOCK\n", $i, $i / 1024, $i / 1024 / 1024;
	print chr(0) x (1024-43);
}
$
Code:
BLOCK          1 kB      0 MB  0 GB BLOCK
...
BLOCK     204800 kB    200 MB  0 GB BLOCK
Then operated on both vfat and ext4 file systems to determine which of the 204800 block signatures where remaining after various operations.

Code:
# tr -d '\0' < /dev/block/mmcblk0p7 | grep BLOCK.*BLOCK > cache.x.y.desc
...
# wc -l cache.?.?.*
   204800 cache.1.1.dd
   201632 cache.1.2.mkfs.fat32
   197041 cache.1.3.copyfile
   204800 cache.2.1.dd
        0 cache.2.2.mkfs.ext4
   204800 cache.3.1.dd
   197380 cache.3.2.mkfs.ext4.twrp
       60 cache.3.3.fstrim
       60 cache.3.4.copyfile
#
So as expected vfat format doesn't change the underlying data returned, it just gets changed as it is used, as show by the signature counts on cache.1.*

ext4 is more interesting, as I tried a manual mke2fs -t ext4 for cache.2.* and TWRP wipe ext4 for cache.3.*. The TWRP wipe is not using secure discard as can be seen from recovery.log, but things are largely back on track after the first fstrim except for a hole from 131081 kB to 131125 kB.

Code:
recovery.log:
warning: wipe_block_device: Wipe via secure discard suppressed due to bug in EMMC firmware
Maybe more interesting is that even though it seems manual mke2fs -t ext4 does a wipe, when I do a fstrim is still says 180MB trimmed, which I don't expect, some inconsistency in fs metadata setup in mkfs maybe? Doesn't matter to me, 1 or 2 wipes, but seems I will be using mke2fs -t ext4 via terminal and then using TWRP wipe/format as required.

FYI, EMMC IDs PNG attached.

>it seems manual mke2fs -t ext4 does a wipe
it shouldn't. mke2fs -i thought- doesn't trim unless invoked with a specific option. repit invokes with the flag:
https://github.com/Lanchon/REPIT/blo...fs-ext4.sh#L51

as to the question of why your markers were cleared anyway, i've no answer.

>even though it seems manual mke2fs -t ext4 does a wipe, when I do a fstrim is still says 180MB trimmed
some kernels inform the newly trimmed space: if you run trim twice you'll get "0 bytes trimmed" the second time.
other kernels (like CM's i9100) always inform (and presumably trim) the full partition free space, so 180MB is ok. if you run trim twice you'll get the same number of bytes both times.

>things are largely back on track after the first fstrim except for a hole from 131081 kB to 131125 kB.
here are some candidate causes:
a) ext4 fs is based on 'blocks' with each block being a constant number of sectors. it makes sense that fstrim should work on whole blocks. also, when ext4 writes a 'short block', ie one not completely filled with data, some sectors and the end of the block might remain unused. it makes sense for ext4 not to write these sectors at all. but the complete block will be marked as used given that the block is the granularity for that, so no sector in the block will be trimmed by fstrim. so some pre-format data might survive in these short blocks after fstrim by this mechanism, which could explain what you observed.
b) another cause could be that the emmc might ignore trim requests for spans shorter than some size or for misaligned spans (remember that erases are performed in large chunks) giving rise to the observed data survival. when formatting no data survives because partitions are position- and size-aligned to 1MB or 4MB, which might be enough to guarantee trim. (i find this explanation unlikely, but who knows.)

about repit:
repit formats and trims ext4 and f2fs in a single operation.

to wipe /data a single repit should be enough:
-data=+wipe

to wipe /sdcard you can take advantage of repit's capability to convert file systems and do two repits:
-sdcard=+wipe+ext4
-sdcard=+wipe

an idea that comes out of this is that repit could temporarily format in ext4 before formatting in other modes that do not support format-time trimming, such as when wiping vfat and swap. this would make the above 2 steps unnecessary.
The Following User Says Thank You to Lanchon For This Useful Post: [ View ] Gift Lanchon Ad-Free
22nd April 2016, 11:08 AM |#15  
Lanchon's Avatar
OP Senior Member
Thanks Meter: 4,447
 
Donate to Me
More
Quote:
Originally Posted by darkz16

[...]
info: disabling swap
BusyBox v1.22.1 bionic (2016-01-10 12:49 +0530) multi-call binary.

Usage: dirname FILENAME

Strip non-directory suffix from FILENAME

info: copying package to '/tmp'
cp: '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' and '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' are the same file

[ERROR 1]

this is very interesting.

the relevant lines are these:
https://github.com/Lanchon/REPIT/blo...t.sh#L318-L322
Code:
    if [ -f "$packageName" ]; then
        if [ "$(dirname $(readlink -f "$packageName"))" != "/tmp" ]; then
            info "copying package to '/tmp'"
            cp -f "$packageName" "/tmp/"
            hint="this package copied itself to '/tmp'; please run it again from there"
the line:
if [ -f "$packageName" ]; then
assures that the file $packageName exists.

but these logfile lines:
Code:
BusyBox v1.22.1 bionic (2016-01-10 12:49 +0530) multi-call binary.

Usage: dirname FILENAME

Strip non-directory suffix from FILENAME
are proof that 'dirname' is being invoked without argument, which means that 'readlink -f "$packageName"' produced an error!

how can 'readlink -f <file>' (which means 'canonicalize') give an error when invoked on an existing file is beyond me. it looks like your implementation of 'readlink' is broken.
what exact recovery are you using? who made it or where is it being distributed?

EDIT:
btw there is a quoting error in the code here but it shouldn't affect this. the line should be:
if [ "$(dirname "$(readlink -f "$packageName")")" != "/tmp" ]; then
22nd April 2016, 11:39 AM |#16  
Lanchon's Avatar
OP Senior Member
Thanks Meter: 4,447
 
Donate to Me
More
Quote:
Originally Posted by darkz16

[...]
cp: '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' and '/tmp/lanchon-repit-20160415-system=1.0-data=6-sdcard=max-preload=min wipe-i9100.zip' are the same file

[ERROR 1]

Whats wrong

so i committed what i think is a fix for this problem that i cant reproduce:
https://github.com/Lanchon/REPIT/com...c7bb9948483c03

and attached is build for you to try.
IMPORTANT: EXTRACT THE ZIP FIRST, DONT JUST FLASH IT!
Attached Files
File Type: zip lanchon-repit-20160422.zip - [Click for QR Code] (801.8 KB, 256 views)
22nd April 2016, 05:53 PM |#17  
Senior Member
Thanks Meter: 18
 
More
Quote:
Originally Posted by Lanchon

so i committed what i think is a fix for this problem that i cant reproduce:
https://github.com/Lanchon/REPIT/com...c7bb9948483c03

and attached is build for you to try.
IMPORTANT: EXTRACT THE ZIP FIRST, DONT JUST FLASH IT!

Hi Lanchon.
I'm installed this zip but i have the same problem.
I'm copy, before zipping the achive, on the \tmp directory of emmc
What have i do wrong?.
Thanks.
22nd April 2016, 07:02 PM |#18  
Lanchon's Avatar
OP Senior Member
Thanks Meter: 4,447
 
Donate to Me
More
Quote:
Originally Posted by chapito

Hi Lanchon.
I'm installed this zip but i have the same problem.
I'm copy, before zipping the achive, on the \tmp directory of emmc
What have i do wrong?.
Thanks.

no log, no kernel info, no recovery info. cant help you.
23rd April 2016, 10:43 AM |#19  
Senior Member
Thanks Meter: 18
 
More
Quote:
Originally Posted by Lanchon

no log, no kernel info, no recovery info. cant help you.

Hi.
Forgive me for not giving more information to solve the problem.
Yesterday I could partition the memory.
I had to change the partition type of memory EMMC to vfat and the script to start.
I used the IsoRec TWRP 2.8.7.0.
I used the last script publicated "lanchon-repit-20160415-system=1.0-data=same-sdcard=max-preload=min+wipe-i9100.zip"
Thanks
The Following User Says Thank You to chapito For This Useful Post: [ View ] Gift chapito Ad-Free
23rd April 2016, 09:31 PM |#20  
Lanchon's Avatar
OP Senior Member
Thanks Meter: 4,447
 
Donate to Me
More
Quote:
Originally Posted by chapito

Hi.
Forgive me for not giving more information to solve the problem.
Yesterday I could partition the memory.
I had to change the partition type of memory EMMC to vfat and the script to start.
I used the IsoRec TWRP 2.8.7.0.
I used the last script publicated "lanchon-repit-20160415-system=1.0-data=same-sdcard=max-preload=min+wipe-i9100.zip"
Thanks

you didn't need to change your sdcard to FAT !!!
you could have used: -sdcard=max++ext4

if only you had read the instructions! that's what they are there for lol
well, it's your data you had to format, not mine! :-p
thanks!
23rd April 2016, 10:59 PM |#21  
Lanchon's Avatar
OP Senior Member
Thanks Meter: 4,447
 
Donate to Me
More
FYI, i created a TWRP issue about the locking behavior that forces us to flash from /tmp:
https://github.com/TeamWin/Team-Win-...ect/issues/664
The Following 3 Users Say Thank You to Lanchon For This Useful Post: [ View ] Gift Lanchon Ad-Free
Post Reply Subscribe to Thread

Guest Quick Reply (no urls or BBcode)
Message:
Previous Thread Next Thread
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes