[TOOL] A QUICK Android OTA payload dumper

Search This thread

ssssut

Member
Sep 7, 2020
5
29
Seoul
Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
See how fast this is: https://i.imgur.com/adpijqf

Source Code: https://github.com/ssut/payload-dumper-go
Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)

Howto:
1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
2. ./payload-dumper-go payload.bin

Notes:
- Incremental OTA payloads are currently not supported but definitely will be in near future.
 

zellleonhart

Senior Member
Jan 19, 2013
579
184
Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
See how fast this is: https://i.imgur.com/adpijqf

Source Code: https://github.com/ssut/payload-dumper-go
Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)

Howto:
1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
2. ./payload-dumper-go payload.bin

Notes:
- Incremental OTA payloads are currently not supported but definitely will be in near future.
Thanks for creating this, I wanted to give it a try on Windows but it came out this error: liblzma-5.dll not found. Do I need to install any per-requisite? Thanks :)

EDIT: managed to get the dll from here https://tukaani.org/xz/ and it's all working nicely.
 
Last edited:

osm0sis

Senior Recognized Developer / Contributor
Mar 14, 2012
14,964
33,919
Halifax
GT-i9250
Google Nexus 4
can i ask how to install liblzma-5 please? in system? in the program?
You just put the .dll in the same directory as the payload-dumper-go .exe; the issue could be pretty easily avoided if it were compiled static.

In fact, I spent some time today figuring out how to static cross-compile payload-dumper-go from my Ubuntu VM to Win32, Linux x86 and armhf, since it's usually better to go for lowest common denominator, and of course having arm since on-device is where payload-dumper-go might be most useful! 🤠

After digging into the recent Docker commit for some hints, then adding stripping and disabling DWARF debugging info generation to have the smallest binary possible, here are my notes for Linux x86:
Bash:
# install latest Go (currently 1.16.2) to /usr/local/go per the Linux instructions at https://golang.org/doc/install
export PATH=$PATH:/usr/local/go/bin

git clone https://github.com/ssut/payload-dumper-go
cd payload-dumper-go

apt-get install liblzma-dev

GOOS=linux GOARCH=386 CGO_ENABLED=1 CC=i686-linux-gnu-gcc go build -a -ldflags '-extldflags "-static -s -w"'
Then, I found that payload-dumper-go's go-xz dependency also in turn being dependent on the toolchain hopefully containing liblzma is extremely problematic/frustrating for Go cross-compiling, but was able to hack the MSYS2 mingw-w64-i686-xz liblzma into the Ubuntu mingw-w64 toolchain to make a static Win32 build:
Bash:
apt-get install mingw-w64
# install include and lib from https://packages.msys2.org/package/mingw-w64-i686-xz to /usr/i686-w64-mingw32

GOOS=windows GOARCH=386 CGO_ENABLED=1 CC=i686-w64-mingw32-gcc go build -a -ldflags '-extldflags "-static -s -w"'
And finally, for Android, NDK gcc wasn't cooperating with `go build` but, since we're building static, Linux armhf will still work fine, but we still need a similar trick to get Ubuntu's own armhf liblzma into the armhf toolchain:
Bash:
apt-get install gcc-arm-linux-gnueabihf
# install include and lib from https://launchpad.net/ubuntu/bionic/armhf/liblzma-dev/5.2.2-1.3 to /usr/arm-linux-gnueabihf

GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build -a -ldflags '-extldflags "-static -s -w"'

I also noticed it doesn't print instructions even though there are some in the code, and have added a PR to fix that: https://github.com/ssut/payload-dumper-go/pull/5

Hopefully @ssssut will still see about adding Incremental OTA support at some point, maybe do something about go-xz to make cross-compiling easier, and ideally add a feature to only dump specific partitions, since extracting the entire payload.bin can be time-consuming (and RAM-consuming!) when all you want is boot.img. 😛

So, without further ado, here are my builds:

[ Attachments removed since they're now superseded by CI releases on GitHub in all major architectures! ]
 
Last edited:

hayvan96

Senior Member
Mar 23, 2018
296
220
Trying this on a SP7 with Oneplus6 firmware -- I get a crash towards the end "panic: Memory allocation failed" (I use the static compiled version by osmosis)
 

osm0sis

Senior Recognized Developer / Contributor
Mar 14, 2012
14,964
33,919
Halifax
GT-i9250
Google Nexus 4
Trying this on a SP7 with Oneplus6 firmware -- I get a crash towards the end "panic: Memory allocation failed" (I use the static compiled version by osmosis)
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
 

hayvan96

Senior Member
Mar 23, 2018
296
220
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
Ok fixed by dumping the xz5.2.5 libs in the same directory (as instructed above) -- but I had to rename libzlma.dll to libzlma-5.dll, maybe this should be added to have a working fix.
 
  • Like
Reactions: osm0sis

osm0sis

Senior Recognized Developer / Contributor
Mar 14, 2012
14,964
33,919
Halifax
GT-i9250
Google Nexus 4
To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract! 🎉🙌

Hopefully @ssssut can make some new official binary release builds (static this time 🤞), and I'll be happy to post some for any architectures not covered and update my Magisk module. 🙂👍
 
Last edited:

luca020400

Recognized Developer
Mar 17, 2014
2,201
5,241
22
Genoa
Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract! 🎉🙌

Hopefully @ssssut can make some new official binary release builds (static this time 🤞), and I'll be happy to post some for any architectures not covered and update my Magisk module. 🙂👍
Man it was using 7GB of RAM here, I had to fix it.
 

osm0sis

Senior Recognized Developer / Contributor
Mar 14, 2012
14,964
33,919
Halifax
GT-i9250
Google Nexus 4
  • Like
Reactions: ipdev and galaxys

Ascii3

Senior Member
Sep 17, 2015
222
64
@ssssut - I tried a few different versions of windows_386 payload-dumper-go up to 1.1.1 on Windows XP Professional with Service Pack 3. Unfortunately, the payload-dumper-go software does not work. When trying to execute payload-dumper-go.exe, I receive the following error message:
[Path the executable]\payload-dumper-go.exe is not a valid Win32 application.
Please update the software so that it may work on Windows XP Professional with Service Pack 3.
 

osm0sis

Senior Recognized Developer / Contributor
Mar 14, 2012
14,964
33,919
Halifax
GT-i9250
Google Nexus 4
Hmm I think it worked fine for me on Windows 10 x86 last I checked, so I guess it just doesn't support XP.. Not sure if there's anything to be done for that.
 

Top Liked Posts

  • There are no posts matching your filters.
  • 29
    Made with Go. By utilizing goroutines, this can extract img files from (full) OTA payload.bin really quickly.
    See how fast this is: https://i.imgur.com/adpijqf

    Source Code: https://github.com/ssut/payload-dumper-go
    Prebuilt binaries: https://github.com/ssut/payload-dumper-go/releases/tag/1.0.0 (for macOS and Windows only)

    Howto:
    1. Copy original image (zip archive or payload.bin) to the same directory as payload-dumper-go exists.
    2. ./payload-dumper-go payload.bin

    Notes:
    - Incremental OTA payloads are currently not supported but definitely will be in near future.
    9
    can i ask how to install liblzma-5 please? in system? in the program?
    You just put the .dll in the same directory as the payload-dumper-go .exe; the issue could be pretty easily avoided if it were compiled static.

    In fact, I spent some time today figuring out how to static cross-compile payload-dumper-go from my Ubuntu VM to Win32, Linux x86 and armhf, since it's usually better to go for lowest common denominator, and of course having arm since on-device is where payload-dumper-go might be most useful! 🤠

    After digging into the recent Docker commit for some hints, then adding stripping and disabling DWARF debugging info generation to have the smallest binary possible, here are my notes for Linux x86:
    Bash:
    # install latest Go (currently 1.16.2) to /usr/local/go per the Linux instructions at https://golang.org/doc/install
    export PATH=$PATH:/usr/local/go/bin
    
    git clone https://github.com/ssut/payload-dumper-go
    cd payload-dumper-go
    
    apt-get install liblzma-dev
    
    GOOS=linux GOARCH=386 CGO_ENABLED=1 CC=i686-linux-gnu-gcc go build -a -ldflags '-extldflags "-static -s -w"'
    Then, I found that payload-dumper-go's go-xz dependency also in turn being dependent on the toolchain hopefully containing liblzma is extremely problematic/frustrating for Go cross-compiling, but was able to hack the MSYS2 mingw-w64-i686-xz liblzma into the Ubuntu mingw-w64 toolchain to make a static Win32 build:
    Bash:
    apt-get install mingw-w64
    # install include and lib from https://packages.msys2.org/package/mingw-w64-i686-xz to /usr/i686-w64-mingw32
    
    GOOS=windows GOARCH=386 CGO_ENABLED=1 CC=i686-w64-mingw32-gcc go build -a -ldflags '-extldflags "-static -s -w"'
    And finally, for Android, NDK gcc wasn't cooperating with `go build` but, since we're building static, Linux armhf will still work fine, but we still need a similar trick to get Ubuntu's own armhf liblzma into the armhf toolchain:
    Bash:
    apt-get install gcc-arm-linux-gnueabihf
    # install include and lib from https://launchpad.net/ubuntu/bionic/armhf/liblzma-dev/5.2.2-1.3 to /usr/arm-linux-gnueabihf
    
    GOOS=linux GOARCH=arm GOARM=7 CGO_ENABLED=1 CC=arm-linux-gnueabihf-gcc go build -a -ldflags '-extldflags "-static -s -w"'

    I also noticed it doesn't print instructions even though there are some in the code, and have added a PR to fix that: https://github.com/ssut/payload-dumper-go/pull/5

    Hopefully @ssssut will still see about adding Incremental OTA support at some point, maybe do something about go-xz to make cross-compiling easier, and ideally add a feature to only dump specific partitions, since extracting the entire payload.bin can be time-consuming (and RAM-consuming!) when all you want is boot.img. 😛

    So, without further ado, here are my builds:

    [ Attachments removed since they're now superseded by CI releases on GitHub in all major architectures! ]
    9
    Made a Magisk module with a wrapper to get the arm build working smoothly on-device: https://forum.xda-developers.com/t/...ices-platforms.2239421/page-149#post-84753275 🤘 :cowboy:
    3
    ideally add a feature to only dump specific partitions
    There is python variant of dumper with this feature (if anyone interested).
    3
    To quote my module post: "Only issue I've seen so far is that on a HUGE payload.bin it can run out of memory and fail to extract the largest partitions, regardless of platform, so I believe that's more of an issue with payload-dumper-go itself than my compiles. It certainly works very well to get boot.img and recovery.img, etc. from a Full OTA quickly. Generally I've had best results extracting on my OnePlus 8T, which is a decently beefy device."
    Looks like @luca020400 and @LuK1337 from Lineage fixed this today and added the feature to select partitions to extract! 🎉🙌

    Hopefully @ssssut can make some new official binary release builds (static this time 🤞), and I'll be happy to post some for any architectures not covered and update my Magisk module. 🙂👍