• Introducing XDA Computing: Discussion zones for Hardware, Software, and more!    Check it out!

[TOOL][OTA][Delta Updates] DeltaJen [CyboLabs]

Search This thread

cybojenix

Retired Forum Mod / Retired Recognized Developer
Apr 7, 2012
801
1,864
What is DeltaJen?
DeltaJen is a tool for quickly making an incremental update based on two update zips. It is simple to use, and easy to extend for devices that don't use a straight up method of flashing a rom.

Why should I use it?
Have you ever hated having to download a full rom just to do a simple update? This tool can take a 200mb rom, take out all the changes, and squish the final package to less than 10mb.

But what if I use CWM instead of TWRP?
No issues here. DeltaJen produces updates that even your stock recovery can apply.

The Geeky parts

The script uses two binaries to efficiently build an update zip. bsdiff and imgdiff. bsdiff can also be replaced by the python module bsdiff4.
No threading or multiprocessing is used currently. as it stands, barely any cpu is used, so threading will be a huge bonus.
Hooks have been added to allow you to easily change how the script adds files, or to add extra lines in the updater-script


How do I use this?
For a straight update zip, which has no extra files or scripts other than what is in /system and boot.img, you run this.
Note that this will change when I update the cli function.
Code:
python DeltaJen.py -b <base_zip> -i <new_zip> -o <output_zip>

# Or from python

from DeltaJen import DeltaJen
dj = DeltaJen("<old_zip.zip>", "<new_zip.zip>", "<output_zip.zip>")
dj.generate()

I want to include this in the build steps of a custom rom
I'm working on it for a team already

Requirements
  • Linux (untested on OSX but should work)
  • python 2.7 or python 3.2+ (all other versions are untested and may not work)
  • imgdiff (supplied below or build from source)
  • bsdiff (sudo apt-get install bsdiff) or python module bsdiff4 (sudo pip install bsdiff4)
  • at least 500mb free ram (sorry, it dumps the files to ram. sorting it out soon).

It does NOT fully support windows yet, due imgdiff not supporting it. You can still run it, but it will only use bsdiff

More detailed instructions will be included soon, and the docs will be improved.

If you want direct help/to contribute/generally chat, come on down to the freenode channel #CyboLabs

last but not leased.
The source

XDA:DevDB Information
DeltaJen, Tool/Utility for all devices (see above for details)

Contributors
cybojenix, @GermainZ

Version Information
Status: Alpha

Created 2014-08-06
Last Updated 2014-08-09
 
Last edited:

cybojenix

Retired Forum Mod / Retired Recognized Developer
Apr 7, 2012
801
1,864
updates

automatic system and boot info detection has been added, the script is less verbose, and you can execute it directly now.

Documentation still to come
 

kl3

Senior Member
Sep 9, 2013
145
175
Any chance this project can be combined with CyanDelta for CM or OpenDelta in OmniROM?

It would be really nice if Cyan/OpenDelta downloads the differential update but creates a small update.zip only containing changed files instead of creating the latest complete ROM zip file.
 

cybojenix

Retired Forum Mod / Retired Recognized Developer
Apr 7, 2012
801
1,864
Any chance this project can be combined with CyanDelta for CM or OpenDelta in OmniROM?

It would be really nice if Cyan/OpenDelta downloads the differential update but creates a small update.zip only containing changed files instead of creating the latest complete ROM zip file.

CyanDelta: I can't comment on because I've seen no documentation from them, nor can I get access to one of their final delta's.

OpenDelta: basically, DeltaJen and OpenDelta have two entirely different methods of working. Omni is wanting a system that will ensure the output is exactly the same every time, and not fail if the system has been modified (a zip is built and the system is flattened before applying)
 
  • Like
Reactions: nimbosa

kantjer

Inactive Recognized Contributor
Dec 11, 2010
12,393
29,848
www.androidfilehost.com
This looks great.
I did a quick test using my windows system, running the script gave me the following output.

Code:
C:\Download\DeltaJen>python DeltaJen.py -b SlimSaberKat-jflte-4.4.4-2
 -i SlimSaberKat-jflte-4.4.4-20140807.zip -o update_zip
WARNING: boot information not supplied.
MTD boot info found
patching system/priv-app/BackupRestoreConfirmation.apk: 1 of 95
Traceback (most recent call last):
  File "DeltaJen.py", line 716, in <module>
    cli()
  File "DeltaJen.py", line 702, in cli
    dj.generate()
  File "DeltaJen.py", line 469, in generate
    self.create_patches(to_diff)
  File "DeltaJen.py", line 378, in create_patches
    p_data = self.compute_diff(b_file, n_file)
  File "DeltaJen.py", line 584, in compute_diff
    if not bs_diff:
NameError: global name 'bs_diff' is not defined

Any tips on what I did wrong
 
Last edited:

cybojenix

Retired Forum Mod / Retired Recognized Developer
Apr 7, 2012
801
1,864
This looks great.
I did a quick test using my windows system, abd have the following output.

Code:
C:\Download\DeltaJen>python DeltaJen.py -b SlimSaberKat-jflte-4.4.4-2
 -i SlimSaberKat-jflte-4.4.4-20140807.zip -o updte_zip
WARNING: boot information not supplied.
MTD boot info found
patching system/priv-app/BackupRestoreConfirmation.apk: 1 of 95
Traceback (most recent call last):
  File "DeltaJen.py", line 716, in <module>
    cli()
  File "DeltaJen.py", line 702, in cli
    dj.generate()
  File "DeltaJen.py", line 469, in generate
    self.create_patches(to_diff)
  File "DeltaJen.py", line 378, in create_patches
    p_data = self.compute_diff(b_file, n_file)
  File "DeltaJen.py", line 584, in compute_diff
    if not bs_diff:
NameError: global name 'bs_diff' is not defined

Any tips on what I did wrong

Ah, line 44 should be 'bs_diff = None'

On windows, you must have the python bsdiff4 installed btw
 
  • Like
Reactions: kantjer

SkandaH

XDA Portal Supporter
Jun 8, 2013
972
2,572
@cybojenix

A good piece of code & nice idea! Just one suggestion - can you please try xdelta (based on VCDIFF) instead of bsdiff?? In MDL, we have a very good result by using it, both in terms of memory & speed. AFAIK, CyanDelta also uses it. If you want a Python wrapper, then see here.

Best of luck!!
 

cybojenix

Retired Forum Mod / Retired Recognized Developer
Apr 7, 2012
801
1,864
@cybojenix

A good piece of code & nice idea! Just one thought - can you please try xdelta (based on VCDIFF) instead of bsdiff?? In MDL, we have a very good result by using it, both in terms of memory & speed. AFAIK, CyanDelta also uses it. If you want a Python wrapper, then see here.

Best of luck!!

the issue with xdelta is it won't support recovery out the box. I've considered running a script to apply them manually (obviously only to replace bsdiff for now), but I won't be doing it at this stage.
 
  • Like
Reactions: Titokhan

Top Liked Posts

  • There are no posts matching your filters.
  • 44
    What is DeltaJen?
    DeltaJen is a tool for quickly making an incremental update based on two update zips. It is simple to use, and easy to extend for devices that don't use a straight up method of flashing a rom.

    Why should I use it?
    Have you ever hated having to download a full rom just to do a simple update? This tool can take a 200mb rom, take out all the changes, and squish the final package to less than 10mb.

    But what if I use CWM instead of TWRP?
    No issues here. DeltaJen produces updates that even your stock recovery can apply.

    The Geeky parts

    The script uses two binaries to efficiently build an update zip. bsdiff and imgdiff. bsdiff can also be replaced by the python module bsdiff4.
    No threading or multiprocessing is used currently. as it stands, barely any cpu is used, so threading will be a huge bonus.
    Hooks have been added to allow you to easily change how the script adds files, or to add extra lines in the updater-script


    How do I use this?
    For a straight update zip, which has no extra files or scripts other than what is in /system and boot.img, you run this.
    Note that this will change when I update the cli function.
    Code:
    python DeltaJen.py -b <base_zip> -i <new_zip> -o <output_zip>
    
    # Or from python
    
    from DeltaJen import DeltaJen
    dj = DeltaJen("<old_zip.zip>", "<new_zip.zip>", "<output_zip.zip>")
    dj.generate()

    I want to include this in the build steps of a custom rom
    I'm working on it for a team already

    Requirements
    • Linux (untested on OSX but should work)
    • python 2.7 or python 3.2+ (all other versions are untested and may not work)
    • imgdiff (supplied below or build from source)
    • bsdiff (sudo apt-get install bsdiff) or python module bsdiff4 (sudo pip install bsdiff4)
    • at least 500mb free ram (sorry, it dumps the files to ram. sorting it out soon).

    It does NOT fully support windows yet, due imgdiff not supporting it. You can still run it, but it will only use bsdiff

    More detailed instructions will be included soon, and the docs will be improved.

    If you want direct help/to contribute/generally chat, come on down to the freenode channel #CyboLabs

    last but not leased.
    The source

    XDA:DevDB Information
    DeltaJen, Tool/Utility for all devices (see above for details)

    Contributors
    cybojenix, @GermainZ

    Version Information
    Status: Alpha

    Created 2014-08-06
    Last Updated 2014-08-09
    11
    Reserved

    Dependencies

    Linux
    Windows

    System Requirements
    • A supported OS
    • Enough ram to hold both zips (two 100mb zips would need 200+mb ram)
    • Enough storage to hold both zips
    4
    aannnddd gapps survivable updates incoming ;)

    it's only a rough list, so please if you find any files that I missed, let me know

    https://github.com/CyboLabs/DeltaJen/commit/6e985feaa4ffff21e2f0c2c2f922a7dd45b73802
    3
    tmp file handling updated to work on windows and linux vm's (which solves the above errors).
    Thank you @kantjer for bringing up the linux vm issue.

    also fixed verbose mode in windows and vm's.


    for reference to other people working with tmp files and python, check this commit
    https://github.com/CyboLabs/DeltaJen/commit/be4b5b53712fc5648160dbcbf3de6e3d6f0916c0
    2
    May I suggest jojodiff?

    In my testing it blows away both xdelta and bsdiff

    bsdiff gives better compression on libs, and no differ will match imgdiff due to the way it works