[TOOL][OTA][Delta Updates] DeltaJen [CyboLabs]

[cybojenix]

What is DeltaJen?

DeltaJen is a tool for quickly making an incremental update based on two update zips. It is simple to use, and easy to extend for devices that don't use a straight up method of flashing a rom.

Why should I use it?

Have you ever hated having to download a full rom just to do a simple update? This tool can take a 200mb rom, take out all the changes, and squish the final package to less than 10mb.

But what if I use CWM instead of TWRP?

No issues here. DeltaJen produces updates that even your stock recovery can apply.

The Geeky parts

The script uses two binaries to efficiently build an update zip. bsdiff and imgdiff. bsdiff can also be replaced by the python module bsdiff4.
No threading or multiprocessing is used currently. as it stands, barely any cpu is used, so threading will be a huge bonus.
Hooks have been added to allow you to easily change how the script adds files, or to add extra lines in the updater-script

How do I use this?

For a straight update zip, which has no extra files or scripts other than what is in /system and boot.img, you run this.
Note that this will change when I update the cli function.

python DeltaJen.py -b <base_zip> -i <new_zip> -o <output_zip>

# Or from python

from DeltaJen import DeltaJen
dj = DeltaJen("<old_zip.zip>", "<new_zip.zip>", "<output_zip.zip>")
dj.generate()

I want to include this in the build steps of a custom rom

I'm working on it for a team already

Requirements

• Linux (untested on OSX but should work)
• python 2.7 or python 3.2+ (all other versions are untested and may not work)
• imgdiff (supplied below or build from source)
• bsdiff (sudo apt-get install bsdiff) or python module bsdiff4 (sudo pip install bsdiff4)
• at least 500mb free ram (sorry, it dumps the files to ram. sorting it out soon).

It does NOT fully support windows yet, due imgdiff not supporting it. You can still run it, but it will only use bsdiff

More detailed instructions will be included soon, and the docs will be improved.

If you want direct help/to contribute/generally chat, come on down to the freenode channel #CyboLabs

last but not leased.
The source

XDA:DevDB Information
DeltaJen, Tool/Utility for all devices (see above for details)

Contributors
cybojenix, @GermainZ

Version Information
Status: Alpha

Created 2014-08-06
Last Updated 2014-08-09

 

[cybojenix]

Reserved

Dependencies

Linux

• Python
• bsdiff or bsdiff4
• imgdiff
• DeltaJen

Windows

• Python
• bsdiff or bsdiff4
• imgdiff
• DeltaJen

System Requirements

• A supported OS
• Enough ram to hold both zips (two 100mb zips would need 200+mb ram)
• Enough storage to hold both zips

 

[dvm84]

Amazing work mate!

I guess that ROM is SlimRoms. Would be glad to see this kinda awesomeness in Slim.

 

[g33kparineeti]

OMG, another @cybojenix project!!! :victory:
Have become your fan lately...

 

[cybojenix]

updates

automatic system and boot info detection has been added, the script is less verbose, and you can execute it directly now.

Documentation still to come

 

[Black_Eyes]

Superb Work Cybojenix
You Rock sir

 

[cybojenix]

kudos to @GermainZ for the massive help with cleaning up documentation.

wiki on github is in progress to aid you all.

 

[Calin]

Woow, nice work man

 

[kl3]

Any chance this project can be combined with CyanDelta for CM or OpenDelta in OmniROM?

It would be really nice if Cyan/OpenDelta downloads the differential update but creates a small update.zip only containing changed files instead of creating the latest complete ROM zip file.

 

[cybojenix]

Any chance this project can be combined with CyanDelta for CM or OpenDelta in OmniROM?

It would be really nice if Cyan/OpenDelta downloads the differential update but creates a small update.zip only containing changed files instead of creating the latest complete ROM zip file.

CyanDelta: I can't comment on because I've seen no documentation from them, nor can I get access to one of their final delta's.

OpenDelta: basically, DeltaJen and OpenDelta have two entirely different methods of working. Omni is wanting a system that will ensure the output is exactly the same every time, and not fail if the system has been modified (a zip is built and the system is flattened before applying)

 

[kantjer]

This looks great.
I did a quick test using my windows system, running the script gave me the following output.

C:\Download\DeltaJen>python DeltaJen.py -b SlimSaberKat-jflte-4.4.4-2
 -i SlimSaberKat-jflte-4.4.4-20140807.zip -o update_zip
WARNING: boot information not supplied.
MTD boot info found
patching system/priv-app/BackupRestoreConfirmation.apk: 1 of 95
Traceback (most recent call last):
  File "DeltaJen.py", line 716, in <module>
    cli()
  File "DeltaJen.py", line 702, in cli
    dj.generate()
  File "DeltaJen.py", line 469, in generate
    self.create_patches(to_diff)
  File "DeltaJen.py", line 378, in create_patches
    p_data = self.compute_diff(b_file, n_file)
  File "DeltaJen.py", line 584, in compute_diff
    if not bs_diff:
NameError: global name 'bs_diff' is not defined

Any tips on what I did wrong

 

[cybojenix]

This looks great.
I did a quick test using my windows system, abd have the following output.

C:\Download\DeltaJen>python DeltaJen.py -b SlimSaberKat-jflte-4.4.4-2
 -i SlimSaberKat-jflte-4.4.4-20140807.zip -o updte_zip
WARNING: boot information not supplied.
MTD boot info found
patching system/priv-app/BackupRestoreConfirmation.apk: 1 of 95
Traceback (most recent call last):
  File "DeltaJen.py", line 716, in <module>
    cli()
  File "DeltaJen.py", line 702, in cli
    dj.generate()
  File "DeltaJen.py", line 469, in generate
    self.create_patches(to_diff)
  File "DeltaJen.py", line 378, in create_patches
    p_data = self.compute_diff(b_file, n_file)
  File "DeltaJen.py", line 584, in compute_diff
    if not bs_diff:
NameError: global name 'bs_diff' is not defined

Any tips on what I did wrong

Ah, line 44 should be 'bs_diff = None'

On windows, you must have the python bsdiff4 installed btw

 

[kantjer]

Ah, line 44 should be 'bs_diff = None'

On windows, you must have the python bsdiff4 installed btw

I have bsdiff4-1.1.4.tar.gz extracted to the folder containing the script and python but that clearly isn't the way to go.

ERROR: python bsdiff4 is required for windows

 

[cybojenix]

I have bsdiff4-1.1.4.tar.gz extracted to the folder containing the script and python but that clearly isn't the way to go.

ERROR: python bsdiff4 is required for windows

you need to compile it as it has c parts.

try this http://www.lfd.uci.edu/~gohlke/pythonlibs/#bsdiff4

 

[SkandaH]

@cybojenix

A good piece of code & nice idea! Just one suggestion - can you please try xdelta (based on VCDIFF) instead of bsdiff?? In MDL, we have a very good result by using it, both in terms of memory & speed. AFAIK, CyanDelta also uses it. If you want a Python wrapper, then see here.

Best of luck!!

 

[cybojenix]

@cybojenix

A good piece of code & nice idea! Just one thought - can you please try xdelta (based on VCDIFF) instead of bsdiff?? In MDL, we have a very good result by using it, both in terms of memory & speed. AFAIK, CyanDelta also uses it. If you want a Python wrapper, then see here.

Best of luck!!

the issue with xdelta is it won't support recovery out the box. I've considered running a script to apply them manually (obviously only to replace bsdiff for now), but I won't be doing it at this stage.

 

[kantjer]

you need to compile it as it has c parts.

try this http://www.lfd.uci.edu/~gohlke/pythonlibs/#bsdiff4

Thanks, but the installer is complaining python 3.4.1 needs to be installed but can't be found in the registry.
I have reinstalled python 3.4.1 using the.msi but still the same error.

 

[cybojenix]

Thanks, but the installer is complaining python 3.4.1 needs to be installed but can't be found in the registry.
I have reinstalled python 3.4.1 using the.msi but still the same error.

what can I say, windows sucks

 

[kantjer]

what can you say, windows sucks

I will try it tomorrow on my linux build machine.
That's where I want to have it running eventually.

 

[DJNoXD]

Great work :good:
Congratulations my friend.