[SOURCES] [DevXchange] Android Sources Highly Compressed & Developer Exchange

regalstreak · Oct 22, 2015

Project abandoned, please do not submit pull requests.

Disclaimer: The wordings in the thread are meant to be noob-friendly and for newer developers that have just dipped their feet into Android platform development. Hence, they won't be accurate in many places for the 'geek'. Also, we had started distributing shallow cloned sources with a depth of '1' instead of non-repo sources (as they are more or less of the same size). So this should not be a problem for the ROM developers in our opinion as they can be fetched by syncing.

Hello!
We, Android Developers, have many problems. One such of them is "SLOW INTERNET" or even "LOW BANDWIDTH".
Due to these problems, we are unable to sync the BIG chunk of Android source codes from the respective repositories.
Hence, most of the new and learning developers just fail at the first step: Downloading The Source Code!
So, here I present you with a thread, which contains links to download "HIGHLY COMPRESSED", untouched (unmodified) and full Android source codes of many different ROMs!
The links to the highly compressed source codes you will find here are not uploaded by me alone. There are many other people that have contributed.
Some of them are: @DigiGoon @MSF Jarvis and @regalstreak (me)

So, Let's begin!
First things first.
You should know what a repo is.
I quote XDA:

What is Repo?
Repo is a repository management tool built on top of Git.

It’s first purpose is to downloads files from multiple git repositories into your local working directory (aka the source tree). That means that you won’t have to manually download or fetch the latest changes from about 300+ projects (depending on if you are downloading AOSP, CM or AOKP for example), repo will do it for you.

The second purpose of repo is to made it easy to submit your code contributions for review to a Gerrit server.

When you download a source code with repo, you get a folder named .repo in your working directory. It contains all the compressed .git format files that need to be in the source code.
After the source has been downloaded, the repo tool decompresses the files from the ".repo" folder and you get the source code in the working directory itself!
This project will give you a highly compressed shallow cloned and full tarball of the sources (for those who can't download it via repo or have limited speeds on github/aosp servers). More information is given in the FAQ section. It is must-read. Without it, you won't understand how to use these.

Note: You cannot go from CM12 source to CM12.1 or going CM12 to CM13 as these are different branches on the CyanogenMod repo. No way doing that. You will have to manually download the new source.

So, We are done with the information!
Now, The interesting part. The sources!

Note: All links on this thread are tap friendly! They open in new tabs

I have divided this into several sections for ease of reading and getting the sources. They are:

regalstreak · Oct 22, 2015

As you know that we have been using the Skadoosh script recently, which is made by me (@regalstreak), @MSF Jarvis and @DigiGoon, all our sources are being uploaded to one common folder on AndroidFileHost.
This is an automated process in which you submit a Pull Request on the Skadoosh's git and the required source will be available to you soon after it.
More information regarding the process is on Post 3 by @MSF Jarvis
The link to the folder where all the sources are uploaded is Here.
The old upload afh folder can be found Here
If you have any other issues regarding the sources, please mention us here. We will be happy to help you. Please read the FAQ before doing stuff.

MSF Jarvis · Oct 22, 2015

Originally, we have had a "Ask on the thread, we'll try" approach to how things work. This is changing now.
While this worked good enough for those asking for the sources, the people who did the actual work were overburdened. Frankly, the process was pretty sketchy. I personally faced the burden part when there were about 6 or 7 ROMs to do! I actually forgot about one guy who made the request earliest, which kinda sucked.
Going forward, this won't happen now. We have a script up on @regalstreak's Github. Updates to it are synced to my server and executed.

The process for requesting sources is now something like this :

Go to the Github repo and edit the compress.bash script
Start a Pull Request
Wait for it to be merged
Wait for even more time to let it upload and one of us to send the confirmation here.
Do whatever the heck you want with those sources!

NOTE: The compress.bash looks like this in a classic example of how to do it WRONG.

Code:

name=AOSP
manifest=[url]https://android.googlesource.com/platform/manifest[/url] -b android-4.0.1_r1
branch=android-6.0.1_r55

./skadoo.sh $name $manifest $branch

This was after This PR was merged and is being shown as the perfect way to mess it up. You are ALREADY passing the branch parameter separately, so adding '-b android-4.0.1_r1' looks like a shining example of blind copy paste. Don't do that. Change the name, manifest and branch with some general application of the faculty of common sense.

Interested users are also suggested to read this strongly worded post.

regalstreak · Oct 22, 2015

FAQ

Q1> How should I extract the source?
Ans:

First you will have to join the archive parts into one big archive. Download all the parts. Then put all parts in one folder. Verify md5sum by

Code:

md5sum -c *.md5sum

Then do

Code:

cat *.tar.* > yourbigarchivename.tar.xz

Just quote me if you don't understand this. I'll help you out.

If your archive is in .tar.xz format, do this:

Open Terminal
Now cd to the drive where you have downloaded the sources
Extract it
Code:
```
tar -xvJf (file name here).tar.xz
```

If your archive is in .tar.gz format, so do this:

Open Terminal
Now cd to the drive where you have downloaded the sources
Extract it
Code:
```
tar -xvzf (file name here).tar.gz
```

Now you have extracted the source.

Q2> I am not able to extract the source. What should I do?
Ans: You cannot extract source with GUI on the Archive Extractor. So, try the method I mentioned.

Q3> Is the size of all files 2GB? I saw the size 2GB on AndroidFileHost!
Ans: The mentioned size on AFH is incorrect. AndroidFileHost has some problems while showing file sizes for files above 2GB. You need to click on Download once to view the correct size!

Q4> How do you compress them? I mean, I need the command so that I can help you out!
Ans: Our script is open source. You can view it here: https://github.com/regalstreak/skadoosh

Q5> What are shallow and full sources?
Ans: Shallow sources are the ones that just have the latest or the HEAD commits in each subproject. You can read more about it from here. Full are the ones that have all the bits from the starting of the subprojects and hence the archives have a large file size.

Q6> How do I use these sources?
Ans: First, you will have to extract them as mentioned above. You should get a .repo folder. In the same folder itself (not inside .repo but the folder that contains .repo), you will have to update the local work tree. You can update it by using

Code:

repo sync -c -f --force-sync --no-clone-bundle --no-tags -j8

This command will help you to use minimum data/internet as it does not download extra tags or clone bundles.
If you do not want to use data at all, you can use this:

Code:

repo sync -c -f --force-sync --no-clone-bundle --no-tags -j8 --local-only

This will not fetch anything from the internet. Happy compiling!

Please press the "Thanks Button" if we helped you and quote us if you have a problem!

regalstreak · Oct 22, 2015

Announcements and General Stuff

Automation also has been made opensource now. You can view the source of the automation stuff at https://github.com/regalstreak/skadhook.git

All status of the sources is here: https://github.com/regalstreak/skadoosh/pulls?q=is:pr+is:closed

regalstreak · Oct 23, 2015

Reserved

Reserved for stuff.

FireLord · Oct 23, 2015

Lovely

Thanks mate..... :laugh:

MSF Jarvis · Oct 24, 2015

What if it's in tar.xz @regalstreak, most people including @DigiGoon use it for its extreme compression.

Sent from a Cool Phone stuck with crappy KingUser

regalstreak · Oct 25, 2015

Replies.. Replies only!

FireLord said:
Thanks mate..... :laugh:

No problemo bruh!

MSF Jarvis said:
What if it's in tar.xz @regalstreak, most people including @DigiGoon use it for its extreme compression.

Sent from a Cool Phone stuck with crappy KingUser

I have mentioned the steps in the FAQ! Please see it

AsadP2013 · Oct 25, 2015

Wow amazing work!

regalstreak · Oct 25, 2015

AsadP2013 said:
Wow amazing work!

Thanks!

Hope it helps!

sumit yadav · Oct 26, 2015

MSF Jarvis said:
What if it's in tar.xz @regalstreak, most people including @DigiGoon use it for its extreme compression.

Sent from a Cool Phone stuck with crappy KingUser

Use tar -xJf file_name.tar.xz to extract *tar.xz....:good:

---------- Post added at 11:48 PM ---------- Previous post was at 11:44 PM ----------

regalstreak said:
FAQ

Q1> How should I extract the source?
Ans: If your archive is in .7z format, do this:

Open Terminal

Download 7zip if not done before (sudo apt-get install p7zip-full)

Now cd to the drive where you have downloaded the sources

7z x (file name here).7z

If your archive is in .tar.gz format, do this:

Open Terminal

Now cd to the drive where you have downloaded the sources

tar -xvzf (file name here).tar.gz

Now you have extracted the source.

Q2> I am not able to extract the source. What should I do?
Ans: You cannot extract source with GUI on the Archive Extractor. So, try the method I mentioned.

Q3> Is the size of all files 2GB? I saw the size 2GB on Androidfilehost!
Ans: The mentioned size on the thread is correct. AndroidFileHost has some problems while showing file sizes for files above 2GB.

Please press the "Thanks Button" if I helped you and quote me if you have a problem!

Hey, first of all....A BIG Thanks to you guys, you guys are awesome and keeping the spirit of xda to help your friends......
A request, Add the method to extract *.tar.xz as "tar -xJf file_name.tar.xz"

regalstreak · Oct 26, 2015

sumit yadav said:
Use tar -xJf file_name.tar.xz to extract *tar.xz....:good:

---------- Post added at 11:48 PM ---------- Previous post was at 11:44 PM ----------

Hey, first of all....A BIG Thanks to you guys, you guys are awesome and keeping the spirit of xda to help your friends......
A request, Add the method to extract *.tar.xz as "tar -xJf file_name.tar.xz"

Im glad that this index is helping youll alot! Yea sure.. Ill add that now!

adil089 · Oct 26, 2015

Wow thanks man I'm really eager to get started but my internet speed had always let me down. But thanks to you guys I can finally start to build my own ROM. Thanks again guys much appreciated.

regalstreak · Oct 26, 2015

adil089 said:
Wow thanks man I'm really eager to get started but my internet speed had always let me down. But thanks to you guys I can finally start to build my own ROM. Thanks again guys much appreciated.

Me faced the same problems! But now i have managed to get a better isp [emoji14] happy this helped you!

(One req: Please spread this thread so people may know that highly compressed sources exist and they can even start with a bad internet connection!) [emoji1]

adil089 · Oct 26, 2015

regalstreak said:
Me faced the same problems! But now i have managed to get a better isp [emoji14] happy this helped you!

(One req: Please spread this thread so people may know that highly compressed sources exist and they can even start with a bad internet connection!) [emoji1]

I will definitely spread this thread because I know a few people who may need it. It's the least that I can do. Thanks.

regalstreak · Oct 28, 2015

@Perseus should this thread be moved to Misc android development? So more people can see it? Or is it fine here...

regalstreak · Nov 1, 2015

The thread will be updated from nov6th.. Got exams [emoji14] Please hold on!

adil089 · Nov 1, 2015

regalstreak said:
The thread will be updated from nov6th.. Got exams [emoji14] Please hold on!

Thanks man. Good look on your exams.

PuLKit4xd · Nov 3, 2015

Thank you very much!
I always wanted to build a ROM but 18GB downloads always stopped me to do so. This is amazing!
However it could have been a little better if the archives were broken into parts so that they could be downloaded on two or more systems and on FAT32 SDcards (4GB max file limit).

Well done!

[SOURCES] [DevXchange] Android Sources Highly Compressed & Developer Exchange

Senior Member

Senior Member

XDA Feed Mod/Inactive RC

Senior Member

Senior Member

Senior Member

Inactive Recognized Developer

XDA Feed Mod/Inactive RC

Senior Member

Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Senior Member

Similar threads

Top Liked Posts