[BUILD][6.0.1][CAF][LA.HB.1.3.1] How to build Marshmallow using Arch Linux

[namanjr]

System Requirements
Arch Linux x86-64
Oxygen OS(extracted image or installed)
adb installed and drivers setup
non-root user
sudo setup for your user (you can install packages as root as a workaround)
40GB free storage
4GB+ ram

Required Packages
First off you're gonna want to make sure you're completely up to date and fetch the packages required to build, with the command below.

#sudo pacman -Syyu gcc-multilib git gnupg flex bison gperf sdl wxgtk squashfs-tools curl ncurses zlib schedtool perl-switch zip unzip libxslt python2-virtualenv bc lib32-zlib lib32-ncurses lib32-readline  jdk7-openjdk rsync maven repo

Then you're going to need to fetch some apps from the AUR. First add their GPG key.

#gpg --keyserver pgp.mit.edu --recv-keys C52048C0C0748FEE227D47A2702353E0F7E48EDB

Now you can fetch them with a tool like yaourt or manually install the two packages.
lib32-ncurses5-compat-libs
ncurses5-compat-libs

#yaourt --aur lib32-ncurses5-compat-libs ncurses5-compat-libs

Setting up build enviroment and syncing
This part will assume you're building in a folder called "android" in your home directory, and requires the above packages to be installed.

Run these commands to install repo in a folder called "bin" in your home directory. Then for a single terminal session set the new "bin" folder as a binary folder in bash.

#mkdir ~/bin
#export PATH=~/bin:$PATH
#curl https://storage.googleapis.com/git-repo-downloads/repo > ~/bin/repo
#chmod a+x ~/bin/repo

Make a folder to store and build android, then enter it.

#mkdir ~/android
#cd ~/android

Setup a virtual python 2.7 enviroment in your "android" folder as repo doesn't work with 3+.

#virtualenv2 venv
#ln -s /usr/lib/python2.7/* ~/android/venv/lib/python2.7/
#source venv/bin/activate

Now it's time to sync the sourcecode with your python 2.7 and repo setup. The following code will sync CAF android 6.0.1 from oneplus. The "4" can be removed or replaced with any whole number becides zero depending on your connection speed. You may omit the "-c" if you have problems syncing or need more than the defined branches.

#repo init -u https://github.com/OnePlusOSS/android.git -b oneplus3T/6.0.1
#repo sync -j4 -c

You can also use my repo if you prefer. I'm working on improving the mess oneplus gave us.

#repo init -u https://github.com/droidman/android.git -b oneplus3T/6.0.1
#repo sync -j4 -c

Compile Your Build
Assuming your in your virtual python 2.7 and followed the first part of the guide, set openjdk 7 as your active java.

#export JAVA_HOME=/usr/lib/jvm/java-7-openjdk

Fix broken provided glib.

cp /usr/bin/ld.gold prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin/ld

If you're rooted and have adb installed, you may want to fetch your prebuilt files from the latest marshallow Oxygen OS or caf based rom. You may also extract them from a Oxygen OS system image but build will not complete without prebuilts.

#cp ~/android/.repo/manifests/pull_library.sh ~/android/pull.sh
#sh pull.sh

Intialize the enviroment and select a platform. You may build "msm8996-userdebug" or "msm8996-user".

#source build/envsetup.sh
#lunch msm8996-userdebug

Time to build! Find the number of threads on your cpu and set 1-2x that as "x". You may also omit "OTAackage" if you prefer to have fastboot/twrp flashable images instead of a zip, (system.img, boot.img, etc).

#make -jx OTApackage

Your compiled files and flashable zip or images will be located in "~/android/out/target/product/msm8996".

How to rebuild!
Cd into directory, sync repo, and setup enviroment.

#export PATH=~/bin:$PATH
#export JAVA_HOME=/usr/lib/jvm/java-7-openjdk
#cd ~/android
#source venv/bin/activate
#repo sync -j4 -c
#source build/envsetup.sh
#lunch msm8996-userdebug

Clean the out directory.

#make clobber

Build. (setup to make a recovery flashable zip with a 8 thread processor.)

#make -j16 otapackage

 

[mgbotoe]

Thanks for the clean guide

 

[hachamacha]

Very nice and thanks for posting it. If you'd rather use whatever you've got installed on a box or VM (to pick one out of a hat, U16.04) , just google "how to build marshmallow {from your distro}". (where you fill in your distro's name & version (opt)).

For example, initial parts of "this page(see link)" might be good for 16.04 LTS users :: http://forum.xda-developers.com/chef-central/android/guide-how-to-setup-ubuntu-16-04-lts-t3363669

All the commands following the basic setup of development environment in the OP "should" work on any distro, starting with the fetch from git. Also, thanks for your personal github code.

Anyway, good job. Cheers.

 

[hachamacha]

I've synced your git version of oneplus' git. The problem I've having and not getting past looks like this: Oops, I guess I lost the clipboard from the VM. It was basically as if it hasn't got the dashd prebuilt or blob or whatever it was looking for. I grabbed the prebuilts via my phone running 3.5.3 which seemed to work fine.

Is your current github code building without trouble for you? I should probably re-sync. Thanks.

 

[namanjr]

I've synced your git version of oneplus' git. The problem I've having and not getting past looks like this: Oops, I guess I lost the clipboard from the VM. It was basically as if it hasn't got the dashd prebuilt or blob or whatever it was looking for. I grabbed the prebuilts via my phone running 3.5.3 which seemed to work fine.

Is your current github code building without trouble for you? I should probably re-sync. Thanks.

The prebuilt script is currently broken, i manually copied over the files that failed with a file manager. You can also mount a system.img from a compiled oxygenOS, Oneplus Open Source or AOOP as ext4 to manually copy over the files too!

 

[hachamacha]

The prebuilt script is currently broken, i manually copied over the files that failed with a file manager. You can also mount a system.img from a compiled oxygenOS, Oneplus Open Source or AOOP as ext4 to manually copy over the files too!

Oh cool. I can do that (the system.img). My question then is "where are the files on in the mounted system.img" and "do they go to "prebuilts" at top level or to "vendor" (which level, since I see vendor repeat two levels down)?

Thanks.

 

[namanjr]

Oh cool. I can do that (the system.img). My question then is "where are the files on in the mounted system.img" and "do they go to "prebuilts" at top level or to "vendor" (which level, since I see vendor repeat two levels down)?

Thanks.

I just extracted the entire root of the system.img to "SOURCECODE/vendor/oneplus", easier than manually going through to see what files to copy over!

 

[hachamacha]

I just extracted the entire root of the system.img to "SOURCECODE/vendor/oneplus", easier than manually going through to see what files to copy over!

Great. Thanks so much for the help. I'll give that a shot next.

---------- Post added at 11:01 AM ---------- Previous post was at 10:25 AM ----------

Great. Thanks so much for the help. I'll give that a shot next.

OK, that worked. I just had to , for some reason, use the samsung simg2img (sparse to full) util in order to mount it as ext4/loop.

Thanks again.

 

[kenzolo]

edit: solved

 

[hachamacha]

Now that I can do a make -j {target}, I've got another question:

The target OTApackage (or is it otapackage) doesn't appear to be a valid choice for me. I get a "Target doesn't exist" error if I use make -j{N} otapackage (or OTApackage). I'd like very much to be able to create the install .zip rather than have to collect & bundle all the bits & pieces. I've tried with the original OnePlus .git and your .git as well.

Thanks much.

 

[namanjr]

Now that I can do a make -j {target}, I've got another question:

The target OTApackage (or is it otapackage) doesn't appear to be a valid choice for me. I get a "Target doesn't exist" error if I use make -j{N} otapackage (or OTApackage). I'd like very much to be able to create the install .zip rather than have to collect & bundle all the bits & pieces. I've tried with the original OnePlus .git and your .git as well.

Thanks much.

Did you setup the enviroment and run lunch? It's all lowercase! Can you post the error it gives?

 

[hachamacha]

Did you setup the enviroment and run lunch? It's all lowercase! Can you post the error it gives?

Oh yeah: On the same VM, I can build versions of CM, no troubles. I've got OpenJava 7 JDK and all the right stuff. I ran lunch (It's something like either #25 or 26 on the menu, and no problems with any of that.

After quite awhile, it comes back with the "no rule to build target otapackage (whatever it was)" . I'll run a build right now and get the exact error:

Jeez! Of course, this is the time it picks to just go right past where it usually gets the error. I'm pretty sure I know what happened. I was using it as you'd printed it in the guide ^^, OTApackage, which in a case sensitive system , doesn't mean a thing. I'll post again if it dies 2 hours down the line. I should move this operation to my Linux box with SSD's but it's in use at the moment.

Thanks again.

edit: Lunch output ::

$lunch msm8996-userdebug

============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=6.0.1
TARGET_PRODUCT=msm8996
TARGET_BUILD_VARIANT=userdebug
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm64
TARGET_ARCH_VARIANT=armv8-a
TARGET_CPU_VARIANT=kryo
TARGET_2ND_ARCH=arm
TARGET_2ND_ARCH_VARIANT=armv7-a-neon
TARGET_2ND_CPU_VARIANT=cortex-a53
HOST_ARCH=x86_64
HOST_OS=linux
HOST_OS_EXTRA=Linux-4.4.0-51-generic-x86_64-with-Ubuntu-16.04-xenial
HOST_BUILD_TYPE=release
BUILD_ID=MXB48T
OUT_DIR=out

Damn. I just realized you had this covered as well, but I missed it first time around.

My compile finally blew out about 1/2 way through with a libart.so build error, so I did this (before I re-read your guide):

$ln -s prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/bin/ld /usr/bin/ld.gold

And we're off to the next error! ;

 

[hamzahrmalik]

I believe someone has already mentioned a problem with dashd not being found, im getting the "no rule to make target..." error regarding dashd, tried to pull /sbin/dashd but the file doesn't exist. What can I do

 

[hachamacha]

I believe someone has already mentioned a problem with dashd not being found, im getting the "no rule to make target..." error regarding dashd, tried to pull /sbin/dashd but the file doesn't exist. What can I do

Did you unpack system.img from 3.5.3 (you can check the disaster recovery thread in q&a here) and then copy the entire contents into $source/vendor/oneplus? Before I had just used a cable and adb enabled in Linux and I died at dashd as well. After I got the system.img prebuilts, it was fine.

Note: My experience:: System.img wasn't mountable as a $mount -t ext4 -o loop ... device , not recognized as that type file, so I found simg2img (For whatever reason, Samsung uses "sparse" image files and apparently so does OP) to modify system.img to a normal non-sparse system.img and then mount -t ext4 -o loop ./system.img /{mountpoint} worked fine and I copied from there to ./vendor/OnePlus/ (however it's spelled , it's the only "oneplus" folder under vendor.

 

[hamzahrmalik]

Did you unpack system.img from 3.5.3 (you can check the disaster recovery thread in q&a here) and then copy the entire contents into $source/vendor/oneplus? Before I had just used a cable and adb enabled in Linux and I died at dashd as well. After I got the system.img prebuilts, it was fine.

Note: My experience:: System.img wasn't mountable as a $mount -t ext4 -o loop ... device , not recognized as that type file, so I found simg2img (For whatever reason, Samsung uses "sparse" image files and apparently so does OP) to modify system.img to a normal non-sparse system.img and then mount -t ext4 -o loop ./system.img /{mountpoint} worked fine and I copied from there to ./vendor/OnePlus/ (however it's spelled , it's the only "oneplus" folder under vendor.

Thanks. Can't find the thread you are referring to, so I just downloaded a 3.5.3 rom zip and I'll see what I can find

Sent from my ONEPLUS A3003 using Tapatalk

 

[hachamacha]

Thanks. Can't find the thread you are referring to, so I just downloaded a 3.5.3 rom zip and I'll see what I can find

Sent from my ONEPLUS A3003 using Tapatalk

Sorry: I couldn't find it for a moment either. Here it is and if you get the .zip file in the thread that the OP user has on a torrent, then you can just unzip and use any of the 3.5.3 .img files. The other .zip files all use the newer .zip style of patching every file from things like system.img.new, etc.

http://forum.xda-developers.com/oneplus-3t/help/oneplus-3t-factory-restore-utility-t3511287

That's the thread.

 

[hamzahrmalik]

Sorry: I couldn't find it for a moment either. Here it is and if you get the .zip file in the thread that the OP user has on a torrent, then you can just unzip and use any of the 3.5.3 .img files. The other .zip files all use the newer .zip style of patching every file from things like system.img.new, etc.

http://forum.xda-developers.com/oneplus-3t/help/oneplus-3t-factory-restore-utility-t3511287

That's the thread.

Finally got that downloaded. How do i mount? I tried mount -t ext4 -o loop ./system.img but that doesnt work. You said something about covnerting to non-sparse, how do i do that?

Thanks

 

[hachamacha]

Finally got that downloaded. How do i mount? I tried mount -t ext4 -o loop ./system.img but that doesnt work. You said something about covnerting to non-sparse, how do i do that?

Thanks

There's a linux util called simg2img (or I think it means (sparse image to full image). Then you can do the mount you just tried above. It fails on the 'sparse' image. I'm not sure why the hell OP is using Samsung's sparse image but maybe it's more common than I know. I just happened upon it and it was installed on my version of Ubuntu from when I was building CMNN.N versions for other phones.

I can upload it if you like (or try). OK: I booted linux and zipped it so unzip simg2img.zip >> /usr/bin (or whatever you're using as an executable path).

 

[hamzahrmalik]

There's a linux util called simg2img (or I think it means (sparse image to full image). Then you can do the mount you just tried above. It fails on the 'sparse' image. I'm not sure why the hell OP is using Samsung's sparse image but maybe it's more common than I know. I just happened upon it and it was installed on my version of Ubuntu from when I was building CMNN.N versions for other phones.

I can upload it if you like (or try). OK: I booted linux and zipped it so unzip simg2img.zip >> /usr/bin (or whatever you're using as an executable path).

Thank you very much, I'll give that a go

Sent from my ONEPLUS A3003 using Tapatalk

 

[hachamacha]

I'm curious as to what 'builders' think is the best solution (assuming you have this problem, or this type of problem on other modules) :

Problem:: (can't build libart.so) (sorry, don't have the error anymore but it has to do with ./prebuilts/*/*/.../*/ld not pointing to a version that is in sync with what's going on to the tree.

A solution(s): ln -s /usr/bin/ld.gold ./prebuilts/gcc/linux-x86/host/x86_64-linux-glibc2.15-4.8/x86_64-linux/ld
(why?) - I found this as a workaround in a) the original OP, b) the changesets for the problem in CM jerritt. Also setting env var "WITHOUT_HOST_CLANG=TRUE". I have yet to find a decent answer for why this might be working.


Currently: I can build the thing, but there are ~23 or so occurrences of ./ld in the ./prebuilts tree. Is that really the best solution , changing each of these things to be soft pointers to /usr/bin/ld.gold?

If anyone has a better answer (or a real answer), would love to hear it. Yes, it builds, but the reason this seems so hokey is because if you did, for example, a "make clobber" before a build to clean out the trash, you'd be back at square one. Of course, you could make it part of a script..

edit: gerritt refs: (they are quite "old" so the problems been around in CM for quite awhile. We're just building stock here.

https://review.cyanogenmod.org/#/c/120824/
https://android-review.googlesource.com/#/c/223100/

edit2: Return of --version or -v from /usr/bin/ld, ld.gold to see the difference:

/usr/bin/ld -v
GNU ld (GNU Binutils for Ubuntu) 2.26.1
[email protected]:~/sandbox/op3t$ /usr/bin/ld.gold -v
GNU gold (GNU Binutils for Ubuntu 2.26.1) 1.11

Edit 3: Explanation of ld.gold:

http://llvm.org/docs/GoldPlugin.html