[SCRIPTSET v2.6][02/28/2012] Build native GNU/Linux applications: the easy way

New: SANE (USB Scanner backend and drivers) - OTG equipped device needed to work for this - YES, Scanner Access Now possible and Easy with your phone

This can (in combination with tesseract-ocr) be very useful to recognize text from scanned images.

For pictures see also http://xdaforums.com/showpost.php?p=23046120&postcount=187

Update 01/30/2012: Now with a small android app (ScriptStarter) to start services and other shell scripts easily from the GUI. See bottom of the post.


WHAT'S POSSIBLE WITH THIS SCRIPT:

With this bash script your are able to build relatively easy from source a lot of native software (over 90 at the moment) from the GNU/Linux community for your Android phone and you will have
the good experience with free software.
All these tools and programs are not "apps for the GUI" for Android but native running command line tools "under the surface" of your Android GUI like the kernel and other services.
Don't ask me if it makes any sense to install program xy (like MySQL) on a phone. This is your decision. It's possible, so try it and see what you can do with it.

Note: I tested most of the compiled software on a Samsung Galaxy S2 (my phone). I cannot test it on a variety of phones because I have no mobile phone shop...


Currently buildable with this scripts (abstract, see the full list in package_selection.conf):

• PHP 5.3.10 (with almost all extensions and XCache)
• MySQL 5.5.19 (if you want a full database server on your phone...)
• lighttpd 1.4.30
• OpenSSH 5.9p1
• OpenSSL 1.0.0f
• Samba 3.6.3
• cURL 7.23.1
• bash 4.2
• BusyBox 1.19.3
• ImageMagick 6.7.4-10
• OpenVPN 2.2.2
• Python 2.7.2
• Node.js 0.6.7
• git 1.7.9
• Wget 1.13.4
• Rsync 3.0.9
• TCPDUMP 4.2.0
• Midnight Commander 4.8.1
• Pure-FTPd 1.0.35
• screen 4.0.3
• iptables 1.4.10 (without bionic bug "getnetbyaddr()" and with iptables-save/restore!)
• BIND 9.8.1-P1
• Lua 5.2.0
• Ruby 1.9.3-p0
• usbutils 004
• MTDUtils latest git
• GCC 4.6.2 (c, c++)
• GNU Coreutils 8.14, Binutils 2.22, Findutils 4.4.2, Inetutils 1.8
• QEMU 0.15.1
• SANE 1.0.22
• tesseract-ocr 3.01
• ...
• a lot of libraries, development tools and other useful programs
  ...

FEATURES:

• automatic download and untar the cross-compilers (if not already done) for cross compiling
• automatic download and untar the source packages (if not already done), that you have selected
• solve dependences and reserves additional needed packages
• apply patches to the sources if necessary
• build them, "install" them in a folder for later "ready to transfer" to your phone and stripes the binaries and libraries to reduce the filesize if selected
• autogenerate a tar file and adb push the file to device if selected
• allows you to use well-known software like PHP, MySQL, OpenSSL, OpenSSH, OpenVPN, bash and lighttpd on your phone (see the full list in file package_selection.conf, see attachement)
• scripting and development on your phone
• no limits with BusyBox (without want to be derogative, BusyBox is a wonderful tool), full featured tools like Coreutils, Binutils etc.
• build scripts are expandable with other or newer packages you eventual want to have

QUESTIONS AND ANSWERS:

Q: Do I need native GNU/Linux programs and what is that?
A: If you do not know what native Linux applications are, then stop here - you don't need them.

Q: What can I do with these tools?
A: Many things. As a developer you can use the libraries to include them in Android GUI apps, as user you can start for example a webserver on your phone and write some HTML or PHP pages
just for fun or replace BusyBox with full featured tools (or only some commands).
Some tools are also very useful to find network errors, check the network traffic and many more.

Q: What do I need?
A: A Linux installation (VM or real), some Linux knowledge and a little bit time to read all instructions carefully. A rooted Android device with enouth free memory (ext2/3/4 formatted)
See README for details.

Q: Costs?
A: Nothing. The scripts are Open Source (GPL v3), the packages are GPL licensed or under a similar license. The cross-compilers are Open Source or free to use for non-profit purposes.
But I would be pleased if I get a donation and/or a press on the "Thanks button", if the scripts are useful for you. It was a long and hard work...


Note: The attachement scriptset-x.yy.zip is double packed (inside .bz2), because I want preserve the right settings of the scripts.

STEPS:

1. Selected the desired packages in the config file
2. Adjust the compiler settings for your phone
3. Run the build script to download and compile the software
4. Copy the generated TAR files to your phone or let "adb" this do for you and unpack them in the prepared sysroot folder

For more information please read the instructions in "README"
Please let me also know if you have any problems or if you have questions or further suggestions.

Thanks go to the Open Source community and a lot of people/webpages with hints about cross-compiling software.

Thanks

smitna

CHANGELOG:
01/14/2012 First public release v0.9
01/14/2012 Update: Removed unneeded big patch for ncurses
01/15/2012 Release of v1.0
Update: Fixed build of gcc. Note: the header files are not copied automatically with the script.
If you want to compile on your phone, set TAR_HEADERS=1 in configuraton.conf
Typos: some translations corrected in build.sh
01/16/2012 Release of v1.1
Update: Removed building of fsck from util-linux-ng, it's in e2fsprogs
Update: ISC Cron / vixie cron: set correct paths in pathnames.h before compiling
Enhancement: More detailed explanation of device specific compiler settings with list of possible values
Update: Removed dependence e2fsprogs libs from e2fsprogs
01/17/2012 Release of v1.2
Update: Build of zip, zipcloak, zipnote, zipsplit, unzip, unzipsfx and funzip now possible without errors
01/17/2012 Release of v1.3
Update: Added Node.js
Typos: some translations and language errors corrected in build.sh and README (sorry, I'm not a native English speaker...)
01/18/2012 Release of v1.4
Fixed TMP_PATH in PHP extension XCache (mmap.c) to prevent startup warning
Fixed BusyBox problems with static build against glibc ("__getpagesize" segfault with ping, traceroute and other networking parts)
and added some BusyBox patches.
BusyBox is now dynamic linked, so don't replace the system BusyBox with this build, because of missing mount/paths to libraries at phone startup
01/20/2012 Release of v1.5
Update: Cleanup in scripts and separation of configuration and code. Now it's only necessary to edit configuration.conf and package_selection.conf
Update: Added gzip
Update: Added experimental adb push to device if the device is connected and permissions are ok
Update: Check for installed programs. Script now exists on check failure to minimize errors while compiling.
01/21/2012 Release of v1.6
Bugfix: MySQL build could fail if gen_lex_hash could not generate the file sql/lex_hash.h in pass 2
Update: adb push should now work, check for connected device (and possible multiple devices) and check for right permissions to use the device
Enhancement: Even more explanation of device specific compiler flags and some hints to find the right settings.
01/21/2012 Release of v1.7
Enhancement: To eliminate sure a failing MySQL build, with every new run the source folder will be removed before completely
Enhancement: file version is now strictly checked (the host version must be the same as the cross version)
Some minor changes and filechecks (e.g. autoconf)
01/23/2012 Release of v1.8
Bugfix: Python has SQLite dependencies and SQLite must be ready before Python configure/make
Bugfix: In certain circumstances the optimizer settings get lost
Bugfix: Removed --enable-multibyte from bash, no chance to use it with NDK
01/24/2012 Release of v1.9
Update: Added ngIRCd (Please read the comment for this server, because of running as root)
Update: Sorted the links and package names in the config a little bit...
01/26/2012 Release of v2.0
Bugfix: OpenSSH wants to touch /var/empty while compiling, should be fixed --with-privsep-path now set (thanks to member dateno1 for his reports of such warnings/errors)
Bugfix: Removed a wrong CFLAG -I path from NDK config (only useful for me, really useless for others...)
Enhancement: Example for a user db for Pure-FTPd virtual user database in README (please use virtual users, it's the best and easiest solution)
Update: OpenSSH build need ssh-keygen for the generation of the default keys, added this info to README and the package selection
Bugfix: Not really a bugfix, but some installers don't set the right permissions of libraries in the sysroot folder, so set them global in the final task
Bugfix: Removed --with-lastlog=$TARGET_SYSROOT/var/log/lastlog from OpenSSH and added --with-lastlog=no, I don't know why this is not working and throws a warning, but we can live without it...
Enhancement: Added option to choose the sysroot variant from the CodeSourcery libs: v5TE (default), v4T (old), thumb2 (v7, fastest if supported by the phone)
Enhancement: Added option to tar the headers for development on the device with CodeSourcery (hint: always set rpath in LDFLAGS to make use of the CodeSourcery libs)
01/26/2012 Release of v2.1
Update: Added getcap/setcap and libraries from wishlist, needs libtoolize. Note: This is in a testing stage, kernel and filesystem support needed!
01/30/2012 Added a ScriptStarter app
09/02/2012 Release of v2.2
Update: Local compilation of static "tic" from ncurses for buildhost, if ncurses installation is missing
Enhancement: "downloading only" mode without compiling the packages (and force download)
Enhancement: some parameters for starting the script with additional functions (show with ./build.sh help)
Enhancement: more logging and outputs
Enhancement: language specs, all outputs can be changed to other languages in build.sh
Update: Added dosfstools
Update: Added Ruby
Update: Added example configs and starter scripts for Samba and OpenSSH
Update: Use of http-URLs if possible and update of some broken external links since last release
Bugfix: util-linux-ng needs gettext, added dependence
Bugfix: Added parameter to Samba to solve user/group settings and socket_wrapper support to set path vars at runtime
Update: Changed PHP version to 5.3.10, Samba to 3.6.3 and git to 1.7.9
02/11/2012 internal Release v2.3
Bugfix: openVPN patch for /dev/tun and links to ip/netstat/ifconfig/route, see hints for symlinks
02/16/2012 internal Release v2.4
02/25/2012 Release of v2.5
Update: Added LAME
Update: Added experimental CUPS
Update: Changed PNG version to 1.5.9
Update: Added Net-SNMP
Update: Added Netpbm
Update: Added PNG 1.2.47 for Netpbm and phpSANE
Update: Added LibTIFF
Update: Added SANE (USB Scanner backend and drivers) - OTG equipped device needed to work for this - YES, Scanner Access Now possible and Easy with your phone
Update: Split patches for bash (4.1 and 4.2) for better and easier (automatic) inclusion of original patches
02/28/2012 Release of v2.6
Update: Added Cuneiform OCR
Update: Added tesseract-ocr (see hints for more details howto use it)

ScriptStarter
02/03/2012 Added a small ScriptStarter for the GUI. With this app, written with JQuery, PhoneGap and Java (wrapper to execute the shell scripts) it's possible to start for example the lighttpd web server, Pure-FTPd or MySQL from within the Android GUI without a terminal and read the output. If a script needs root access, there is a checkbox to allow this. For this to work, copy the script sush in the script folder. All scripts ending with .sh can be started (please choose after the first start your script folder on the device to add the scripts to the selection, normally your [SYSROOT]/sbin folder).
Three buttons (start, stop and status) are predefined for start-stop-status scripts, there is also a parameter input field for more start values.
Free GPL v3 source code also included (Eclipse project). Please don't expect too much eye-candy. The GUI could be much better, but my CSS skills are very limited...
I added a few example scripts in the zip file, for example a battery stats reader. Before starting these scripts, have a look in the source code to be sure that they are compatible with your device.
Update 0.97 02/03/2012: Now the stderr is also printed in the output window
Update 0.99 02/10/2012: Replaced scrolling api with a much better solution (jScrollPane) for the output window. Not the real Android scroll feeling, but it's usable.

Someone tried it? Problems?
Please, I need your feedback...

dateno1 said:

i will test on cortex a8 (motolora xt720)

TARGET_MARCH="armv7-a" (not work on armv7 )
TARGET_MTUNE="cortex-a8"
TARGET_MFPU="neon"
TARGET_MFLOAT="softfp"

it make dynamic binarys but i don't want dymanic type (because it difficult to divide one thing and maybe not work on android's libs)

Click to expand...

Click to collapse

I hope I understand your question. Why not dynamic?
Yes, you have to set the parameters for your mobile phone. For TARGET_MARCH try "armv7", for TARGET_MFLOAT "softfp" and for TARGET_MFPU "vfp" but check this last setting carefully for this phone.
It's very hard to build a full static binary (and sometimes nearly impossible) and it's also hard to build against bionic (with NDK). I had no good experience with it . A lot of compiler runs wasn't sucessful. Edit: I also had for example massive problems with openssl. I couldn't find a way to build it with NDK or static and then build other programs like cURL with openssl support.
Static binaries are also too big and not necessary. For this we have a sysroot environment to be independent from bionic and without the restrictions. The compiled libs and binaries are linked to the CodeSourcery glibc and have set the custom "rpath" for the dynamic loader. That's no problem I think.

dateno1 said:

that's simple

if someone want to make some binary for all version of android (even same cpu) it will not work well (defend on basic lib(ex : libc) is ok but complex things will not work)

i know make static type binary is very difficult but it valable

Click to expand...

Click to collapse

You are right, if you want to build a "portable" binary, you have to set very safe settings, like TARGET_MARCH="armv5", the full list is in the "README". But it's not my main target to offer a solution that is easy portable to other phones, instead you can build the software with optimizations for your phone to "get the best" for it

dancer_69 said:

I want to try this, to install some apps(most interested fro mc), but I don' t wanna change my sd card.
Can I use phone' s /data partition as mysysroot directory? I have 1.5GB space available on data partition, it is 2GB all.
And If yes, I must create the dir: /data/mysyroot? And where to create this dir on pc? On root directory of my linux system?

Click to expand...

Click to collapse

You can use the /data partition. Just create a new folder, for example /data/sysroot. On your PC you have to create the same temporary with full user rights. This is not your working dir for building. Create a new dir in /home/user and unpack the files there. In settings insert the /data/sysroot as TARGET_SYSROOT and change the other settings for your phone and choose mc=yes. Then start the ./build.sh

dancer_69 said:

Thanks for your quick answer.
Just to make clear because my english understanding is not in a high level
I must create the /data/sysroot dir under "/"(root directory) on linux pc with superuser rights?

Click to expand...

Click to collapse

Yes, that is right. But this is the only step you need to do as root or better with root rights. After this work as usual as user.

dateno1 said:

package selection : mysql only

target directory (prefix) : /sddata/mysql (on sdext on my system)
armv7-a
cortex-a8
neon
softfp

then run ./build.sh 2> error.txt

it is my error messages

http://pastebin.com/g7Yi4bGz
sorry you can see this link only today (my mistake)

this setting can make some binary and lib run on my phone but fail to make gcc or mysql (when i try gcc it just fail to making gcc only (other things were made) )

Click to expand...

Click to collapse

Please notice first, that you should not run this script as root!!

Then there is something wrong with the build of the first needed packages:
ZLIB=yes
OPENSSL=yes
CURL=yes
GETTEXT=yes
LIBXML2=yes
READLINE=yes
NCURSES=yes

- The script cannot find the readline sources. Do you have a subfolder readline-6.2 in folder sources in your builddir? Same for the other source folders. The script first downloads the packages to folder packages, extract the tars in folder sources, repeated for all selections.
(look at line 67 in your log: ./build.sh: 1402: cd: Python-2.7.2: No such file or directory)
- You have no "xz", install it first and make sure, that the other necessary packages are installed. MySQL and gcc are "two pass" compilations and this builds generating temporary binaries, runable on your linux pc. So you need all for a successful compilation of such binaries like gcc, make, python etc. on your linux machine. If you are using Ubuntu, then install the "meta-package" build-essential, look in the README.
Do you have the other unpacking tools like gunzip etc. Please install also "bison"

You should do following steps:
1. First select only the necessary packages (see above), all others to no.
2. If zlib, python, readline an the others are cross-compiled (without errors) and the libraries are in ./sysroot/lib, then deselect them (=no) and select mysql=yes and start a second run with ./build.sh

dateno1 said:

i'm not edit necessary part
ZLIB=yes
OPENSSL=yes
CURL=yes
GETTEXT=yes
LIBXML2=yes
READLINE=yes
NCURSES=yes

i just edit mysql=no => mysql=yes

ok try to install packages and run as not root

Click to expand...

Click to collapse

Good luck!
If you can wait a little bit:

In a short time I release a new version with better error and program checks and some small bug fixes. With this version it's not longer necessary to edit the build.sh, it's now all in only two separated configuration files (configuration and package selection).

dateno1 said:

with this error

note: the mangling of 'va_list' has changed in GCC 4.4

full error log on http://pastebin.com/jd1Pw9Xw

Click to expand...

Click to collapse

Hello dateno1,

first: you have *NOT* changed to a user account as promised

1. The va_list is only a note, not an error, no problem (but ask the MySQL devs why they haven't adapted the code to gcc 4.6.1).
2. Your Python cross binaries and libraries are not ok (from you broken first build?). Please start from a "fresh state" (delete the folder sources/Python-2.7.2 and sysroot) or start from a total clean state.
3. Please use the latest 1.5 version of the script because of the xargs warnings at the end
4. Delete the complete sysroot folder, there are some folders not in place (data and docs). There is a script remove_all_files.sh
   Start this script in your buildroot with ./remove_all_files.sh to remove all files and symlinks in subfolder sysroot.
   Edit: delete the sysroot folder complete, there are some other missing folders from your first build!
5. Start again with ./build.sh and activated packages (the "ALWAYS NEEDED" one and MySQL)