[2015/03/07] BotBrew: *nix tools for Android

Search This thread
By:
Advanced…
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
Thanks for trying this, mateorod. I'll try to figure out what's going on with the hello package. The hello executable is actually just a shell script -- just have a look inside /system/bin/hello (cat /system/bin/hello). I wasn't intending to make anything special with it... just an example of how an Opkg package might be constructed. I appreciate the time you've put into this!

Oh, and if you're using the GUI: it automatically remounts /system for you, and it should raise a root request when that happens. it should also request root when it actually installs something (including itself, during the bootstrap phase). And, after it's done what it wanted to do, it makes /system read-only again (but only if it had to remount rw in the first place).

Updates from my end:

  • python now works with hashlib
  • made lots of development packages, which is good for building stuff directly on the device (but i haven't actually tested them yet)
  • packaged runit, a service watchdog that keeps your stuff running
  • packaged dropbear, a small ssh client and server (with support for runit)
  • packaged dtach, which provides the gnu screen functionality for sharing shells
  • packaged weechat, an irc client

nb: runit is currently not able to start itself; but you could start it using `nohup runsvdir-start &`. this should also bring up dropbear, if you have that installed too. runit might also consume more electricity by monitoring processes; but that's going to happen anyway with dropbear running... I'll have to test battery usage.

I have a couple of screenshots demonstrating runit, dropbear, dtach, and weechat. I have dropbear running on my [CM9] nookcolor, monitored by runit. I have weechat running on the nookcolor, under dtach. On my netbook, I am ssh'd into the nookcolor and sharing the weechat screen. The reason that the netbook view is slightly funky is that I'm letting the nookcolor control the terminal size/shape; I could just as easily make it netbook-friendly (but then it would look strange on the nookcolor).

kyf74.png


2ni4ep4.png
 
Last edited:
  • Like
Reactions: Fritos2
M

mateorod

Inactive Recognized Developer
Nov 16, 2011
1,981
3,503
New Orleans
www.gigmasters.com
Thanks so much for walking me through all that! Obviously I am totally new to this. Until I rooted this nook a couple months ago, the last computer I owned was killed by the Love Bug. For real.

Anyway, point is, if this is not part of the project, or simply not doable, just point that out to me, I don't even have a good handle on the possibilities, let alone the limitations.

I was thinking about attempting to compile a text editor . Are these tools something i could build Vim with? If you know the package i should use, you could just point me in that direction, I wont make you hold my hand like you had to for the first build. Thanks for the time.

Edit: OK, a second look yielded a how-to featuring python and gcc(it looks like? Maybe some dependencies?) I am going to try and see if i can't figure it out. Stop me if you think I am gonna hurt myself...
 
Last edited:
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
Fritos2 said:
I just run a django server on my optimus black finally. :):)
Click to expand...
Click to collapse

Oh, wow. Excellent work!

mateorod said:
I was thinking about attempting to compile a text editor . Are these tools something i could build Vim with? If you know the package i should use, you could just point me in that direction, I wont make you hold my hand like you had to for the first build. Thanks for the time.
Click to expand...
Click to collapse

So, there are two ways of building stuff for the nookcolor. If you look at my git repo, you'd find "recipes" for building vim (and other stuff) on a regular Linux computer. In fact, the vim in my repository was built this way. The other way is to do it directly on the nookcolor, using gcc and development headers+libraries. I just tested this on CM7/SD and it works (I had some trouble with CM9); here's how:

(... and the third way is to not build it at all, and just use the vim that comes with CyanogenMod.)

1. install the prerequisites (using either command line or gui)
you need: gcc-4.6 make libncurses-dev

1b. free up some space
at this point, you want to make sure you have at least 80mb free in /cache, because that's where we'll be doing the work. a good way is to clear out the opkg archives: rm /cache/opkg/archives/*

2. prepare the workspace and grab the source code
Code: 
mkdir /cache/vim
cd /cache/vim
wget ftp://ftp.vim.org/pub/vim/unix/vim-7.3.tar.bz2
tar jxvf vim-7.3.tar.bz2
rm vim-7.3.tar.bz2

3. change /bin/sh to /system/bin/sh everywhere
Code: 
for file in `find vim73 -type f`; do sed -i 's/\/bin\/sh/\/system\/bin\/sh/g' $file; done

4. configure and build
Code: 
./configure --with-tlib=ncurses --disable-gtktest --prefix=/system
make

5. install
Code: 
make install DESTDIR=/cache/vim/out
(alternatively, you could just install to the root by omitting the DESTDIR stuff, but then it might make a mess. better play it safe for now.)

6. enjoy! (test)
Code: 
cd /cache/vim/out/system/bin
./vim
if you don't see any pretty colors, you might want to install ncurses-common
 
Last edited:
M

mateorod

Inactive Recognized Developer
Nov 16, 2011
1,981
3,503
New Orleans
www.gigmasters.com
GUI bug

I removed opkg to clear some.space in /cache as detailed above, but I intended to use the UI to get libcurses among other packages. Botbrew prompted me to install opkg again, as it is needed for operation. After the install, the prompt reappears. You have the option of simply pressing the back button and accessing the UI, but a user could have some confusion as to whether they were done. After every installation of opkg, the prompt reappears upon first run of the app.

Botbrew-2

I don't see make in the ui either, although you could just be waiting to include that in the next update. Thanks for the work!

Edit: Ok, that's not it. It loops continuosly from saying I am "good to go" then goes immediatly back to the welcome prompt. I have adjusted my r/w mounts, and will go through a terminal and see if that makes a difference. The only bug could be telling noobs like me that I'm good to go when in fact I am probably just a moron : )

Edit 2: Yep, just dumb! Had applications running as root still blocked through my firewall. The Botbrew app's only problem is not differentiating between successful/unsuccessful installations and reporting every attempt as "good to go". No installation script is shown like with a successful install, however.
 
Last edited:
  • Like
Reactions: inportb
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
You'd clear the opkg archives, but not remove opkg itself... there is currently no way to do this using the GUI, but you could use the command line.

Regarding the bootstrap bug: I partially implemented Opkg in Java (lol), so the app now has better control over the process. I believe it is fixed, and a new BotBrew.apk has been released.
 
Last edited:
liam.lah

liam.lah

Senior Member
Feb 12, 2010
653
31
Nevada
Dude, this is freakin superb. a package manager really shortens the gap between android and any other linux distro.

At any stage of development would you be open to taking requests for stuff to be added to the repos?
 
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
Sure. What open source command line software would you like to see in the repo? I can't guarantee success (i.e. GNU Screen has been a sticky one), but I'll try.
 
racks11479

racks11479

Senior Member
Jan 2, 2011
990
748
Fresh Meadows
inportb said:
Ooh, this one depends on Java. I'll see if I could get Java packaged; or maybe get apktool to run on dalvik :)
Click to expand...
Click to collapse

Yes I know, I've been trying to get it to run on android and it goes back to the java requirement. This is why what you're doing would be awesome. Especially for progress on a script I've been working on.

I'll be lurking on irc as well as this thread to keep track on BotBrew! :D

Thanks again,
Racks
 
liam.lah

liam.lah

Senior Member
Feb 12, 2010
653
31
Nevada
inportb said:
Sure. What open source command line software would you like to see in the repo? I can't guarantee success (i.e. GNU Screen has been a sticky one), but I'll try.
Click to expand...
Click to collapse

To be honest, i can't think of anything that really needs to be in there, though today i used imagemagick to convert some pictures of documents i took into .pdfs

Im not sure how much effort it is for you to add each individual package, but little tools like this could be cool.
 
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
Alright, I'll take a stab at ImageMagick too.

apktool is difficult for two reasons: aapt is part of the Android platform tools, and Android is a monolithic chunk that doesn't like to be taken apart; it also depends on Java, but I think we could get around that by compiling the Java to Dalvik. I think it would be nice to have some self-hosted Android tools though, so I'll poke around.

I'm also in the middle of restructuring the build system. As the list of ported software grew, I felt the need to make improvements.

/edit:

So I got ImageMagick; but before I push it to the repo, I need to add more image formats to make it worthwhile.
 
Last edited:
  • Like
Reactions: mateorod
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
ImageMagick is now working, and these formats are supported :)
You might see some notable omissions, such as png... but we'll figure this out some day! I was also unable to make Ghostscript, so those functions are missing as well. But we could now make pdf's out of jpegs, and so on.

Meanwhile, I've been busy restructuring the build system. There have been no commits to the git repo because I'd like this to be an atomic update. I'm also working on making a native LLVM+Clang, which would replace GCC if it works.

Thanks for playing along, and feel free to suggest more software. If you'd like to learn how to make your own packages and host your own repositories (you could even use Dropbox), just let me know. I might also start accepting package submissions and hosting them in a secondary repo.
 
  • Like
Reactions: racks11479
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
Awesome. Soon, you will be able to read/make png files (I hope).

Today, I'd like to talk about standardizing a filesystem hierarchy that's friendly to both Android and *nix. As you know, Android's filesystem hierarchy is slightly different from *nix's, and part of the porting process involves tweaking the places where the programs expect to find things. I am currently maintaining some 61 bits of software, and it takes a few days to make drastic changes such as these; rather than waiting, I'd like to propose the following standard now:

/system instead of /usr for things that don't change regularly (programs, libraries, etc)
/data instead of /var for things that change often (databases, etc)

/system/bin for user and system executables (programs)
/system/libexec for program executables (accessory programs)
/system/lib for object code libraries
/system/etc for read-only single-machine data (configuration files)
/system/include for C header files (useful for building software onboard)
/system/share for read-only device-independent data (documentation, miscellaneous scripts)
/data for modifiable data (data produced by running programs)

Subdirectories of /data include /data/lib, /data/local, /data/log, and /data/spool.

Furthermore, Opkg provides a mechanism to install to alternate destinations, because mobile devices are often resource-constrained. I'd like to enable destinations such as /data and /cache to be used for package installation. In this case, I'd simply prefix everything such that /system becomes /data/system, and so on. At the moment, installing to sdcard is not possible because its filesystem is largely incompatible.

*nix users would also be familiar with the concept of a home directory where user data (documents, photos, music, ...) are stored. We already use the SD card for that, and because Android is a single-user system, I think it would be reasonable to make /mnt/sdcard the global "home directory."

What do you think?

p.s.: You might have noticed that I left out sbin -- this directory usually contains small, statically-linked programs used to boot the system; we don't need to worry about boot-up because Android already handles that for us, so I figured I'd lump that with /system/bin.
 
Last edited:
M

mateorod

Inactive Recognized Developer
Nov 16, 2011
1,981
3,503
New Orleans
www.gigmasters.com
Okay, I was gonna leave you alone, I swear. I went to try spartscusrex on his terminal ide post, but there you were, commenting on the very issue I was about to ask about! Kismet, I told myself.

I am trying to run the smali decompiler/editor/recomplier from within Terminal IDE, to edit the services.jar file needed to supercharge ICS, as well as decompile a dicey classes.dex file from a suspect app. When I run the command line on the IDE, namely:

java -jar baksmali.jar -x services.jar -o classout

I get a Java I/o exception error, something I normally associate with (forgive me) some kind of connection error. I don't think that is the problem in this case.

I am not asking you to diagnose my issues within the IDE, but from what you said in that post is dropbear something that could execute that command line?

More specifically to your project, if I even understand it correctly after all this time, if I did get the editor working, is Botbrew a method for hosting/distributing the edited services.jar for the nook in specific, through my dropbox, say?

Thanks for all your patience, my friend.
 
Last edited:
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
mateorod said:
I am trying to run the smali decompiler/editor/recomplier from within Terminal IDE, to edit the services.jar file needed to supercharge ICS, as well as decompile a dicey classes.dex file from a suspect app.
Click to expand...
Click to collapse

Aha. This resembles what racks11479 and I are trying to do, from a different direction.

mateorod said:
I get a Java I/o exception error, something I normally associate with (forgive me) some kind of connection error. I don't think that is the problem in this case.
Click to expand...
Click to collapse

The IOException might result from an unsuccessful attempt to read to or write to the filesystem. Are you trying to work with files on a read-only /system partition? Also beware of file permissions. And note that if your program uses temporary files, those may trigger the exception as well.

mateorod said:
I am not asking you to diagnose my issues within the IDE, but from what you said in that post is dropbear something that could execute that command line?
Click to expand...
Click to collapse

Dropbear is a remote access service (using SSH, to be exact). In general, you could do with dropbear what you could do using adb shell or terminal, but it has some more tricks.

mateorod said:
More specifically to your project, if I even understand it correctly after all this time, if I did get the editor working, is Botbrew a method for hosting/distributing the edited services.jar for the nook in specific, through my dropbox, say?
Click to expand...
Click to collapse

Yes, you could make a package for it and host it on Dropbox (or send it to me and I'll drop it in my repo). I'd suggest that you also host a copy by itself, for people who prefer not to use package management tools.
 
  • Like
Reactions: racks11479 and mateorod
liam.lah

liam.lah

Senior Member
Feb 12, 2010
653
31
Nevada
This does work on other devices too, right? I haven't seen you post this anywhere else. I bet something this cool could become pretty popular if it is given a bit more exposure.
 
I

inportb

Retired Recognized Developer
Dec 29, 2008
438
194
inportb.com
I've only tested on a Nook Color, but it should work on other armv7-a devices (and perhaps even armv6l and armv5l). But that's the problem; I have no way of testing this with other devices xD

I just updated Opkg. Before you go ahead and upgrade it, note that I've started to implement the filesystem standard outlined above. Have a look at your package list and remember which packages you have installed, because Opkg might lose track of them during the upgrade! If that happens, you'll have to reinstall these packages.

There is a new repository for testing new packages (enable it by installing repository-botbrew-testing and refreshing the list), and it's now filled with software rebuilt for the new filesystem standard. If you were able to test some packages, please let me know so I could move them to stable or work on them some more. Make sure you have the latest BotBrew.apk, or it might be confused with multiple versions of the same programs.

New stuff in stable:
gcc-4.6 is much smaller (only 50MB!)
opkg (see above)

New stuff in testing:
nmap (network security scanner)
tcpdump (command-line packet analyzer)
wireshark (more tools for network traffic analysis, i.e. tshark)
rsync (synchronizes files between folders, or remotely through ssh)
imagemagick (image manipulation tool, should support png now)
llvm+clang (new development tools... not quite ready yet!)
 
Last edited:
  • Like
Reactions: mateorod
You must log in or register to reply here.

Similar threads

D
Android 3.0 Honeycomb Preview Image v04 - 2/7/2011
Replies
2K
Views
1M
fattire
fattire
mrg666
[ROM+Kernel] MiRaGe - Custom CM7.2 for Nook Color - 07272013
Replies
2K
Views
758K
Z
zagzag99
V
[ROM][CM7] [v1.3] Size-agnostic SD Card image and CM7 installer for SD Cards.
Replies
2K
Views
1M
A
AdilJ
fattire
[ROM] CyanogenMod 10 Nightly Builds for Nook Color
Replies
2K
Views
708K
leapinlar
leapinlar
S
[DISCONTINUED] CM9/ICS Nightly Builds
Replies
2K
Views
663K
cammykool
cammykool

Top Liked Posts

24 Hours All time
  • There are no posts matching your filters.
  • 27
    I
    inportb
    gAZkBXR.png

    The remainder of this post contains historical information. Please read the update. Thank you.






    __________________
    ​
    BotBrew is
    • a repository of *nix software (such as bzip2, curl, openssl, python, and ruby) for ARM-based Android
    • a package manager powered by Opkg, a lightweight program that feels like dpkg+apt
    • a service manager powered by runit
    • a build system for anyone looking to build and package his/her own scripts and programs
    Thanks: mateorod and xela92 for testing the heck out of this thing; racks11479 for delicious artwork; you for using this project, reporting bugs, and making it better
    If I missed anyone, let me know!

    Warning: BotBrew has been used successfully on a variety of rooted ARM devices, and is developed using a Nook Color, but you should still make backups before trying, just in case.

    Install BotBrew
    [ Google Play | BotBrew.apk | BotBrew.debug.apk ]

    Read the Quick-Start Guide & Manual
    [ botbrew.com/manual.htm ]​
    Click to expand...
    Click to collapse
    Alternative (Command Line) Installation
    If the BotBrew app fails to bootstrap, you might be able to bootstrap using the command line:
    Code: 
    wget http://repo.botbrew.com/anise/bootstrap/install.sh -O- | su
    Such a setup should be compatible with the app, though it may not work perfectly. The manual has more tips for command line usage.

    Enjoy!

    Install BotBrew+1
    The next release of BotBrew, named basil, will be powered by Debian's Dpkg and Apt. This is a non-trivial update, so the app is being rewritten from scratch. I've posted some usage instructions, in case you are adventurous enough to try. Thanks! This app may be used in parallel with the current-stable BotBrew release, without conflict.
    [ Google Play | GitHub ]​
    Click to expand...
    Click to collapse
    Changes
    5/20: the next release of BotBrew is in development!
    4/20: improve support for long package names; fix list of repairable packages
    4/16: new UI for devices with wide-enough screens; experimental support for moving to /sd-ext
    4/9: Google Play release of BotBrew "anise"; previous release is now BotBrew.oldstable.apk; lots of updates since oldstable
    3/16: lots of installer and filesystem changes for cross-device compatibility, work started on multiuser support
    3/10: installers now depend on botbrew-core, which will (in the future) pull in basic packages that everyone should have
    3/4: swipe left and right to see all/installed/upgradable packages
    3/3: added ability to start installation of *.opk files from file managers
    3/2: added ability to start installation from browser after clicking *.opk link
    3/1: fixed some BotBrew.apk bugs; updated command line installer
    2/27: reworked BotBrew.apk; new packages in stable repo
    2/18: testing repository now open; new opkg -- please read before upgrading
    2/16: make BotBrew.apk display latest versions in the package list
    2/2: bugfix release of BotBrew.apk
    View
    14
    I
    inportb
    Alright, here's a very rough build that is able to bootstrap the base system and do some basic package management. Services do not work yet, but they're coming soon! This app may be used in parallel with the current-stable BotBrew release, without conflict.

    Source: https://github.com/jyio/botbrew-gui
    Issues: https://github.com/jyio/botbrew-gui/issues

    Thanks for the help testing this thing! If you have bugs to report or features to request, feel free to post here or (if you have a GitHub account) open issues directly.

    Quick Start

    Warning: If you're using this in parallel with the stable BotBrew, you'd find that some packages are available through both apps. Please do not try to install the same thing in both apps, because they would let you, and bad things might happen.

    Notice: If you bootstrap to a FAT filesystem (such as the one on your SD card), everything will be installed to a loop-mounted disk image. Disk image support is very basic. The image is 100MB; if you want to resize it, you must do so manually.
    Click to expand...
    Click to collapse

    Let's assume that you bootstrapped to /data/botbrew-basil. The bootstrap process helps you prepare BotBrew for some common usage patterns. In case you want to do it manually, just keep reading.

    Running Commands

    For convenience, you could just install the botbrew-wrapper package. This gives you a botbrew2 command, just like the old-style botbrew command. Run it without arguments to start a shell, or prefix your commands with it.

    If you like doing things the long way, you could do without the botbrew2 wrapper. For one-shot jobs, you may simply prefix your commands:
    Code: 
    /data/botbrew-basil/init -- command goes here
    If you want a shell so you don't need to prefix everything:
    Code: 
    /data/botbrew-basil/init
first command
second command
...

    Installing Debian

    Before you could install Debian, you must enable a Debian repository. Just install repository-emdebian.

    There are currently two "flavors" available to you:
    1. Install botbrew-debian-apt for a robust installation of apt/dpkg. This does not include things like grep and findutils, which some packages need; keep that in mind when you see these errors, and install the required packages.
    2. For full Debian support, also install botbrew-debian-minimal afterwards.

    Removing BotBrew

    To remove everything other than the app, first exit the system from the preferences window, then:
    Code: 
    rm -r /data/botbrew-basil

    Enjoy!
    View
    3
    I
    inportb
    Thanks for all the support, folks! I know I haven't been posting a lot, but here's a little screenshot to show that a working pre-release is imminent. There's still a lot of work to do, but I'd like to at least support installing/upgrading/removing packages. The reason for the delay is that dpkg+apt is substantially more complex than opkg, and there was a lot of planning involved. Anyway:

    jidws3.png

    Package maintainers: you might have noticed that opk's are very similar to deb's. Tools for building deb's are widely available, and could be used in much the same way as opkg-utils. You could continue to host repos via your favorite file host (i.e. Dropbox) if you want. The difference is that debs have some more control fields (to support a more powerful package management experience), and some of them may be interpreted differently. I'll keep you posted. Thanks for using this project!
    View
    3
    I
    inportb
    BotBrew Coriander

    Hey friends, it's been too long. While BotBrew slept, Android has moved on. The world has moved on. I was unable to keep up, so I kept you waiting. I know it's been frustrating for you; believe me, it has also been painful for me to watch my own project suffocate slowly. With your support, I'd like to change that.

    If we wanted to build BotBrew today, what would it be? BotBrew has always been a tool for Android/Linux hobbyists like us to tinker with our mobile devices. In the beginning, Anise was a proof-of-concept that showed it was possible to install Linux tools directly on Android with a familiar package manager. Basil took the concept further by installing a minimal Debian system. There are other great tools for that now, such as LilDebi.

    vG64tSU.png

    Pwnna and I have been busy. Here's what BotBrew Coriander supports today:

    • Multiple "containers"
      • Debian template for ARM
      • Possibility of user-generated custom templates/appliances
    • Basic disk image support baked-in
    • Improved bus factor

    Here's what we're working on now:

    • Autostart specified containers during boot
    • Integrated task/process/service manager
    • Template distribution
    • Package manager integration

    Here's what we envision for the future:

    • Better management tools, like Heroku or AWS for your mobile device
    • Optional root, so you could distribute your appliances to *anyone*

    But what's important to you? How do/did you use BotBrew, and where should we focus the development effort? Thank you for your continued support.

    (This is not a beta release, but that's coming soon. Source code is also coming soon. Because this is no longer limited to the Nook Color, a new thread is also on the horizon.)
    View
    3
    I
    inportb
    I haven't put anything new on Google Play yet, but I've gone ahead and pushed the source to GitHub. I've also loaded the issue tracker with some of what I want to do.

    We're getting close!
    View

New posts