Android and Jenkins. What is the limit?

Wait, what? Why would there be a limit for Android and Jenkins? Is Jenkins somehow limited in building Android apps?
Take it easy, all is fine, I may have phrased it the wrong way.
We’re not talking about building Android apps with Jenkins, which has no limitations as far as I know.
We’re talking about building something - anything - with Jenkins, using an Android device as a Jenkins node, or, why not, as a Jenkins controller.
Sounds appealing or strange enough to get you intrigued? Then stay tuned, I don’t know yet how far we’ll go.

Jenkins and aarch64, a short history.

I joined the Jenkins project in April 2022. At that time, we could already find aarch64 docker images (be it for the agents or the controller) and regular installers for aarch64 Linux. The oldest image for a controller I found was from August 2021. The oldest image for an agent was from February 2022.
You see, there is nothing new under the sun. Jenkins does work on aarch64 Linux and has been running maybe for years.
My main subject is about building Android apps with Jenkins. It’s pretty easy when using Linux on an x86_64 machine, but it can be more difficult on an aarch64 machine. Not because of Jenkins, of course, but because all the tools needed to build Android applications were not available until late 2021 with Android Build tools 31.0.0.
Of course, you could use Rosetta to build your applications, and even with Docker.
Back to my experience with Jenkins and aarch64. I have several aarch64 Jenkins agents and controllers, some of them using docker, some of them being installed directly on the Linux machine thanks to the standard instructions.
Nothing special to worry about, Jenkins works fine for me with aarch64.

Android and aarch64, a short history few things to know.

Until (almost) recently, it was difficult to build Android applications on an aarch64 machine. The main reason was that the Android build tools were not compatible with aarch64 machines.
Before version 31.0.0, there was a bug in the Android build tools that caused the aapt2 tool to crash when building resources on aarch64 machines. This issue was resolved in version 31.0.0, which added native support for aarch64 and fixed the aapt2 crash on these machines.
Thus, Android Build tools are now natively compatible with aarch64. Isn’t that fantastic? No need to use Rosetta to build Android apps on aarch64 anymore.
Guess what, until I started to write this article, I was stuck with build-tools 30.x, because… Why not? I didn’t need to build Android apps on aarch64 machines, so I didn’t care.
A friend of mine works on an M1 Mac, which happens to be an aarch64 machine. He wanted to build Android apps on his Mac, and he was working with Docker of course, which translates x86-64 to aarch64, as long as you specify that you’re using an x86-64 image, to begin with. I know, it’s strange, x86-64 is called amd64 in Docker, but that’s not the point.

[...]
android-agent:
    platform: linux/amd64
    build: ../
    restart: unless-stopped
    volumes:
        - android-agent-data:/home/jenkins:rw
        - ../adbkey.pub:/home/jenkins/.android/adbkey.pub:rw
        - ../adbkey.txt:/home/jenkins/.android/adbkey:rw
    environment:
        - JENKINS_AGENT_SSH_PUBKEY=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBpNqXQ4x7fPPUBbYPxKF77Zqq6d35iPCD2chg644OUD
        - STF_HOST_NAME
[...]

We could theoretically build Android apps on Raspberry PI 3B+ from now on… with Jenkins of course.
But why am I telling you about building Android apps on aarch64 machines with Jenkins? Didn’t I tell you in the introduction that I wanted to build something with Jenkins on an Android device, and not the other way around?
You’re right. The goal is to run parts of Jenkins (first of all an agent, then maybe a controller, then a whole Jenkins instance thanks to Docker) on an Android device.
So why am I telling you about building Android apps on aarch64 machines with Jenkins then? Because it would be kind of ironical, and tons of fun to build an Android app with Jenkins on an Android device, wouldn’t it? That would even allow me to introduce you to the Jenkinsception concept. More on that in a future post.

First thing first, let’s get sshd working on an Android device.

What does sshd have to do with installing Jenkins? We have a few different ways to start and connect a Jenkins agent to a Jenkins controller, but the easiest one for me is to use an ssh agent. The first step is then to get sshd working on an Android device. There are various ways to access an Android device through SSH. Here are some of them:

• Using ADB: You can use the Android Debug Bridge (ADB) to connect to your Android device over SSH. This requires that you have enabled USB debugging on your device and have ADB installed on your computer. I don’t know if the following howto still works, but it’s a good start: https://dtbaker.net/blog/howto-enable-ssh-on-android-and-network-adb/
• Using SSHDroid: SSHDroid is an app that allows you to start an SSH server on your Android device. Once you have installed and started SSHDroid, you can use any SSH client to connect to your device over SSH. The problem is SSHDroid is an old app that is not maintained anymore. It is not compatible with Android 10 and above, and it is not available on the Google Play Store anymore. You can still find it on the Internet, but it is not recommended to use it.
• Using SSHelper: SSHelper is an SSH server that can be installed on an Android device to enable SSH access. Once installed, it provides you with an IP address and port number to use for connecting to the device over SSH. It’s also an old app, without any update since 2020. I would not recommend it.
• SimpleSSHD: SimpleSSHD is a lightweight and easy-to-use SSH server app for Android. It supports key-based authentication, and you can configure it to run at startup.
• S.S.S.H: S.S.S.H is another SSH server app for Android that supports key-based authentication and allows you to configure multiple users and ports.
• Using Termux: Termux is a terminal emulator and Linux environment app that allows you to start an SSH server on your Android device. You can then use any SSH client to connect to your device over SSH. Unfortunately, updates for Termux are not available on the Google Play Store anymore, but you can still find it on GitHub or F-Droid.

These are some popular ways to access an Android device through SSH. However, it is important to note that SSH access to your device can pose security risks, so it is recommended to use caution and only enable SSH access when necessary.

Termux is my go-to choice when it comes to using some kind of Linux on Android. There are packages available that will allow you to install new software, and package updates, like a “real” Linux distribution, and so on. I almost feel at home when using it.
Reading the documentation, I found out that Termux has an SSH server (OpenSSH) built-in. It’s not enabled by default, but it’s easy to enable it.
The following instructions are more or less a copy of what is available on the Termux wiki. I just added some more details to make it easier to follow.

Starting and stopping the OpenSSH server

Since Termux does not use initialization system, services are started manually from the command line.
To start the OpenSSH server, you need to execute this command: sshd. If you need to stop sshd, just kill its process: pkill sshd.
SSH daemon logs to Android system log, you can view them by running logcat -s 'sshd:*'. You can do that either from Termux or ADB.

Setting up password authentication

Password authentication is enabled by default. This will allow you to get started with it much easier. Before proceeding, make sure that you understand that password authentication is less secure than a pubkey-based one.

1. Ensure that everything is up-to-date and package openssh is installed:

    pkg upgradepkg install openssh
   

   Please note that $PREFIX is a variable that points to the Termux installation directory. It is usually /data/data/com.termux/files.

2. Password authentication is enabled by default in the configuration file. But you can still review it at ($PREFIX/etc/ssh/sshd_config), it should be like this:

    PrintMotd yes
    PasswordAuthentication yes
    Subsystem sftp /data/data/com.termux/files/usr/libexec/sftp-server 
   

   If it’s not looking like this, you will have to edit this file. Note that vi is not installed by default, but nano is. You can use it to edit the file.

3. Set a new password. Execute command passwd. While the program allows a minimal password length of 1 character, the recommended password length is more than 8 to 10 characters. Passwords are not printed on the console.

    $ passwd New password: 
    Retype new password: 
    New password was successfully set.
   

Setting up public key authentication

Public key authentication is the recommended way for logging in using SSH. To use this type of authentication, you need to have a public/private key pair. For a successful login, the public key must exist in the authorized keys list on the remote machine while the private key should be kept safe on your local host.

In the following example, it will be assumed that you want to establish public key authentication between your PC (host) and your future Jenkins agent which happens to be an Android device running Termux (remote). It also will be assumed that you’re running a Linux distribution on your PC, or WSL2, or even Cygwin. It would be better if both machines were using the same network (e.g. both are connected to the same Wi-Fi network). It is also assumed that you know your Android device’s IP address.
If you have access to your router webpage, you should be able to see which IP has been assigned to your Android device. If you don’t have access to the router webpage, you can find your IP address on an Android device by following these steps:

• Open the Settings app on your Android device.
• Scroll down and tap on “About phone” or “About device.”
• Look for the “Status” or “Network” section and tap on it.
• Find the “IP address” or “Wi-Fi IP address” option, which will display your device’s IP address.

Alternatively, you can also find your IP address within Termux and type the following command: ip addr show, except that the package may not be installed yet. You may have to issue pkg install iproute2 first. Look for the inet line next to the wlan0 line that has your IP address given by your Wi-Fi router.

1. If you do not have ssh keys, you can generate them. In this example, we will generate an RSA key. On the PC, execute this command: ssh-keygen -t rsa -b 2048 -f id_rsa (replace id_rsa with the name of your key. For me it would be ssh_key_for_jenkins_agent_2023-03-10). \
   The command shown above generates a private RSA key with a 2048-bit key length and saves it to the file id_rsa. In the same directory, you can find a file id_rsa.pub – it is a public key.
   For me, the command was:

    ssh-keygen -t rsa -b 2048 -f ssh_key_for_jenkins_agent_2023-03-10
    Generating public/private rsa key pair.
    Enter passphrase (empty for no passphrase):  
    Enter same passphrase again:
    Your identification has been saved in ssh_key_for_jenkins_agent_2023-03-10
    Your public key has been saved in ssh_key_for_jenkins_agent_2023-03-10.pub
    The key fingerprint is:SHA256:yoykbWyCHuqrANFBkO41vuXMC7kLhsVfe8caLWQEUqk user@PC
    The key's randomart image is:
    +---[RSA 2048]----+  
    |.+o ..o.         |  
    |.. . ...         |  
    |o .  .  .        |  
    | + oE  .         |  
    |o = o . S        |  
    |o+ B.* = o       |  
    |++oo& = + +      |  
    |= o=o+ . =       |  
    |=+.o... .        |  
    +----[SHA256]-----+  
   

   The key was generated in the current directory, not in $HOME/.ssh. I tend to move the generated key in that $HOME/.ssh directory (mv ssh_key_for_jenkins_agent_2023-03-10* ~/.ssh for me). I then change the directory to $HOME/.ssh (cd ~/.ssh) and change the permissions of the key (chmod 600 ssh_key_for_jenkins_agent_2023-03-10).
   Important note: 2048 bit is the minimal key length that is considered safe. You can use higher values, but do not use higher than 4096 as the remote server may not support that big of a key.

2. Copy the key to the remote machine (your Jenkins agent wannabee running Termux). Password authentication has to be enabled to install a public key on the remote machine. Now do: ssh-copy-id -p 8022 -i id_rsa IP_ADDRESS (replace id_rsa with the name of your key and IP_ADDRESS with the IP address of your Android machine).
   
   Alternatively, you can manually copy the content inside id_rsa.pub (public key) which is already on PC and looks like ssh-rsa <A LOT OF RANDOM STRINGS> user@host and paste it to the Termux file $HOME/.ssh/authorized_keys (remote machine).
   Remember you would have first to connect through ssh user@IP_ADDRESS -p 8022 (replace IP_ADDRESS with the IP address of your Android machine) so you can copy the content of the public key using any text editor available on PC and paste it inside an ssh session handled by Termux.
   What looks strange to me is that user could be just about anything. I tried to log in without supplying a user (which means I was using my PC username) and it worked. I also tried to log in with a different username, and it worked too. When issuing the whoami command inside Termux, it shows the username of the Termux user, which is u0_a504 in my case. I don’t know if this is a bug or a feature.
   
   If everything went fine, you will see a message like this one:

    Number of key(s) added: 1
   

   If your system has an ssh-agent, you should now add your newly generated key to the agent.
   Now try logging into the machine, with: ssh -p '8022' 'IP_ADDRESS' (replace IP_ADDRESS with the IP address of your Android machine) and check to make sure that only the key(s) you wanted were added.

   If you don’t have an agent running, you will have to use a slightly different command: ssh -i id_rsa -p '8022' 'IP_ADDRESS' (replace id_rsa with the name of your key and IP_ADDRESS with the IP address of your Android machine).
   That would give for me:

    ssh -i ssh_key_for_jenkins_agent_2023-03-10 -p 8022 192.168.1.xx  
    Welcome to Termux!
   

3. From this point password authentication can be disabled. Edit with nano the file $PREFIX/etc/ssh/sshd_config and replace the line beginning with PasswordAuthentication with PasswordAuthentication no.
   Back to the Termux app, execute the command pkill sshd && sshd to restart the sshd server with the updated configuration file. Of course, if you were to do that from your PC, you would be disconnected and the ssh server would not be restarted.

4. Now you can log in to the remote machine without a password. Just execute the command ssh -p '8022' 'IP_ADDRESS' (replace IP_ADDRESS with the IP address of your Android machine), or with the more complex command (-i) if your machine does not use an ssh agent.

Installing java on Termux

We all know that Jenkins is written in Java. We also know Android apps are written in Java or Kotlin, so we could hope that we magically could skip this step.
I’m afraid we can’t. The virtual machine that runs Android apps is not the same as the one that runs on your PC. We’ll detail later on the main differences between the two.
The Android virtual machine (called dalvik) is available on Termux, but it is not capable of executing our agent.jar file. The java command is not available yet.

$ dalvikvm -showversion
ART version 2.1.0 arm64
$ java --version
bash: /data/data/com.termux/files/usr/bin/java: No such file or directory

For the time being, let’s just assume that we need to install Java on Termux.
Let’s find out which java versions are available on Termux:

pkg update && pkg search openjdk
Checking availability of current mirror:
[*] https://packages-cf.termux.dev/apt/termux-main: ok  
Sorting... Done
Full Text Search... Done
openjdk-17/stable 17.0-25 aarch64
  Java development kit and runtime  
openjdk-17-source/stable 17.0-25 all
  Source files for openjdk-17  
openjdk-17-x/stable 17.0-25 aarch64
  Portion of openjdk-17 requiring X11 functionality

Nice. Jenkins supports Java 17 since 2.355 and 2.346.1 LTS, so let’s go with OpenJDK 17.

pkg install openjdk-17

Now the java command is available:

java --version
openjdk 17-internal 2021-09-14
OpenJDK Runtime Environment (build 17-internal+0-adhoc..src)  
OpenJDK 64-Bit Server VM (build 17-internal+0-adhoc..src, mixed mode)

Creating a Jenkins ssh agent

You should now be able to connect via ssh to your Android device running Termux (whenever Termux is running and if you have issued the sshd command).
Your ssh server also knows about the ssh key you generated on your PC. We will now create a credential based on that key within Jenkins that will allow you to connect to your Android device running Termux from Jenkins later on.

Creating a Jenkins ssh credential

For this part, there is almost nothing specific to Android. You can follow without a doubt the official documentation, and more specifically how to create a Jenkins credential.

Setting up a Jenkins ssh agent

It’s now time to set up your agent.

You could use Android as a label for your agent. Choose the Launch agent via SSH option. The hostname should be your phone’s IP address (named ‘IP_ADDRESS’ in the previous steps).
The credentials should be the ones you created in the previous steps. The remote root directory should be /data/data/com.termux/files/home. The host key verification strategy should be Non verifying Verification Strategy. The Launch method should be Launch agent via SSH.

Don’t forget to click on the Advanced button and change the port to 8022. You could also specify the path of the java executable you installed in the previous steps which happens to be /data/data/com.termux/files/usr/bin/java.
As I have installed the ‘Platform Labeller’ plugin, I have also checked the ‘Automatic Platform Labels’ checkbox. We’ll see later on if it can cope with Android devices that don’t use the lsb_release command.

The very last thing to do is to click on the Save button.
You should now see the complete list of your defined agents.
While the agent has been created, it may have not started yet.
If that’s the case, click on the name corresponding to your newly created agent (‘Android Phone’ for me) and click on the Launch button to start the agent.
After some time, you should see in the logs Agent successfully connected and online, which means you can now use this agent to run your builds.

Using a Jenkins ssh agent

Let’s create a new job and use our newly created agent to run it.
The simplest job that comes to mind is a Freestyle project that just runs the uname -a command. That should give us some information on the Android device we are running on while proving that the agent is working.
Once again, there is nothing specific to Android for this step, so you can follow the official documentation.
The only changes to the documentation I have made are:

• I have used the Android label to make sure the job is run on the Android agent
• I have used the uname -a command instead of the echo $NODE_NAME command

Started by user admin
Running as SYSTEM[EnvInject] - Loading node environment variables.  
Building remotely on Android Phone (aarch64 aarch64-unknown+check_lsb_release_installed aarch64-unknown+check_lsb_release_installed-unknown+check_lsb_release_installed android unknown+check_lsb_release_installed-unknown+check_lsb_release_installed unknown+check_lsb_release_installed) in workspace /data/data/com.termux/files/home/workspace/Android First Job
[Android First Job] $ /bin/sh -xe /data/data/com.termux/files/usr/tmp/jenkins13760213506108463207.sh  
+ uname -a
Linux localhost 4.4.192-perf+ #1 SMP PREEMPT Fri Dec 10 13:53:37 WIB 2021 aarch64 Android
Finished: SUCCESS

We now have a working Jenkins agent running on Android thanks to Termux. Now what? Of course, we will be limited to the commands and packages that are available on Termux.
For example, I can’t see gcc in the list of available packages, which could be troublesome.

pkg search gcc
Checking availability of current mirror:
[*] https://termux.astra.in.ua/apt/termux-main: ok  
Sorting... Done
Full Text Search... Done

No gcc? You’re right, no gcc in the official Termux repository, but the Termux community comes to the rescue with some repositories that provide additional packages, like gcc. After installing the repository, we can install gcc.

pkg search gcc
Checking availability of current mirror:
[*] https://termux.astra.in.ua/apt/termux-main: ok  
Sorting... Done
Full Text Search... Done
gcc-6/termux 6.5.0-2 aarch64
  GNU C compiler  
gcc-7/termux 7.4.0-2 aarch64
  GNU C compiler  
gcc-8/termux 8.3.0-3 aarch64
  GNU C compiler  
libgccjit-8-dev/termux 8.3.0-3 aarch64
  GCC just-in-time compilation  
libgomp-7/termux 7.4.0-2 aarch64
  openmp library for gcc  
libgomp-8/termux 8.3.0-3 aarch64
  openmp library for gcc-8

As you can see, we have a few gcc versions to try out. But… What if we need gcc 10 for example? I guess we would have to compile it ourselves like in the good old days. Problem solved for gcc, but what about other packages?
We will somehow be limited by the availability of the packages on Termux. What if we could work around that limitation?
What about running Docker on Termux? Docker has no limit on packages as long as we choose the right base image, right? So we could run a Jenkins agent on Termux through a Docker image based on another distribution that happens to supply all the packages we need.
The slight problem that may arise is that Docker is not easily installed on Termux, and once installed, it won’t work out of the box. But that’s for another post.

Android apps are running some kind of JVM, right? So why not use a Jenkins inbound agent?

Android apps are written in Java or Kotlin programming languages, and they run on a Java Virtual Machine (JVM) called the Android Runtime (ART) or the Dalvik Virtual Machine (DVM).

It is possible to access the JVM from an ADB shell and run Java code using the dalvikvm command, which is a command-line tool that allows you to execute Java code on the DalvikVM.

Nevertheless, there are preliminary steps that you need to take before you can run Java code on an Android device. You need to compile your Java code into a .class file, transform it into the DEX format thanks to the d8 tool, push the resulting .dex file to your Android device, and then run the Java class using the dalvikvm command.
It’s possible to some extent to automate these steps, but it’s not trivial.
Also, the dalvikvm command is a low-level tool that may not be suitable for running complex Java apps., which may have additional dependencies to function properly.
If ever that would work, it would be a very hacky solution (which is fine with me), but where would we go from there? I mean, we have a subset of the Linux commands available in the ADB shell, but we can’t install tools, packages, etc. How would we install gcc for example? So what could our Jenkins agent do? Not so much I’m afraid…
We could still use Termux; as we’ve seen earlier Termux uses the base shell that is available through adb. We can use the Dalvik VM while using Termux, so we could keep the best of both worlds (Android & Linux-like). If ever we could launch the inbound agent through Dalvik, that is. Another solution would be to create a library from the agent.jar file and integrate it into an Android app. That part could work but then the resulting agent would be even more limited… There wouldn’t be any shell available, as the app is sandboxed. We would have an agent able to do almost nothing…
I’d like to know more nonetheless, so I’ll write down my thoughts about that in another article once I’ve done my homework.