Migrate to Target API 29+ Supported Exec

[cryptrr]

Android API 29+ target does not support executing binaries from the application data folder.

So migrating to an alternate python binary packaging method will let us bump the target API to 29+.

PRs are welcome.

[robertkirkman]

Since you are currently targeting API level 28, like F-Droid Termux does,

AutoPie/app/build.gradle.kts

Line 22 in 927927b

targetSdk = 28

https://github.com/termux/termux-app/blob/7bceab88e2272f961d1b94ef736f1a9e20173247/gradle.properties#L24

But in this issue, you are stating that you would prefer to migrate to targeting API level 29+, like Google Play Termux does,

https://play.google.com/store/apps/details?id=com.termux

https://github.com/termux-play-store/termux-apps/blob/851862c7f4660938b68f39b2826b46732d9a8de5/gradle.properties#L9

would you like any integration/bootstrap/Java code copied from termux-generator to this app to come from F-Droid Termux or from Google Play Termux?

If you are unfamiliar, this can be your introduction to the situation of how F-Droid Termux and Google Play Termux are both officially Termux and both get updates, but they are different kinds of Termux,

they both get some updates around the same time, but they get other updates separately, and their code is slightly different in a lot of places,

and one of the features that Google Play Termux has that F-Droid Termux doesn't have is that Google Play Termux has a pretty advanced technique that allows it to target API level 36 and be on Google Play Store but make Python still work anyway.

however, there are some bugs that are in Google Play Termux that are fixed in F-Droid Termux, and there are also some bugs that are in F-Droid Termux that are fixed in Google Play Termux. This is because some packages are at higher versions in F-Droid Termux, and some packages are at higher versions in Google Play Termux instead. It's hard to predict which one works better for a specific task, without trying out each one.

[robertkirkman]

termux-generator can make building custom bootstraps+APKs with either Termux's code easy,

for example, using ./build-termux.sh --type f-droid, or just ./build-termux.sh (default), will use F-Droid Termux's code to build bootstraps and APKs that have Target API level 24,

and using ./build-termux.sh --type play-store will download Google Play Termux's code instead, and use it to build bootstraps and APKs that have Target API level 36.

[robertkirkman]

however, F-Droid Termux is still primarily maintained as the recommended branch of Termux because it is compatible with Android 7, 8, 9 and 10 (and also all newer Android versions). Google Play Termux's minimum API level is too high, so it is only compatible with Android 11+, and does not work on Android 7, 8, 9 or 10.

[cryptrr]

I had plans to target API 29+ but it wasn't the top priority.

I was unsure about releasing on Google play so I was thinking of using some Unix exec tricks to get it to support higher APIs but that won't play good with Google play.

But I'd like to release a fully sanitized version where users can't shoot their foot to Google play especially since it'll be deep dive into linux and great a learning opportunity for me.

Please do shoot the necessary reading material here. I'd love to have a deep dive.

[robertkirkman]

here are the discussion pages where the method that allows Google Play Termux to run almost any program but still be on Google Play Store was developed:

• Revisit the Android W^X problem termux/termux-app#2155
• Avoid executing ELF files directly termux/termux-exec-package#24

I do not personally understand every single detail of how it works, but I do know that one difference from F-Droid Termux is, it involves the use of more direct invocation of /system/bin/linker64 than F-Droid Termux does (/system/bin/linker64 is also required for a device to be able to run F-Droid Termux, but it is used differently in Google Play Termux), which is preinstalled in all Android devices, to run programs, which for some reason, if set up in a certain way I don't know further details about, is sufficient to pass Google's Play Store W^X-violation policy checks and resulted in the new builds of Google Play Termux appearing on Google Play around mid 2024.

I believe a stripped-down example of how Google Play Termux runs binaries could be similar to this command, but I'm not entirely sure:

/system/bin/linker64 /data/data/com.termux/files/usr/bin/ls

That part is abstracted through a large amount of boilerplate in the termux-exec library, which is applied to both Termux and Google Play Termux through LD_PRELOAD, but I believe termux-exec is only absolutely required for Google Play Termux to work, since F-Droid Termux still mostly works if termux-exec is removed, but just becomes glitchier (since in F-Droid Termux and Google Play Termux both, termux-exec is also used to auto translate script shebangs from #!/bin/bash to #!/data/data/com.termux/files/usr/bin/bash when Termux executes scripts).

keep in mind that "users can still shoot their foot" with Google Play Termux because it's actually not really that much more restricted from the perspective of users than F-Droid Termux, you can still run a huge amount of commands, it just doesn't work with some things but it does work with most, or at least a little more than half of all things in my experience.

[robertkirkman]

I had plans to target API 29+ but it wasn't the top priority.

I'll try integrating F-Droid Termux/termux-generator in F-Droid Mode (--type f-droid) to AutoPie first, and if that works, then it's usually not really too hard to rewrite another variant/branch of it that works with Google Play Termux also.

F-Droid Termux currently has a more stable package manager than Google Play Termux, but that difference is more of a temporary bug than a really long-term problem, so basing the code on F-Droid Termux first is probably the best idea.

[cryptrr]

The W^X issue is where I got the idea to create AutoPie. Initially I wanted to sidestep the issue by running pure python packages without any need of exec but found that severely limiting. This is why I went with shiv to package all the binaries and python wrappers into a single file and make a self contained system with its own PATH variables.

I thought about making simple pure python integrations and making a repo where people could search and install them. But yeah nothing matches the sheer power of calling binaries (because everything is already done for you) so only if Android totally puts the hammer down on all exec will everyone will be forced to move to pure interpreted code.

[robertkirkman]

I thought about making simple pure python integrations and making a repo where people could search and install them. But yeah nothing matches the sheer power of calling binaries (because everything is already done for you) so only if Android totally puts the hammer down on all exec will everyone will be forced to move to pure interpreted code.

Your app supports pip install to get packages from PyPi, but unfortunately, not all packages from PyPi are pure Python packages. Many of them depend on at least a small amount of C, C++ or Rust, and some PyPi packages are just a thin Python wrapper around a large program in C or C++.

I hope that if done well, integrating Termux with AutoPie will allow all fixes that Termux maintains for Python on Android to become available for AutoPie also. It will take me some time to set up a working initial demo, but I think it won't take me more than a week.

Google might put in more restrictive Play Store policies, but I don't think Google will ever do something that makes the sideloaded/F-Droid version of Termux completely stop working on a future Android version, because that would severely restrict the very powerful abilities of Android when all community apps are considered.

[cryptrr]

True that google do not want power users leaving the platform. That must be the reason they are coming up with KVM virtualized containers in Android 16 and forcing OEMs to adopt it.

KVM containers are actually great in that it allows to run things that was impossible to do in termux.
It also would need not be built with Android ndk and any arm64 binary would suffice. When we have entire termux packages built with the ndk.

So in the future, Google might be looking to make Termux and other shell based applications into moving their backend shell execution into an 'isolated' container for Google Play apps. But it's the level of isolation that Google wants us to accept will be what makes or breaks that feature.

[cryptrr]

@robertkirkman Is there anything you'd like from me to make your work easier? please do tell. I'm really stoked by the idea of termux integration. It'll be so powerful.

[cryptrr]

• Easiest way to integrate with the Termux repos without AutoPie's own repo? #6 (comment)

@robertkirkman How exactly does the termux apt prevents connections to the termux repos when the package name is changed. Is that done completely based on patches to the apt source or are there other tricks involved?

[robertkirkman]

I will give a status update on what I am doing so far,

Unfortunately I have to admit that, sort of to prepare for AutoPie but also because I was more comfortable doing it first, I have been working on finishing the full implementation of X11 support in termux-generator,

since the version of termux-generator currently public in my repo works with the Terminal App and CLI programs, but it doesn't actually work with Termux's official GUI (Termux:X11) yet, I admit that I felt more comfortable making sure that X11 is implemented in termux-generator first, then adding a second GUI (AutoPie) second,

The good news is that locally, I have pretty much finished all the code for the the implementation of X11 in termux-generator. I am just finishing up all the final details, before I upload the next version of termux-generator. It'll enable building Termux-based apps that each work with their own paired APK of Termux:X11 (and after AutoPie and Termux are integrated, it would even allow running X11-related commands from AutoPie to control the X11 server).

fourtermuxx11.webm

@robertkirkman Is there anything you'd like from me to make your work easier? please do tell. I'm really stoked by the idea of termux integration. It'll be so powerful.

I think that one thing that might help is if you could try creating a branch of the AutoPie source code that has the whole GUI interface with the buttons and can compile to a working APK, but with the original bootstrap and terminal code removed, so that it can more easily be replaced with Termux - but you don't need to rush to do that yourself, especially if your plan is to try to keep the original bootstrap in the app as a fallback/backup.

I was planning to do that first when I start editing AutoPie's code, but if you already know exactly how to set that up, then it might save me some time.

I'll also take into consideration the bugs you opened here, and while making AutoPie + Termux integration, I will make sure to test the conditions in those bugs to make sure that none of them are reproducible while using AutoPie + Termux.

[robertkirkman]

@robertkirkman How exactly does the termux apt prevents connections to the termux repos when the package name is changed. Is that done completely based on patches to the apt source or are there other tricks involved?

Unfortunately, that topic is very technical and is very hard to explain in a comprehensible way.

I will try to summarize some points that can hopefully make this clearer for you:

• Using pkg install or pkg upgrade (install any new packages or upgrade current packages) in any Termux build that uses the default mirror, downloads .deb files from the official mirror, https://packages-cf.termux.dev/

• Inside of every .deb file that comes from https://packages-cf.termux.dev/, there are many, many instances of the string "com.termux", simply everywhere throughout. Shebangs, C program runtime library search paths, configuration file paths, TMPDIR paths, asset file paths, every kind of place where there could ever be an absolute path, is often patched in Termux packages.

  • It is important to realize that while some paths, like shebangs and library search paths, could be patched by editing a precompiled APK or binary, those methods unfortunately miss a lot of edge cases, and there is no currently-known method of patching already-compiled APKs that can handle all edge cases

• The only truly reliable way to replace every single one of these instances of "com.termux" is to download 100% of the source code of every single binary and library in the entire Termux App and its bootstrap with no exceptions, and run a script that automatically detects every one of these instances of "com.termux" in the source code before compilation, and replaces it with the desired name, then recompiles and repacks every single precompiled binary that is desired, into every single package that is needed in the entire app's bootstrap.

• That process is essentially what termux-generator does, which replaces every single instance of "com.termux" that needs to be replaced, and carefully avoids replacing every instance of "com.termux" that shouldn't be replaced, in the entire source code of Termux, then compiles the entire resulting source code in a way that fixes many errors that would otherwise happen.

• Putting all these details together, the conclusion is that any app that uses Termux source code but which is installed in a folder that is not named "com.termux", must do at least one of the following:

  • include all desired native packages pre-patched and pre-compiled
  • and/or must have its mirror https://packages-cf.termux.dev/ replaced with a 3rd party or self-hosted mirror that hosts packages compiled for the specific name that matches the folder the Termux-based app is installed in
  • and/or include packages like udocker, proot-distro that can be used to launch rootless containers of essentially entire other distros (like containers as you know them on PC or GitHub Actions), like Debian, Ubuntu, or termux-docker

[cryptrr]

The X11 integration looks great. I'd love to test that out.

All the Shell Operations are done in the ProcessManagerService class. It has an runCommandForShareWithEnv2() function that is called once from the useCases.

And the the shell initialization is done in the getShell() inside the same class

data class AutoPieUseCases (
    //This is the generic command runner that calls every other type of command
    val runShareCommand: RunCommand,

    val runCommandForDirectory: RunCommandForDirectory,
    val runCommandForUrl : RunCommandForUrl,
    val runCommandForFiles: RunCommandForFiles,
    val runCommandForText: RunCommandForText,
    val runStandaloneCommand: RunStandaloneCommand,
)

What these usecases does is simply that it transforms the data from the viewModel for running the runCommandForShareWithEnv2()

Can you me send me bootstrap package. I can make a quick and dirty version which can initialize the Termux bash instead of the sh and set the env variables to load to init the shell.

We can swap out the issues and functions then as we go.

[robertkirkman]

Can you me send me bootstrap package. I can make a quick and dirty version which can initialize the Termux bash instead of the sh and set the env variables to load to init the shell.

We can swap out the issues and functions then as we go.

I'm sorry for not sending you one yet, I know that it is frustrating that I take a long time to do things. I'm going to try to make an effort not to take so long for some things.

What I think should be helpful is that really soon, I'm going to try to upload a CI workflow to termux-generator that can generate an entire termux APK and release it in the Releases page of termux-generator.

Also, I'm not sure if you're familiar with this, but do you know how when you extract an official Termux APK, there's not a normal bootstrap .zip in there that you can open, it's a different file that's weird? (.so file)

termux-generator generates APKs that are a little bit different from normal Termux APKs in that, they actually always have the bootstrap inside as a normal .zip file in the assets folder,

so what this means is, if I get the GitHub Actions workflow to work, all I should need to do is get it to publish an APK built with the specific package name you ask for, or even make it so that you can fork the termux-generator repo and just enable GitHub Actions in it, and generate them automatically through a workflow_dispatch interface,

then all you'd need to do would be to extract the APK with 7zip, Winrar etc, and in the assets folder there would be a whole bootstrap, or four, one for each architecture, if desired.

(if you are not familiar with GitHub Actions, this explanation is basically that I'm going to try to send you a button in GitHub Actions you can type the app name and packages in and click to just generate an APK + bootstrap inside, which would look something like this)