Android 10 Developer Changes.

[canewsin]

How can we make Android 10 Compatible Binary

Untrusted apps that target Android 10 cannot invoke exec() on files within the app's home directory. This execution of files from the writable app home directory is a W^X violation. Apps should load only the binary code that's embedded within an app's APK file.

Obviously we store the output binaries in home dir of app, because python in bin dir, points to lib folder.

I was currently using termux system for binary compilation they had the same issue, since this is python only bins, I want to migrate build system to this repo. but cannot find solution for it.

related wiki : https://github.com/termux/termux-packages/wiki/Termux-and-Android-10

[canewsin]

more links :
https://developer.android.com/about/versions/10/behavior-changes-10#execute-permission

[yan12125]

Thank you very much for the info. Most likely the solution will be building libpython.so on PC and load it with dlopen(). However, there are reports that this approach does not work (#18).

[canewsin]

Thank you very much for the info. Most likely the solution will be building libpython.so on PC and load it with dlopen(). However, there are reports that this approach does not work (#18).

Can we able to make bin/python and lib/libpython.so as single file ?

[yan12125]

I guess you mean making bin/python a standalone executable, so that it can run without loading libpython.so? It's possible (by building python without --enable-shared), but I doubt it will help as you still need to exec() bin/python.

[yan12125]

As a side note, iOS has similar exec() restrictions years ago. Efforts on runnuing Python on iOS (https://bugs.python.org/issue23670) may help.

[canewsin]

Do you have any idea on how can we overcome this ?
References of Issue :

Removed execute permission for app home directory
Untrusted apps that target Android 10 cannot invoke exec() on files within the app's home directory. This execution of files from the writable app home directory is a W^X violation. Apps should load only the binary code that's embedded within an app's APK file.

In addition, apps that target Android 10 cannot in-memory modify executable code from files which have been opened with dlopen(). This includes any shared object (.so) files with text relocations.

[canewsin]

I'm Curious because your repo obviously doesn't work anymore on android devices when app is targeting android 10, as well as helpful to me.

[yan12125]

Do you have any idea on how can we overcome this ?

What to overcome? Building libpython.so, embedding it into APK and load it should still work fine with apps targettong Android 10.

[canewsin]

Generally lib-dynload dir contains all native modules can we able to compile them as single so bin file.

[canewsin]

Thank you very much for the info. Most likely the solution will be building libpython.so on PC and load it with dlopen(). However, there are reports that this approach does not work (#18).

I already compiling them on pc, but their placement is different, lastest update is i compiled python without --enable-shared, but has a doubt that will it contain dynload modules or not !

[yan12125]

Generally lib-dynload dir contains all native modules can we able to compile them as single so bin file.

I'm afraid there is no simple way.

[GRRedWings]

My fork is out of date and not as current as this one, but I embed libPython in the app, and it runs without any problem on Android 10. So embedding it is always an option.

[yan12125]

I embed libPython in the app, and it runs without any problem on Android 10

Great to hear that! Mind to share an example app for testing? Frankly, I'm not familiar with Java and creating an Android app is somewhat hard to me.

[GRRedWings]

While back we tried building your library the same as my fork and never succeeded. I can try to build a sample app, but until we figure out what failed last time, we won't be all the way there. B --------------------------- Sent from my phone. Please forgive the short nature of the message, if needed I will follow up with more details.

…

On Fri, Jan 31, 2020, 10:43 PM Chih-Hsuan Yen ***@***.***> wrote: I embed libPython in the app, and it runs without any problem on Android 10 Great to hear that! Mind to share an example app for testing? Frankly, I'm not familiar with Java and creating an Android app is somewhat hard to me. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24?email_source=notifications&email_token=ABB63V765F6Y742Y4CCMBFTRATVV7A5CNFSM4KKBPCMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKQSU2A#issuecomment-580987496>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB63V64AAEHYQCBOOEZ5CLRATVV7ANCNFSM4KKBPCMA> .

[yan12125]

Ah, I was asking for a sample app that works with your fork, not mine :) I hope such an app can me testing this issue and #18.

[GRRedWings]

Gotcha, I'll work on that. My app is way too complex to be an example app

…

--------------------------- Sent from my phone. Please forgive the short nature of the message, if needed I will follow up with more details.

On Sat, Feb 1, 2020, 8:43 AM Chih-Hsuan Yen ***@***.***> wrote: Ah, I was asking for a sample app that works with *your* fork, not mine :) I hope such an app can me testing this issue and #18 <#18>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24?email_source=notifications&email_token=ABB63V3E7O2H63PXBGX4U2TRAV36JA5CNFSM4KKBPCMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKQ5NRI#issuecomment-581031621>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB63V2RVQELQBJKB5S5FA3RAV36JANCNFSM4KKBPCMA> .

[GRRedWings]

I'm still working on it. I only have an hour or two here and there to work on it so it's slow going.

[yan12125]

No hurry :)

[canewsin]

since python is build as PIE bins there is no problem, and where we put those assuming we load libpython.so while executing python, i now successfully running normal python scripts one thing bothering me is, i was not able to figure how to compile native modules of python like gevent

[GRRedWings]

I have pushed a sample app. Hopefully that helps https://github.com/GRRedWings/python3-android/tree/master/AndroidApp

…

On Wed, Feb 5, 2020 at 6:35 AM Chih-Hsuan Yen ***@***.***> wrote: No hurry :) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#24?email_source=notifications&email_token=ABB63V4QUIQTOMNWOF5FDBTRBKP73A5CNFSM4KKBPCMKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEK3DOPQ#issuecomment-582367038>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABB63VZPHXLMPVHD5NTBQSDRBKP73ANCNFSM4KKBPCMA> .

[yan12125]

From termux/termux-app#1072 (comment),

This probably will go away in future, current implementation bans execve() from data but still allows dlopen()ing (System.load in Java) downloaded files

As Python libraries (libpython3.X.so, lib-dynload/*.so) are accessed via dlopen() as demonstrated in @GRRedWings' sample application, Android 10 changes are currently not an issue. If one day dlopen() is also restricted, we can restart this discussion.

[hwangjr]

Anyone have any ideas to execute python download from remote? Now for android 10, the only way is to put the executable python to jniLibs, but the python library is very huge.

[GRRedWings]

I find using the so adds around 26MB to support 32 and 64 bit arm. Not insignificant, but you're going to need most of that somewhere on the device anyways.

[hwangjr]

now my app is only 6MB, and if add python support, 25MB is very huge for the app. Does android 11 support to exec python downloaded from remote?

[GRRedWings]

Right, your app is 6 MB, but it's going to need to get Python from somewhere, and that will have a cost of download and space correct?

I have only used it as an embedded library, but I suspect if Android 10 requires it embedded, it is for security and no future release will go back to remote.

[hwangjr]

okay, thanks a lot for your reply.

[GRRedWings]

Have you tried to target something less than 10. I have found that if you target less than an API version, functionality that was taken away by the OS update will still work.

[hwangjr]

yep, there is no solutions for this issue without make target sdk below 29 or put executable file to libs/ABI folder.

[canewsin]

currently i put python bin jniLibs and libs in assets dir and serve them as separate files and serve them as dynamic modules even though apk size may not decrease for other than play store builds download size in play store.
see this repo for example
https://github.com/canewsin/python_ide

[hwangjr]

if i put executable file: libpython3.8.so to libs/ABI, and download the dependencies from remote server(now you put them in assets folder), will this work? @canewsin

[canewsin]

That will work, I do that for non Play Store builds, since dynamic modules are not downloadable outside I pack those as zip files and download and unzip those and chmod +x if necessary.

[yan12125]

As a record, there are more discussions about this issue on termux/termux-app#2155. I'm afraid all workarounds will be broken sooner or later. For example,

target something less than 10

The current Google Play Store policy already requires Android 10 as the minimum target, and Android 11 will be required since August. A long term solution might be alternative application stores like F-Droid...

[GRRedWings]

Another option is expansion files. As @canewsin described, you don't package it in your APK but download it later. This works with the play store. It may be possible you could be smart about your platform and only download the necessary version.

https://developer.android.com/google/play/expansion-files