Running DexPatcher-tool on-device (Android and TWRP)

[ShapeShifter499]

There might be a case where there is a critical upgrade for a bug but you don't have a computer around so you have to chose between a OS update or MicroG services support. The question is, if we have root why can't we patch these files on the device itself?

[Lanchon]

hi,

it's entirely possible to patch in-device. in fact, the patcher could use the 'addon.d' mechanism to patch automatically after every rom flash, just like gapps self install after every rom flash.

the issue is that haystack is not a project i care about or want to invest time in, it's just a dexpatcher demo.

the problem with this approach

if you ask me, patching roms like this shouldn't be needed. projects like cyanogenmod should go extint. CM has a horrible track record of choosing google, their marketability, an app developers over CM users. they seemed to believe android had to be "fair" to developers and "play nice" in the ecosystem. they did horrible stuff like this:

• crippled CM one day, in preparation for their conversion to Cyanogen Inc, by removing all code that was GPL or had a non-proprietizable license; CM lost important features to users because of this.
• rejected patches that lied to apps to protect user's privacy. for instance, lying about location, contacts, accounts, phone number, ESN, IMEI, AndroidID and the list goes on forever.
• rejected OpenPDroid, another set of privacy oriented patches with more or less the same aim.
• rejected signature spoofing for microG.

and many, many more. the rationale is always fairness to app developers or some other equally stupid excuse, as if app developers owned your phone and your phone should be poised to help them. even as if app developers could be trusted!

the reality is that they rejected every change that would make CM not aligned with Google's idea of what android should be, a necessary step to be able to license Google's Play Store and other proprietary software from them.

what is needed is a community that cares about freedom (it doesn't exist, they only care about immediate features) and defends themselves from malevolent dictators such as steve kondik. the followup for cyanogen, call it X Android, should have these characteristics:

• no copyright assignment agreement enforced to accept submissions. XA.org must NOT be able to relicense XA, because it must not own XA; the people, the contributors, must own XA.
• all additions to the codebase must be accepted in a set of predefined FLOSS licences. by default the GPL v3 licence should be used for contributions, unless the code already has another FLOSS license. this means that XA will not be proprietizable by any company: nobody will be able to take features of XA and bundle them in a proprietary software phone. if they want XA features, they'll have to make the device and all its OS free.

these safeguards ensures that the commercial value of XA is close zero, and precludes business-oriented decisions that hurt XA users.

back to haystack and dexpatcher

dexpatcher has a million ways to improve. i have so many feature's i'd like to implement in my head. unfortunately my time is limited, and there seems to be very few people interested in dxp anyway; and that demotivates me.

one issue with dxp is that it was not designed to handle various codebase versions to patch on. my use case for dxp is this:

• i patch an app.
• if the app gets an update, maybe i'll update my project and rebuild. dxp does a great job of alerting me of any inconsistencies with the new codebase, so i can update my patch to be compatible with it. (however, it could still improve some more).

dxp was never designed for the following case:

• build a patch that applies cleanly and safely not to one, but to various codebase versions.

in haystack the corollary of this is seen as the individual "hook" patches that adapt haystack to each OS version.

so if i ever implement dexpatcher changes i have in my head to support "multi-targeting", haystack could be unified into a single patch that applies cleanly to all OS versions.

in that situation, making an in-device patcher would be much simpler. i still would be hesitant to use my time for that goal, but maybe...

some final notes

keep in mind that patching takes time. on average phones (slow), it wouldn't surprise me if patching took around 5 minutes, even after all the optimization work i put into dxp.

odexed roms still cannot be patched, so haystack is not an ideal solution. i've heard that Android 7 is mostly non pre-odexed. finally!!!! that odex crap retired from the platform!! this change, if confirmed, would increase the value of haystack.

patching is trivial to do in-device on android, because android can readily run java bytecode (via dalvik bytecode conversion), but not so easy to do in recovery. to patch in recovery (ideal) you need a java runtime (JRE) that works in recovery. such VMs probably exist but one has to be built for the environment and each supported architecture. these VMs tend to be simple implementations and thus very basic in their optimization strategies (read: bad performance). so the 5 minute patching could well turn into 15 minutes in these low quality VM implementations.

of these VMs, i favor "avian" for its build simplicity. i tested dexpatcher on amd64 avian (works just fine) and benchmarked it, and was pleasantly surprised to find good performance: not that far from the openjdk jre numbers.

[Lanchon]

if you want to make a solution yourself for current haystack, you can.

dexpatcher runs just fine in the dalvikvm. take a look here to see how:
http://stackoverflow.com/questions/10199863/how-to-execute-the-dex-file-in-android-with-command

[Nanolx]

I worked something out. In

https://github.com/Nanolx/NanoMod

you'll now find the sub directory framework-patcher this is indeed an on-device framework-patcher. You can use it in two ways:

• flashable zip
• manually run commands

to create a flashable zip issue mod.sh patcher which will create the zip from the repo and flash as usual

manual instructions:

adb push framework-pacher/dexpatcher /tmp/dexpatcher/
adb shell
/tmp/dexpatcher/framework-patcher.sh

current issues:

• when flashing as zip framework-patcher and dexpatcher output is not shown
• only tested on one device currently (Galaxy Tab 4)
• may not work on x86/x86_64 (because I have not expanded LD_LIBRARY_PATH)
  • who uses that anyway?

This is the first try. I hope it works for most out-of-the-box if not, provide me /tmp/recovery.log and more infos at the NanoMod repo.

[Nanolx]

Ah... about the speed: on my Galaxy Tab 4 that process takes about 3 - 4 Minutes.

Edit: on my Moto X Play it also takes about 3 Minutes. Quite speedy this dexpatcher thingy. 🥇

Another note: the framework-patcher is able to install the patched services.jar into three places:

• NanoMod Magisk Module (if it's installed)
• NanoMod microG Module (if it's installed)
• directly into /system/framework (fallback)

[ShapeShifter499]

@Nanolx Does this mean I can go down to having to only flash four things whenever LineageOS (or other rom) has an update all without a computer?

For example
The rom update zip
Magisk
The NanoMod Module
The zip that initiates the dexpatcher

[Nanolx]

Almost:

• ROM
• Magisk
• Patcher

NanoMod stays intact until you wipe /data (regardless of Magisk re-install).

That is, if the patcher zip works for you, too. If that's the case: no more pc needed.

I uploaded a pre-built patcher zip to the NanoMod afh folder, btw.

[ShapeShifter499]

@Nanolx thank you! I'll give it a try on the next LineageOS release since I just got everything setup. Maybe next week though since their "nightly" builds are more like weekly builds.

[Nanolx]

I pushed an updated version to github which fixes several issues and uploaded a new zip to https://www.androidfilehost.com/?a=show&w=files&flid=150729 (it's the "NanoMod-patcher-YYYYMMDD.zip" file)

[Nanolx]

@Lanchon I noticed the following regarding dalvikvm/dexpatcher on device:

• some devices need to boot atleast once into the ROM after full wipe, else dalvikvm fails to initialize
  • happens on Galaxy S6, but not on Galaxy Tab 4 or Moto X Play
• some devices don't need to boot into the ROM once after full wipe, but they take awfully long to patch

Since my technical knowledge about dalvik is ... not so much, maybe you can sched some light into this?

I will try with avian and see how that works out.

Edit: updated post.

[Lanchon]

@Nanolx,

you'll now find the sub directory framework-patcher this is indeed an on-device framework-patcher. You can use it in two ways:

flashable zip

dalvikvm is not available in recovery. so what is your zip doing? does it schedules dexpatcher to run on next boot? or are you trying to setup an environment in recovery that is able to run dalvikvm?

current issues:

when flashing as zip framework-patcher and dexpatcher output is not shown

hehe, use Flashize for that!

some devices need to boot atleast once into the ROM after full wipe, else dalvikvm fails to initialize
happens on Galaxy S6, but not on Galaxy Tab 4 or Moto X Play
some devices don't need to boot into the ROM once after full wipe, but they take awfully long to patch

well firstly i don't know how you are running dalvikvm. i also don't know which version of android you are testing on. finally i don't know dalvikvm well either lol.

i believe the dalvik vm does not exist anymore since Android 5 and ART. so dalvikvm must be an interface to ART. it looks like ART in Android 6 is unable to run dalvik code directly and hence the long first boot time and apk install time, in which ART is compiling everything to native code. so there is no more VM? i don't know. maybe the VM is kept for compatibility with dynamically loaded bytecode.

so what happens when you invoke dalvikvm in Android 6? who knows! maybe the bytecode is compiled ahead-of-time into a temporary or cached OAT file. or maybe the bytecode is interpreted by the vestigial VM. or maybe it is also JIT'ed, but i think JIT is not present in Android 6.

by your account, it looks like in some devices the framework (including java.* packages) might not be oated, which would happen in first boot, and either dalvikvm fails to run or interprets everything, slowing to a crawl. or maybe it is oated, but dalvikvm can't find the oats because it's not configured properly yet. or maybe dalvikvm always compiles code passed to it Android 6, but it is unable to compile it before first boot and fails or runs dexpatcher terribly slow.

in Android 7, it's even more loose, because we know it has a working interpreter, plus JIT, plus AOT compiler. so at least one should expect dalvikvm not to fully interpret dexpatcher.

i can say one thing: if you run dex2oat on dexpatcher before running it should run much faster than what you've seen!! or not, lol, who knows!

whatever you find out, do tell us :)

I will try with avian and see how that works out.

avian's classpath is unable to run dexpatcher. it is very incomplete and dexpatcher wasn't written to support that scaled-down runtime.

but... avian-pack does! and, considering that it's a simple VM, it even runs at an acceptable speed; i've tried it. unfortunately avian-pack -supposedly arm compatible- is distributed pre-built only for intel, which is a drag. i gave a try to building it for arm but the process failed and i didn't have time to continue. supposedly you can build a completely statically linked fat binary with even the android classpath in it: a complete Java VM in a single file that you can use anywhere, including in recovery. and if you succeed building a static avian-pack for arm, i'll give you a big wet kiss.

[Nanolx]

@Lanchon Thanks!

My zip mounts /system and runs the dalvikvm command from the ROM, on clean flash it takes ages - as you said because everything is still in unprepared state - on an used ROM it's pretty fast (only 3 - 4 Minutes on mid-range devices like Galaxy Tab 4 or Moto X Play).

I didn't know about Flashize, but I managed to wrap around the shell output like

print () {
  print_stuff_and_catch_error
}

command() {
  do_stuff || on_error
}

(command | print) || exit

I haven't tested with dex2oat, will check that.

The build-process for avian-pack currently runs, dunno what comes out.

Edit: you're right the build process is a mess:

• uses bashisms but does not enforce /bin/bash
• sets CC="gcc -fPic" on linux so the configure scripts looks for "/usr/bin/gcc -fPIC"
• re-defines _POSIX_C_SOURCE
• undefs GNU_SOURCE but does not redefine thus missing definitions
• many other errors

[Nanolx]

I made avian-pack finally compile on my linux-x86_64 box, but not (yet) an cross-compiled one for use on android.

• patch for avian-pack
• patch for avian sub-module
  • apply after make git-refresh

In case you're interested.

[Lanchon]

definitely use flashize! it takes a shell script and outputs a flashable zip. output from the shell script gets displayed in recovery. there's also optional output logging, zip extraction, and setting up environment (eg, binaries for the script, such as busybox).

also take a look at this:
https://justanapplication.wordpress.com/2014/09/30/building-the-android-runtime-art-for-mac-os-x-part-eight-testing-testing/
it seems art is a higher-level interface to the runtime than dalvikvm. but now that you are working on avian-pack, this is irrelevant.

[Nanolx]

OK, I'll have a look at both.

Currently trying to build avian-pack for arm, let's see how this will work out.

[Lanchon]

why (char*) strerror_r(error, buffer, length); ?
strerror_r returns an int, not a pointer.

[Lanchon]

hey, have you made any progress?

[Nanolx]

I spent my time with different things, for now I have not succeeded in creating a static build of avian-pack.

But I further improved the dalvik-vm based one, so that the only left issue is that dalivk-cache needs to be in place (= boot into ROM first).

[Lanchon]

hmmm... so no big wet kiss for you then :-p

[ShapeShifter499]

I've been using @Nanolx's dalvik-vm based patcher for a while now. It's pretty easy to update to a new LineageOS nightly release now when I'm away from a computer.

[Nanolx]

@Lanchon maybe I get a kiss soon(tm)?

I finally managed to create a dalvik-vm based patcher, that does not require ROM boot, instead instantly works. At least for me, need field-tests.

For details check out: https://forum.xda-developers.com/showpost.php?p=73451936&postcount=1142

• GitLab [Main]: https://gitlab.com/Nanolx/NanoMod
• GitHub [Mirror]: https://github.com/Nanolx/NanoMod

see the patcher sub directory.

Next step would be addon.d support, once the "new" patcher has been proven to be working for at least most users.

Well yes, no avian, but dalvik is fine, too, no?

[Lanchon]

lol on the kiss... crap! you remembered that!!! lol

so looking at your code got me thinking... how hard would it be to bundle a complete android 2.x or 4.x dalvik vm in a flashable zip?

anyway some comments:

this is not needed, dxp already defaults to the right number of jobs in --multi-dex-threaded mode:
https://github.com/Nanolx/NanoMod/blob/777c95e920456c88c4ab9eaa901c624a9e05b20d/patcher/META-INF/com/google/android/update-binary#L167-L170

consider letting go of ui_print() and other recovery script crap by using Flashize:
https://github.com/Lanchon/Flashize
it can convert shell scripts into flashable scripts.

among other things, this means that using the same script, you can adb push it and adb shell run it for testing, or turn it into a flashable zip for releasing.

for example, i developed REPIT as a regular shell script that i test-run using a 'run' script and release using a 'make' script:
https://github.com/Lanchon/REPIT/blob/89830c6a4bc703a2535a15bdc4c631594a6765bd/run#L22
https://github.com/Lanchon/REPIT/blob/89830c6a4bc703a2535a15bdc4c631594a6765bd/make#L29

[Nanolx]

I've dropped the cpu core check accordingly.

I'll have a look at Flashize soon(tm), but first I want to ensure everything works as-is currently.

The problem with a bundled DalvikVM would be SDK support I suppose, because Google made several changes with each new SDK, I doubt that an SDK 26 DalvikVM works on SDK 19 or vice-versa. Using the ROM's built-in DalvikVM avoids that issue.

[Lanchon]

flashize - flash-me.zip <<< 'echo "Hello World!" "$@"'

:)

[Lanchon]

The problem with a bundled DalvikVM would be SDK support I suppose, because Google made several changes with each new SDK, I doubt that an SDK 26 DalvikVM works on SDK 19 or vice-versa. Using the ROM's built-in DalvikVM avoids that issue.

but we don't want to support any android sdk, just java. we take android's vm, bionic, etc from some old android (with or without jit, depending on how large we want the zip to be) and use to run dexed pure java code.

[Nanolx]

So, for the record: I've added addon.d support.

(Once it works as it is now, I'll have a look at flashize.)

[Lanchon]

thanks, great! im not home, so im slow to respond. (i keep on adding stuff to dexpatcher anyway :)

[Nanolx]

Using official Pixel C LineageOS I've succesfully tested 7.x patches (core, hook, ui) against 8.0 services.jar/Settings.apk, so you can use them as 7.0-8.0.

https://forum.xda-developers.com/showpost.php?p=73577659&postcount=237

[Lanchon]

hmmm... thanks for letting me know. this is surprising to say the least, as android O introduces dex file version 038. this dex version supports the equivalent of java's invokedynamic.

dexpatcher is ready to support dex 038, but dexlib2 (a part of smali project used by dxp) does not yet support 038 so an Oreo release of dxp is blocked for now.

so it is interesting that lineageOS Oreo is building 037 dex files... the VM must support 038 for sure, so i guess some Oreo androids built differently might produce 038 and be incompatible with haystack, while others work. so IMHO we can't really declare compatibility until such time that there is an Oreo dexpatcher.

meanwhile it's great to know that at least the patches wont require changes :)

[Lanchon]

hey @Nanolx,

just wanted to let you know: dexpatcher-tool since forever has had the ability to apply several patches in sequence in one go. this produces the same end result that a series of independent applications would but is enormously faster because 95+% of the patch time is actually spent building the output dex file(s).

this sequence:

dexpatcher -o patched1.dex source.dex patch1.dex
dexpatcher -o patched12.dex patched1.dex patch2.dex
dexpatcher -o patched123.dex patched12.dex patch3.dex

is equivalent to:

dexpatcher -o patched123.dex source.dex patch1.dex patch2.dex patch3.dex

only the latter executes 3 times faster.

by the same token, you can check if patches will apply cleanly or produce errors very quickly by asking dexpatcher not to produce output in this way:

dexpatcher --dry-run source.dex patch1.dex patch2.dex patch3.dex

you can add approximately 5% processing overhead at the beginning of all patching to check that all patches will apply or otherwise fail fast.

[Nanolx]

I haven't thought about that, will check it out.

[cosmicdan]

but we don't want to support any android sdk, just java. we take android's vm, bionic, etc from some old android (with or without jit, depending on how large we want the zip to be) and use to run dexed pure java code.

I've done this kind of thing a while back; I made an on-device odexer/deodexer for internal purposes back when I worked for a company that customized China ODM devices and sold them rebranded globally (it was build-in to the Recovery system for multi-targetted OTA's on odexed devices). I remember having a lot of trouble with getting binaries working. I'd mainly be interested in trying to get Dalvik from AOSP 2.2 in since it was the first DVM with JIT, but there are a few initial concerns:

1. Will such an old DalvikVM correctly compile to ARM64 PIE binary?
2. Does DexPatcher target the 1.6 JDK or 1.7? If the latter, the old DalvikVM would only support 1.6 bytecode, right?

I suppose I'll have to try it out. Pretty sure it's going to be much more complicated than dumping the dalvikvm sources from AOSP 2.2 tree into AOSP 6.x build tree though 😜

[Lanchon]

i was thinking more in line of grabbing appropriately licensed pre-built binaries, say from old cyanogenmod builds.

dxp is mostly java 6. but since 6 is deprecated everywhere (even open jdk 7 i can't make it work due to old encryption stack), i gave in to temptation and started using try with resources. try with resources needs library support (the Closeable interface). but it would be trivial to revert, maybe just one commit. the problem would be dependencies of dxp, which might require 1.7. besides try, i've been using other language-level java 7 stuff that does not require library support (eg, the diamond operator), for which no revert is needed (magic is in javac 7).

so revert of dxp to java 6 is a 10 minute deal, but i consciously started using 7 because i feel that dependencies are already tying me up to 7. if not, they might soon. so the take away is: java 6 is too old. ideally a solution would support java 7.

EDIT: my dependency is dexlib2, which brings in guava and others. if dexlib2 (plus deps) is java 6, i should be java 6 too.

[cosmicdan]

OK, according to this StackOverflow thread, it’s suggested that dex’s can be compiled from Java 1.7 and still work down to minSdk 8 – unless the source Java is using try-with-resources in which case it will require minSdk 19 (KitKat). Since KitKat is the last version that seems to have the real/full DalvikVM, it’s probably a better idea to use it’s version. Using CyanogenMod prebuild binaries would work, I’ll just need to find a CM11 M12 firmware for ARM64... EDIT: Oh, ARM64 binaries not required - ARM will work just fine, the resulting dex's are the same for both platforms, of course.

[Lanchon]

yes, many language features of java 7 are sugar totally resolved by the compiler (so they work over previous implementations of the java classpath) but some require updated libraries. at the VM level, java 7 also introduces invokedynamic, but javac 7 never emits that opcode. (javac 8 final does, to implement lambdas. javac 8 prereleases used anonymous classes instead.)

and yes! there's no reason to go for an arm64 build, arm should run just fine. unfortunately this leaves x86_64 unsupported...

[Nanolx]

Did you happen to test on Oreo on-device yet? Patching now takes about 3 to 4 times as long as on Nougat, besides that all AOSP/LineageOS based ROMs are patchable, so I assume they are still using dex version 037?

[Lanchon]

i never tried anything on-device, i don't even use haystack to tell you the truth lol
but i might try your patcher some day soon! :)

i have no idea about oreo, im stuck on android 6. obviously it is not compiling nor JITting, for whatever reason. regarding the dex versions, if you run dexpatcher in --debug mode it will show you that info. you can also enable --stats to profile performance. do tell me what you find out! :)

[ArchangeGabriel]

I know this is a bit off-topic, but regarding LineageOS there is a far better option: lineageos.microg.org. ;)

[cosmicdan]

Correction: https://lineage.microg.org/ It’s cool, but it’s yet-another-AOSP-fork. Some of us don’t even have kernel sources available for our device, let alone an AOSP-fork that is as reliable as a stock ROM. NanoMod is widely applicable, thus more useful. Regards, Daniel Connolly From: Bruno Pagani Sent: Wednesday, 27 December 2017 4:16 To: Lanchon/haystack Cc: Daniel Connolly; Comment Subject: Re: [Lanchon/haystack] Running DexPatcher-tool on-device (Android andTWRP) (#5) I know this is a bit off-topic, but regarding LineageOS there is a far better option: lineageos.microg.org. ;) — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[Lanchon]

yes OT but fantastic news for me, thanks for sharing!!!

i always said that dexpatcher is for apps you dont have sources of, and that i oppose to the idea of haystack given that we do have sources for android. still people were using more brittle solutions so i made haystack as a proof of concept use of dexpatcher.

[ArchangeGabriel]

Thanks for the link correction. But yes as I said, this is for LineageOS only, but that’s an huge improvement for our LineageOS users (everything they do in addition to LineageOS are likely things that people interested in μG will want, like F-Droid with the system privileges extension —even if in that regard Oreo change of per-app install permission is more interesting).

[Nanolx]

Long time no see, I just tried dexpatcher 1.6.1, but if fails with a NullPointerException on a real device (will do more tests and post logs during the next days).

But: I've managed to cook-up a patcher that uses vdexExtractor and dexpatcher in order to deodex services.jar and then patch it for signature spoofing on-device. Which works on my Nexus 6 just fine with official LineageOS 15.1:

https://forum.xda-developers.com/showpost.php?p=77443818&postcount=3697

basically on older ROMs (Nougat?) it uses smali/baksmali instead of vdexExctractor, though that's untested for now.

Also I can confirm that the patch for 7.0 also applies properly on 9.0.