Consider adding ability to also load GPS providers from non /system/app for Android N

[pvagner]

There is Unified NLP as a part of MicroG. Since Android N, Unified NLP is not loaded because Android OS no longer loads GPS providers from non /system/app folder. @danielegobbetti has created and @mar-v-in has accepted a patch changing android_frameworks_base in an attempt to address this. Can you please also add it?

[Lanchon]

commit that causes issue:
aosp-mirror/platform_frameworks_base@c7bacab#diff-361fa96857135a852bd2307b130d38edR95

[Lanchon]

FYI, some discussion regarding the correctness of that patch:
microg/UnifiedNlp@f5a4569

[xsmile]

Any news on this? Apparently the discussion came to a reasonable end.

Greetings

[Lanchon]

yes it did.

i'm sorry, it's impossible for me to devote time to this right now.

[xsmile]

@Lanchon:

I tried to start the process of creating a patch by myself, however I had issues running build-patch with a sample patch of yours. I hope it's fine if I post the log here instead of creating a new issue.

It would be great if you can give me a hint in the right direction. I will continue to try and create the patch by myself. Thanks for the quick answer.

$ ./build-patch patches-src/sigspoof-core 25 filesets/25-7.1.1-lineage-mako
>>> target directory: sigspoof-core-dex
>>> dedex fileset
>>> target directory: sigspoof-core-dex/tmp/dedex
dex2jar filesets/25-7.1.1-lineage-mako/core-libart.jar -> sigspoof-core-dex/tmp/dedex/core-libart.jar.dedex.jar
com.googlecode.d2j.DexException: not support version.
	at com.googlecode.d2j.reader.DexFileReader.<init>(DexFileReader.java:151)
	at com.googlecode.d2j.reader.DexFileReader.<init>(DexFileReader.java:211)
	at com.googlecode.dex2jar.tools.Dex2jarCmd.doCommandLine(Dex2jarCmd.java:104)
	at com.googlecode.dex2jar.tools.BaseCmd.doMain(BaseCmd.java:288)
	at com.googlecode.dex2jar.tools.Dex2jarCmd.main(Dex2jarCmd.java:32)
...
>>> build patch: services.jar
warning: [options] source value 1.5 is obsolete and will be removed in a future release
warning: [options] target value 1.5 is obsolete and will be removed in a future release
warning: [options] To suppress warnings about obsolete options, use -Xlint:-options.
Fatal Error: Unable to find package java.lang in classpath or bootclasspath

EDIT:

This seems to be a dex2jar issue with Android N, see pxb1988/dex2jar#93.

EDIT 2:

Next issue.

$ ./build-patch patches-src/sigspoof-core 25 filesets/25-7.1.1-lineage-mako
...
>>> build patch: services.jar
patches-src/sigspoof-core/services.jar/GeneratePackageInfoHook.java:58: error: cannot access List
                    if (p.requestedPermissions.contains(FakeSignatureCore.PERMISSION)) {
                                              ^
  bad class file: sigspoof-core-dex/tmp/dedex/core-oj.jar.dedex.jar(java/util/List.class)
    default method found in version 50.0 classfile
    Please remove or make sure it appears in the correct subdirectory of the classpath.

[Lanchon]

so patches reference symbols from the system. for that reason, the system dex files are passed through dex2jar and the result is fed to the java compiler during patch building. (only the method prototypes are important here, the actual code is ignored.)

this dedexing process (dex2jar) is done just-it-time during patch building. but because it's slow, it can also be done ahead-of-time. (there's a haystack script for that.) when AOT dedexing is done, a dedex subfolder is added to each fileset. the patch builder will see this and use the AOT dedexed files, instad of dedexing JIT to a temp folder.

well... unfortunately dex2jar cannot handle Android 7+: the dex version changed (035 to 037) and dex2jar seems to be abandonware now. the solution is to manually dedex AOT the Android N fileset to a dedex subfolder. how? two methods:

• the one i never tried:
  • manually use enjarify. unfortunately enjarify does not keep annotations, which measn stuff like generics will be stripped from APIs; not good.
• the one i used:
  • rewrite the v037 dex files of the fileset to v035 dex files into a temporary fileset.
  • use the haystack script to pre-dedex the temporary fileset.
  • move the produced dedex subfolder to the real fileset (v037).

how to rewrite the dex files? well, dexpatcher to the rescue! :)
dexpatcher --multi-dex --api-level 23 <input-jar> <output-dir>
then copy <input-jar> and replace the dex file(s) within the copy with the ones in <output-dir>. do this for each jar/apk in the fileset. then dedex the new modded fileset. then move its dedex folder to the real fileset.

the important thing is forcing API 23, aka Android M. the outputted dex will have some stuff only allowed in dex v037 from the source and thus will be malformed. dex2jar will fail to understand parts of it, but it will translate most of it which is more than enough.

[xsmile]

The AOT dedex process was successful. However after placing the dedex-auto directory in my fileset (according to the build script it first checks for the directory dedex, then for dedex-auto) and trying to build the core patch, it still fails.

$ rm -rf sigspoof-* && ./build-patch patches-src/sigspoof-core 23 filesets/25-7.1.1-lineage-mako
...
patches-src/sigspoof-core/services.jar/GeneratePackageInfoHook.java:30: error: cannot find symbol
import android.util.Log;
                   ^
  symbol:   class Log
  location: package android.util
patches-src/sigspoof-core/services.jar/GeneratePackageInfoHook.java:58: error: cannot access List
                    if (p.requestedPermissions.contains(FakeSignatureCore.PERMISSION)) {
                                              ^
  bad class file: filesets/25-7.1.1-lineage-mako/dedex-auto/core-oj.jar.dedex.jar(java/util/List.class)
    default method found in version 50.0 classfile
    Please remove or make sure it appears in the correct subdirectory of the classpath.

The command at this point is:

javac -bootclasspath filesets/25-7.1.1-lineage-mako/dedex-auto/core.jar.dedex.jar:filesets/25-7.1.1-lineage-mako/dedex-auto/core-libart.jar.dedex.jar:filesets/25-7.1.1-lineage-mako/dedex-auto/core-oj.jar.dedex.jar:filesets/25-7.1.1-lineage-mako/dedex-auto/ext.jar.dedex.jar:filesets/25-7.1.1-lineage-mako/dedex-auto/framework.jar.dedex.jar:filesets/25-7.1.1-lineage-mako/dedex-auto/framework2.jar.dedex.jar -classpath filesets/25-7.1.1-lineage-mako/dedex-auto/services.jar.dedex.jar:/home/user/Downloads/haystack-master/./dexpatcher/dexpatcher-annotation-1.2.0-beta2.1.jar -source 1.7 -target 1.7 -d sigspoof-core-dex/tmp/parts/services.jar patches-src/sigspoof-core/services.jar/GeneratePackageInfoHook.java

[Lanchon]

had forgotten, you are right about dedex-auto. rename your dir to dedex, to signal that this is a manually produced dedex. that way the auto/JIT dedexer will never try to regenerate that dedex (even if asked to regen).

will look at the compiler error now...

[Lanchon]

lol! :(

it looks like android.util.Log has been moved out of the fileset and into somewhere else. to fix this you have expand the definition of a fileset to include whatever jar now holds class Log. and re-pull. and dedex the new jar, lol!

and......

it seems p.requestedPermissions is no longer accessible on Android 7.1!
looks like some nontrivial upgrade work needs to be done here, as Android changed (again) the code we are hooking. i can't really do this now, i'm sorry. and unless you can handle it mostly by yourself, i suppose helping you though it would take more time than doing it myself. maybe leave it for when i have time? i expect that would be April... :(

btw, watch your api level!
./build-patch patches-src/sigspoof-core 25 ...

[xsmile]

Strange, I applied both prebuilt core and hook patches on LineageOS 7.1.1 and the signature spoofing works fine. My assumption is that I did something wrong during the build process.

I will try to figure out what might be wrong and post my results, if any.

Thank you for your time, you already spent too much of it helping me :)

[Lanchon]

well, the Log class will be found and linked at runtime, even if not in the fileset, so no issue there.

but the access to the permission list will probably cause a verification error.

why didn't dexpatcher catch that?

well, for historical reasons. dxp was not made to patch several different versions of a bytecode with the same binary patch. thus, part of the type checking is expected to be done by the compiler.

this is why the symbols are imported: so patch code can use all of them. the compiler checks.

alternatively, instead of importing you can define the symbols you use like this:
@DexIgnore
private int a;

dxp then ignores a. it still doesn't check anything at patch time, but you no longer need symbol a imported.

if you want to check for the existence of 'a' at patch time you can do this instead:
@DexEdit
private int a;

so now patch fails to apply if 'a' doesn't exist. but... this is not good because dxp will replace the definition of a. if a has annotations, they are deleted. if a changed to protected in a later rev of the app, it will be made private again and the app will fail (dxp will warn about this visibility change though).

what is needed is a new tag:
@DexCheck
private int a;

that verifies at patch time that a is accessible but doesn't change it. accessible at least with private access (but actual 'a' could be less restrictive). and at least in this class (but 'a' could be defined in a supper class).

i have the intention of improving dxp to be able to patch different target code with the same binary patch. it requires 3 features, one is DexCheck, it's all thought out. unfortunately i don't have the time to implement now.

[Lanchon]

in short, dxp was designed to be used like this:

• get an apk to patch
• code the patch
• compile and apply patch
• update to new version of the apk
• compile and apply patch
• fix the reported errors
• compile and apply patch

the combo javac+dxp does a really good job of signaling where changes are needed. (this is your case btw!) but skipping javac can let errors creep though patch time to run time.

dxp could do a better job in the skip-javac case and i intend to fix that, but only if the patch programmer declares what to check, which means more work. this extra work is essentially declaring all of the used source symbols yourself, and forgo the convenience of having them auto-imported for you.

alternatively you can check for verification errors on an actual device or emulator after patching.

[xsmile]

I made some progress. The build process was successful after I downgraded JDK to 7 and build-tools to 23.0.3. I'm not entirely sure what went wrong during the compilation, but now it works.

Here is my self-made patch. Testing it showed that microG passes all self-checks and that I can configure and use an external provider while having GmsCore installed as a user-app.

@Lanchon:
It would be great if you can confirm that I didn't do anything bad in the source. You can find it in the linked attachment. That should be my last request. I won't annoy you anymore :)

[Lanchon]

congrats, you made it!!!

google wants to break away from javac and dx (load of crap if you ask me, they should maintain dx all the same) with a direct to dex compiler called jack. AFAIK, only jack supports java 8 features, which led me tu suppose dx wasn't updated for N. this means: 1) dx shouldn't understand java 8 bytecode (but haystack compiles with -source 1.7 -target 1.7 so using javac 8 shouldn't be an problem). 2) dx should produce version 035 dex, since 037 is supposedly needed only to support java 8 features.

i don't know why you needed to downgrade to java 7. but of course you know that build-tools later than 23 require a java 8 JRE to run, so the build-tools part is explained at least.

regarding your code, i've looked at it; looks handsome! :) no, really: i'd need to look at android source and think a bit to have an opinion. at first glance it looks ok. if it works, just relax, i'm sure everything's fine :)