Distribute flashable zip for Android 7+ for broken UnifiedNlp integration

[xenithorb]

So I've recently migrated to Android 7.1.1 as LineageOS has picked up the ball and started producing builds for bacon again.

My update workflow has changed in the following respects:

1. Where I would have normally had Xposed+FakeGapps installed, I now use tingle, as it's the only reliable way I could currently find to allow spoof'd signatures - given that Xposed for 7+ doesn't exist yet. This numeral is merely informative.

2. The point of the issue: As you know, given the UnifiedNlp README and the patch you produced here you are at least aware of the issue and the simple fact that going forward, UnifiedNlp and therefore many apps will not work with the traditional "Just install the app" workflow, due to the fact that only privileged system apps may be a location provider in 7+.

Your solutions are:

1. Apply the patch (which you can only do if you intend to build your own ROM from scratch)

   • CyanogenMod/LineageOS will not be including this patch: https://review.cyanogenmod.org/#/c/177082/
   • Patching a ROM and building said rom is a non-trivial process for average users

2. As root, install GmsCore in /system/priv-app/ (You said it slightly different for UnifiedNlp-only but the problem remains the same, and I verified this works myself)

This leaves us with option #2, which if you're a nightly user (hint: No other user of LineageOS atm) is frankly really a pain. And worse, a conundrum:

1. microG / GmsCore still needs to be installed prior to all GCM apps, and if it's reinstalled could it lose those subscriptions?
2. Apps in /system/priv-app/ will be erased on upgrade

Given that, I think it would be really nice if you could produce a flashable .zip that also includes an addon.d script to "rollover" microG to the next upgrade like Gapps does it. (hopefully preserving the push subscriptions too) - note when I say "push subscription" I mean the apps listed under the "Google Cloud Messaging" menu and how that operates with the necessity of GmsCore having to be present first.

Bonus points: Perhaps a second zip could be produced that does what tingle is doing and one-shot both the patching of the framework to support spoof'd signatures, and also install microG in the correct area.

Thanks for all of your hard work! I've been using this for a long time now and it's otherwise has worked great and has given me great peace of mind.

[ale5000-git]

@xenithorb: I have already created it: microG unofficial installer

[xenithorb]

While I appreciate your work, I do find it a bit opinionated given the circumstances ( I really don't want google software installed, just microG ). I also don't think it removes the burden of this project to do something official that supports current and future versions of Android.

I think we need a zip that does, at minimum:

1. Installs microG components (only) GmsCore and GSF-proxy into /system/priv-app/
2. Persists those packages into an upgraded /system via addon.d method or other means.

[ale5000-git]

1. The problem is that some components haven't real open source alternatives that work flawlessly, so I opt for an hybrid way to "open source" as much as it can be now.
   The open source alternative for accessing the Play Store is not updated (excluding maintenance fixes).
   The zip created by me can be used also from mostly "noob" people (it is really easy to use), doesn't require any knowledge to be used, and just require less than 1 minute to be installed, it also remove GApps automatically.
   The most important thing is that it allow an hassle-free transition, allow to switch from GApps to microG almost without notice any difference (I mean no feature lost, the are many positive differences).

2. Done.

[xenithorb]

Yalp store on fdroid works great. Its a good alternative for seeking out updates and new software.

[ale5000-git]

@xenithorb: The problem is that there is the risk of getting your Google account banned, and if you want to use both it and Apps that need Google account you need to insert the account twice or use two accounts (both options are annoying and unfriendly), correct me if I'm wrong.

[xsmile]

I would like to provide another alternative using haystack as a patching toolkit and a patch I have recently implemented to use UnifiedNlp as a user app:
Lanchon/haystack#3 (comment)

[xenithorb]

Well, here I am 1 nightly LineageOS update later, and I accidentally tripped up and did the exact thing I foresaw and was trying to prevent here. :(

I forgot to add GmsCore.apk back to /system/priv-app and as a result, after rebooting, it lost all the GCM subscriptions. Now I have to go and delete all of the GCM apps and reinstall every single one. As it turns out, backing them up and restoring doesn't work either so this will incur some amount of data loss... No good!

Anyone have a workaround for that? One of the other problems is I have no idea which apps actually can utilize GCM, so I have to go from memory of what might use it.

[xsmile]

@xenithorb:

Use fr.simon.marquis.preferencesmanager to manually remove GMS settings from apps you except to have push notifications. Follow this example:

• select an app, e.g. WhatApp
• select file com.google.android.gms.appid.xml
• remove all entries (appVersion, lastToken, ...)
• kill the app process

The next time the app is started, it will aquire a new GMS token and it will be listed in microG. The entries might differ for various apps, but you get the idea.

[xenithorb]

Ok, that worked for the most part. Not every app seems to follow that convention, but for those that didn't I was able to look a little deeper and usually find some gcm_register_id or some other attribute that I could delete. Most did trigger a re-registering when such settings was missing. Thanks! @xsmile

[LuccoJ]

I echo @xenithorb in finding such a zip desirable for myself, but strictly without any proprietary components included. If Yalp Store runs the theoretical risk of getting your Google account banned, then so will any other free Play Store replacement potentially developed in the future (including µg-based ones), because they'd potentially breach the very same ToS. This would mean that under @ale5000-git the zip would never possibly become fully free, even the day everything had a free alternative, and for my preferences, that is not a desirable route.

Besides, it's always possible to get Google Play apps from places like APKPure, which are just websites. There are potential security concerns but I think that specific one, for instance, has a good track record, and it is always possible in line of principle to verify signatures before installing.

Of course, that's me, and other people have different opinions, and if I wanted to I could learn to create my own zip, so I'm just putting my opinion out there, not trying to force change :-)

[xenithorb]

The only viable way to install this going forward on Nougat+ is with a zip, or by manually pushing the file to /system/priv-app (allwhile not forgetting to do that ever or your app data disappears!). That's why I believe the way forward is for microG to supply a flashable zip. If at all possible, it should:

1. Modify the framework.jar (as Tingle does) (can python scripts be used in flashable zips?)

   • This is because Xposed does not support Nougat and won't for the foreseeable future, and the previous method involved FakeGapps+Xposed. I actually find "the tingle method" of patching framework.jar cleaner, and would wish for that to be done during the flashing process.

2. Copy the main GmsCore.apk to /system//priv-app

3. No store needs to be forced, proprietary binaries should not be distributed, and the user can download their choice of store whether Yalp or just F-Droid. (I prefer both, as there are simply certain things you need from the play store from time to time, as not all FLOSS projects publish on F-Droid)

[LuccoJ]

@xenithorb to nitpick,it's not projects themselves that publish on F-Droid, generally, but the F-Droid maintainers (mostly one of them)... so if an actually FLOSS project is not on F-Droid, it could be because there is a long backlog, in which case anyone can help by submitting working metadata for the project to F-Droid.
If there are different issues, on the other hand, then that means it usually boils down to the project not really being pure FLOSS but rather some kind of OSS-with-strings-attached, even if it's just by depending on a nonfree compiling environment.

[Shadow53]

FWIW, I provide a couple of zip files as well for people to install microG on nougat. One installs the Play Store, another FakeStore and leaves the user open to choose F-Droid, Yalp Store, or otherwise. Maybe I'll make an aroma one in the future, who knows. The zips do not currently have addon.d support, but I intend to add it as well.

I'm linking to the download page on the off chance I end up changing the link to the file: https://shadow53.com/no-gapps/downloads/

(For the record, I do not intend to compete with @ale5000-git. I just prefer to use my own solutions when I can and had my own basic zips when ale5000's was released)

[ale5000-git]

@Shadow53: There isn't any problem :)

PS: I intend to add support for configuration but I haven't had time yet.
Edit: Added.

[ale5000-git]

@xenithorb: The latest version of my zip have addon.d support.
Currently it backup only the apk of microG Service Core but once I get some reports that it works and some free time I will extend to all components.

[Nanolx]

I also made my own solution: https://github.com/Nanolx/NanoMod

Unlike the others it's a Magisk module, so it does not get overridden by ROM update, except you wipe /data, but it kicks in Magisk as hard-dependency. The github repository also contains framework-patcher.sh a Bash script that makes using haystack much easier (works on GNU/Linux, BSD, Mac OSX).

There are three packages:

• full package (with lots of extra stuff, not as relevant here I guess, ~180 mb)
• microG only package (~14 mb)
• F-Droid only package (~6 mb)

I modified microG GmsCore and Play Store, so that (in-)app-purchases work (I followed mar-v-in's instructions in issue #309 for that purpose).

The module supports a setup file /sdcard/.nanomod-setup which controls what app store to use, where you can choose none, Play Store or Yalp Store (for the microG only package, the full package supports for more options). See the README in the github repository.

[xenithorb]

What's the upgrade path for microG for your solution? I take it because they're signed differently, F-Droid isn't going to upgrade it as normal.

I switched to using Haystack myself, and I wrote a bash script that uses adb to shuffle things around and generally do what I need after a TWRP flash. That was my evidentual temporary solution.

Nice that it has an easy to grok config file!!

[Nanolx]

If you manually update it, you would loose ability to do (in-)app-puchases, as Play Store and microG GmsCore need to have same signature for that to work.

But the path is /magisk/NanoMod/system/priv-app/GmsCore in case you don't care.

Though, once there's an update, I'll push it to my github repo and then you could just use mod.sh microg for the microG package (or mod.sh zip for the full package) to create a new installable zip.

[xenithorb]

That seems a bit opinionated for my usage unfortunately. (Even though I'm testing it out right now)

1. My personal use case is F-droid + microG (so I am using separate microG and F-droid) modules that I generated from your repo.
2. I don't care about microG being signed the same as play, because I don't, nor will I ever use play.

I was hoping it wasn't a modified apk so that F-droid could update microG, as I have the microg.org F-droid repo installed and I use that to update it regularly to test builds. Unfortunately, for it to work (and the reason for this ticket) is that I have to move it over to /system/priv-app each time in order for it to work after updates now. Otherwise F-Droid upgrades it just fine. Otherwise everything else is nice - and it's a bit of a segue for me into Magisk, which I haven't done anything with before. Pretty neat if you ask me.

Side node: the framework-patcher.sh (haystack) doesn't seem to work after installing Magisk-12.0, it freaks out and quits on some java error

[Nanolx]

You do patch the files from TWRP and use the correct api level? Cause Magisk does nothing on /system, so it can't actually make a difference.

[xenithorb]

Figured out the problem, it was my own past corruption of java. False alarm.

Everything works ok. A few things aren't clear:

1. Does Magisk neeed to be flashed after every update? (I update lineage every week)
2. Does framework-patcher.sh still need to be done after each update? (Is there any way to avoid this?)

[Nanolx]

1. yes, as it modifies the boot image
2. yes, after every update, there's no practicable other way, except using crDroid or OmniROM, which com pre-patched with fake sign

[xenithorb]

So the only thing I really did was remove the necessity to do cp /sdcard/GmsCore.apk /system/priv-app/ in my script :/

I do quite like Magisk though, and your framework. Kudos!

We still need a solution to patch the *.jar files on the phone. There's a discussion about the possibility of doing that here: Lanchon/haystack#5

[xenithorb]

You see the problem that I have with the manual nature of some of these items is that it's reduced microG to only being used by techies, and then only ones willing to plug their phones into their computer to do a convoluted update. Even non-techies that have techie friends won't be able to use this because - while the usual process was to get someone "in the know" to set you up for the first time - that's no longer viable because it basically breaks normal updater updating. (i.e. a normal user who just updates from within LineageOS' updater) - if they don't remember or know to re-flash Magisk or patch the framework.jar to accept spoofing, then microG stops working for them and they will probably not know why. Ultimately I want this to eventually pass the "wife test," so I can setup my wife's phone and have her update it as normally and it not break. A sad state of affairs we're in, IMHO.

[ale5000-git]

@xenithorb: I have a lot of ideas to simplify all things ( and pass the "wife test" ;) ), I just lack time currently.
At the end patching will be just 3 click on the phone.

[Nanolx]

I created a on-device framework-patcher. See Lanchon/haystack#5 for more information. This is the first try, so may not work for everyone. Please test and report back.

[ale5000-git]

For those that didn't noticed Tingle does already work when executed inside Android (it is still a bit slow though).

[Torvin]

@xenithorb I would really appreciate if you share some more info about the second option you mentioned in the issue description. I have microG installed with Tingle, but UnifiedNlp (coarse location) won't work ("The system did not bind the UnifiedNlp service"). Do I need to uninstall it first? Do I then just copy the apk to /system/priv-app/ after and reboot?

[ale5000-git]

@Torvin: Not "copy to" but "move to".

[Torvin]

@ale5000-git move? from where?..

[ale5000-git]

Since you have it installed, you should have it in the data partition.
There are apps that can convert it to system apps if you don't want to do it manually.

[Torvin]

@ale5000-git I see the /data/app/com.google.android.gms-2 directory. Do I just move it to /system/priv-app/ and reboot?

[ale5000-git]

Yes.
I usually rename it to GmsCore for consistency but the name isn't really important.

[Torvin]

@ale5000-git it worked, thanks. what is the upgrade procedure for the next microG release? do I now have to manually download apk and put it into /system/priv-app/ root folder and reboot - and that would trigger the installation? is my understanding correct?

[ale5000-git]

You can put the updated file in /system/priv-app/ if you want but just update it as normal app works and retain the permissions (also long as the original file in /system/priv-app/ is still here).

PS: It doesn't really need to be in the root folder and usually it isn't (at least on Android 5+, on older versions it is different).

[Torvin]

@ale5000-git so the usual app update will work? great! thank you for your help very much!

[ale5000-git]

Yes.

[okias]

does anyone offer clear microG package without GooglePlay binaries and other mess? NanoMod seems to pack them too, without choice to disable it..

[ale5000-git]

@okias: Mine will have settings to disable it, I just need time.

[Nanolx]

Of course it can be disabled in NanoMod. The documentation has the instructions covered. https://gitlab.com/Nanolx/NanoMod#installation Am 18. Juli 2017 12:44:05 MESZ schrieb David Heidelberg <notifications@github.com>:

…

does anyone offer _clear_ microG package without GooglePlay binaries and other mess? NanoMod seems to pack them too, without choice to disable it.. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: #284 (comment)

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

[okias]

@Nanolx
after some fighting with DOCS, here is "open-source" .nanomod-setup:

nanomod_microg=1
nanomod_fdroid=1
nanomod_apps=1
nanomod_play=2
nanomod_overlay=1
nanomod_zelda=1
nanomod_mapsv1=1
nanomod_init=1
nanomod_gsync=0
nanomod_swipe=0
nanomod_forcesystem=0
nanomod_reinstall=0

[Nanolx]

I'm working on improving the docs, but my time is limited. But a user had a great idea: a separate 'SetupWizard', that uses ARONA to create the configuration files. Afterwards you can simply flash any of the other packages. No ETA, but will come for the next version. Am 18. Juli 2017 13:00:30 MESZ schrieb David Heidelberg <notifications@github.com>:

…

@Nanolx after some fighting with DOCS, here is "open-source" .nanomod-setup: > nanomod_microg=1 > nanomod_fdroid=1 > nanomod_apps=1 > nanomod_play=2 > nanomod_overlay=1 > nanomod_zelda=1 > nanomod_mapsv1=1 > nanomod_init=1 > nanomod_gsync=0 > nanomod_swipe=0 > nanomod_forcesystem=0 > nanomod_reinstall=0 -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #284 (comment)

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

[ale5000-git]

@okias @xenithorb
I have released a test build with live setup here.

@LuccoJ @xenithorb
Now there is a setting to choose the market app between Google Play Store and FakeStore.

[irfus]

On an unofficial build of omnirom (based on 8.1.0), which comes with the signature spoofing patch, I'm unable to get the unifiedNLP service to bind. Tried manually pushing the gmscore apk to /system/priv-app as well as the flashable installers by @Nanolx and @ale5000-git, with no luck. Any known workarounds? The documentation gives the impression that running as a system app should have been enough.

[Shadow53]

There are certain apps which may conflict with microG's UnifiedNlp service, one of which is com.qualcomm.location. I know @Nanolx and my zips both remove that conflict, I would assume @ale5000-git does too. You may have discovered another conflicting app.

I say this because the service binds fine on my OP5 running official Omni 8.1.0.

[irfus]

How can the source of the conflict be identified? Logcat output didn't seem very informative to me, but I could post that or any other relevant info here if needed.

[ale5000-git]

@irfus: If you use mine be sure to use version 1.0.26 alpha or higher (I have improved the cleaning).

Also post the /system/framework/framework-res.apk file so I can see what location service it search.

[irfus]

Unfortunately the rom is odexed, so I'm not sure if the apk is going to be useful (posting it anyway).

I also filtered the logcat output by the keyword "location", also attaching that if that's useful. (can also post a full logcat, if necessary).

Thanks!
location.log

framework-res.apk.zip

[ale5000-git]

@irfus
I'm a bit late but a lot has changed, could you please try the latest version of microG unofficial installer to see if there is still a problem?