CAUSE: 3 Something went wrong. ApiException: 39508 #1021
Comments
|
The app also displays "Updated: Yesterday, 07:46", so it did not register properly the check that was made today (14.08) at 07:36 |
|
Hi @kbobrowski , according to the wrong link to the faq, maybe it had been introduced by #864 (CWA v1.2.0) ? CWA displays yesterday's date - what do the Google Exposure Notifications ("COVID-19-Benachrichtigungen") display (update date in main screen)? Can you provide a screenshot of the error message, if error is reproducable (would be only for my own curiosity, tbh) |
|
@kbobrowski , some additional questions: |
|
Hi @vaubaehn , Google EN just displays the last date from the logs I posted, so "Last checked for potential exposure on today at 09:36". 39508 error happens every time I open the app now as the app holds "last time diagnosis keys were provided" at "yesterday, 07:36", so it will exceed limit of 20 calls every time. Will check tomorrow if the issue repeats again. Not sure if it was background / foreground, I guess first could be background and second foreground. |
|
@kbobrowski , one feature request for the ENF, that our devs here might direct to Google, could be that ProvideDiagnosisKeys() creates an object with some statistics about some technical results of that task, e. g., if the task had been completed successfully, time of last API-call, remaining count of API-calls. That would support the EN-apps (CWA) to validate if they may call ProvideDiagnosisKeys() without running into an exception. |
|
At the expense of making my app useless for the next couple of days I was shifting the date forward which was resetting internal GMS count and forcing update of diagnosis keys, and was not able to reproduce this error. @vaubaehn that's a good idea to have some more information about whether the call to public class Quota {
private long day = 0;
private int count = 0;
public void updateCount() {
long currentDay = System.currentTimeMillis() / TimeUnit.HOURS.toMillis(24);
if (currentDay > day) {
day = currentDay;
count = 0;
}
if (count < 20) {
count += 1;
}
}
public boolean canCallProvideDiagnosisKeys() {
return count < 20;
}
}But right now CWA is only executing |
|
@kbobrowski , right, CWA could improve here, but I suspect ENF to fail matching keys under some conditions (#774 (comment)) , but counting the call though. In that case, CWA couldn't do anything about it... |
|
@vaubaehn right, EN may fail to match keys and still count the call, we have seen it e.g. with error 10 - these kind of bugs have to be fixed by Google. But the disadvantage of current CWA implementation is that in this situation it is practically guaranteed that 39508 error will follow as CWA is not maintaining any count of calls to I think CWA should maintain internal count of calls and simply not attempt to call if the quota is already exceeded. What would be very useful is to instead display the information about it (perhaps in another UI section), together with information about original error, which is currently not displayed to the user if it has happened in the background. |
|
By the way, I found us often handling with the rate limit/quota of ProvideDiagnosisKeys(), but I still couldn't find any information on why Apple and Google actually set a quota. |
|
I don't know why the quota is here, I can only speculate that the reason is "privacy" (there are some theoretically possible scenarios which may lead to some privacy implications if there is no quota at all) or "battery" |
|
I hope it's not the latter ("battery")... because that might underline the CPU load caused by cryptography, In that case, I don't know where this all might lead to, when a) amount of users of CWA should increase significantly one day, b) reported positive cases are drastically increasing (2nd wave * more users), c) big events are planned/more using of public transport in cold weather (much more collected RPIs)... Anyway, have a nice week-end! |
|
People are also reporting this error on twitter: https://twitter.com/elliwir/status/1294262706762063873 @vaubaehn Have a nice weekend as well! |
|
I couldn't resist to have one more look here... |
|
@vaubaehn it's still v1.5 for me |
|
There definitely more people with this problem. Here is just one other recent example: https://twitter.com/DieterKlinkner/status/1294588826908753920 One thing that caught my attention is that this person seemingly also has a quite intensive routine to try and get the app going. |
ghost
assigned
|
I got another user reporting this problem and for them not opening the app for 24h doesn't seem to have solved the issue. Device: Samsung Galaxy A6 Any ideas? |
|
My working assumption is the timeout - if fetching diagnosis keys and submitting them to API is not finished within 60 seconds then the transaction is terminated, and if some diagnosis keys were already submitted in this terminated transaction, then next retry may lead to exceeded quota. While testing this with artificial delay in the transaction it also happened very rarely (only once in many tries) that transaction hanged without completing a single call to One possibly important thing here is that CWA already operates with very thin timeout margin - without introducing any artificial delay and on the fast wifi network (100 Mbps) the whole process of fetching 14 diagnosis keys and submitting all of them to the API takes about 45 seconds on a slow Android 6 device, which is 75% of available timeout. Interestingly the description while fetching keys says that it may take "couple of minutes", but after 1 minute there is hard termination. The only counter argument to this is that in my case (logs from the first post in this issue) on Android 10 all 14 diagnosis keys have been submitted, and then transaction was repeated, so it's quite unlikely that timeout happened while API was processing 14th key and just before updating last-fetch-date, still possible though. |
Wow… that's much longer than I'd have thought this process takes… I've always assumed that this is done in a few seconds max. I guess the limiting factor in this case is not the bandwidth but the (cryptography-) performance of the SoC, right? What would speak for this hypothesis is the fact that the Galaxy A6 mentioned above is certainly not the most potent device… On which device did you observe the 45s sync time? |
|
@daimpi I guess slow device (Motorola G2) plays a role here, but it seems that just downloading keys takes significant portion of this timeout limit (if it has to download all 14 of them). Similar issue reported in #1030 , if it is the same pattern as in my case (all 14 keys successfully submitted, but then CWA retried the same day), then I think timeout hypothesis would be less likely |
|
Same here. Moto G6 Plus |
|
Another exposure log provided by @not-a-feature (#1030 (comment)): so this time CWA only submitted 9 keys on 18.08 02:31, and then retried submitting 11 keys 18.08 02:33. So I guess the timeout hypothesis is still possible, perhaps new Play Services is taking a bit more time to process Would be great if CWA allowed for exporting logs for the last 48 hours or so, just storing them in a text file and allowing user to export it from the app, otherwise there is no easy way to know what was the first error that happened, 39508 is just a consequence of it. |
|
Same here today. LG-H815 (LG G4) Usually background update worked like a charm for me (usually around 8:23am, +- 3 minutes). => "Cause 9002 Timed Out while waiting for 60000 ms" highly unlikely. |
|
I was trying to reproduce something by calling |
The user reported back today that after not opening CWA for another 24h the problem seems to have finally disappeared for them 🙂. |
|
@vaubaehn are we talking about @LSDJGLKJG or @PapaBravo ? Former looks to me like a classical case of silent 9002 timeout (now instead of 60s presumably after 180s), while the latter seems to be a case of API (10), no? |
|
@daimpi we are actually talking about @LSDJGLKJG 👍
But I'm not sure, this is why I asked for more error details :) |
|
Version 1.5.0 helped me out after 24h wait. Thanks alot for fast fix and good handling. |
|
still receiving the "CAUSE: 3 Something went wrong. ApiException: 39508" message on my
|
|
@vthomw Have the same error message every time I open this app. Xiaomi 8a, MIUI 11.0.1 (Android 10) |
|
I guess I have to wait 24 hours with the app closed .. lets see what happens tomorrow. |
|
@vaubaehn Thanks, thb I'm wasn't entirely familiar with all the details of the new 1.5 approach. A few questions for my understanding:
Do you infer this from the timestamp being after 00:00 UTC, or something else?
I totally forgot that CWA now has a quota calculator, that's cool 👍. Yes you're right this should actually prevent 39508 if it was working correctly 🤔.
Let's imagine the following scenario: a very slow device which is hitting the timeout limit of 3min. In this case there will be 20 times 9002 timeouts but they will not be "silent" b/c GoogleQuotaCounter prevents a 21st call which would trigger error 39508 which would cover up the 9002 error. Is this the correct reasoning?
Good point, let's wait for more info from @LSDJGLKJG. |
|
Hey, I will just wait and see how it goes over the next few days, I guess. |
|
@vaubaehn Sorry that I can't give you more error details, now that the error seems to fixed. |
|
@LSDJGLKJG great that it seems to be working for you now 👍 🙂 Could you maybe share your EN log from today as well, just for comparison?
Could you share a screenshot of this?
Unlikely. The real computational cost seems to come from the expansion of diagnosis Keys (via @mh-) :
We recently had a detailed discussion on this topic in the community Slack if you're interested to learn more 🙂.
Sounds good, let us know how it goes 🙂. |
|
Alright, here it is: |
|
I had the same problem error#39508. |
Same with my phone 😀 it's working now. Thank you! |
|
The update solved my problem. After about six weeks I can check the risk status again. Samsung Galaxy S5 mini (SM-G800F) |
The quota calculator is currently not armed. It tracks the quota and can be armed in a future update while already knowing the correct values, but currently it will not prevent submission to the ENF. This decision was made to not accidentally hide an underlying issue, as the quota of 20 calls should be enough on 95% of devices, those that have the newer ENF, where we can submit all files in one call. We should not be able to reach the quota limit 🤔 . |
|
Works fine today, too. I'm really happy. |
|
@d4rken thanks for clarifying 🙂. Did you see my post above? There @LSDJGLKJG's phone actually hit the limit of 20 calls. My guess was that this was caused by silent 9002 timeouts, as this is something we have seen before the release of CWA 1.5 with low powered devices. What I'm wondering: if CWA gets the timeout, will it next time again submit the whole package of DKs to ENF, or will it only submit the "rest" which hasn't been processed yet? Because if it every time submits everything again, what is stopping it from timing out on every submission and then retrying again? If this happens it's easy to hit the limit of 20 calls. Does this make sense? |
I missed that. I currently don't why it would retry this often.
It will submit all available (those in the cache) packages. If only the download times out for a few files and the transaction timeout it not reached, it will continue with those files that are available. So even if a few downloads timeout each time, even a slow device/network should eventually have and submit all files, if at least one download completes per attempt. At least one key package has to have been downloaded because the
That's theoretically possible, I'm just not sure what would lead to this. 20+ times in a row. Could the device be downloading files slow enough to reach 3 minute timeouts on every attempt? It's all on the same day too. So there would have to be a least one valid existing file, and in 20 attempts it never manages to download all existing ones, so a success ratio below 1 per attempt. As it seems to be fixed, maybe that was the case 🤔. The timeout increase in the hotfix should improve this behavior. Would arming the quota calculator have helped here? It would only have hidden the issue, wouldn't it? @LSDJGLKJG Wifi at home? 4G? |
|
@d4rken @vaubaehn thanks for the detailed explanations (also over at #1187). I'm still trying to fully wrap my head around everything, but this helps a lot 😊 ❤️. Regarding
Yes I would think so too. @schneik1 reported a similar story wrt multiple checks above (just that theirs didn't go all the way to 39508):
but unfortunately their log seems to have been corrupted, so I couldn't check this yet. @schneik1 könntest du einen neuen log für uns hochladen, da dein alter scheinbar beschädigt ist? TIA 🙂 |
|
Yes, @schneik1 , a new ENF log would be nice. The other one looks a bit fishy to me. 🐟 |
|
@vaubaehn well actually… 🤓 Ok I'll see myself out 🏃 😂 |
|
success .. update to 1.5.0 and a 24 hr wait fixed it. And my CWA still shows a green low risk. |
|
@daimpi Hier der 2.Versuch. Nach Installation von 1.5.0: Gruß |
|
@schneik1 Super, vielen Dank! Nun fühle ich mich von einem ENF log geküsst, nicht von einem Delfin ;) Schön, dass es jetzt funktioniert, aber eine Frage hätte ich trotzdem noch, weil das ENF log doch etwas merkwürdig aussieht: Wie oft öffnest du die Corona-Warn-App? 1x am Tag, alle paar Tage, 1x die Woche, oder seltener? Oder schaust du gerne in den Google-COVID-19-Benachrichtigungen rein, ohne die App zu öffnen? |
|
@schneik1 ... und noch ein paar weitere Fragen:
|
|
UPDATE IMPORTANT: Hello everyone, we are happy to hear many positive feedback after updating to 1.5 but since this issue with over 300 comments is too bloated, it makes it really hard to seperate new issues. This is why I needed to lock this conversation. I did create a follow up topic in case you are still receiving the same error after updating to 1.5 Thank you very much for your understanding. Best regards, Corona-Warn-App Open Source Team |
and others
This bug was mentioned many times due to various causes but this seems to be a new case.
Exposure checks log:
Internal Tracking ID: EXPOSUREAPP-2213
The text was updated successfully, but these errors were encountered: