Troubles downloading items imported to wallabag from pocket

[joelostblom]

Thanks for creating plop-reader, its really useful!

I recently migrated from pocket to wallabag and I am currently trying it out with the service hosted on wallabag.it. I imported all my existing saved articles from pocket to wallabag, via the wallabag web interface. However, for some reason these items do not sync to my pocketbook with plop-reader. New items that I add by for example sharing the article to wallabag via mobile Firefox, are synced with plop-reader as expected. I have both unread and archived articles imported from pocket, and neither of these are syncing.

Since I am new to both plop-reader and wallabag, I realize I might be doing something incorrectly, but I cannot figure it out. Do you have any idea how to solve this issue?

[pmartin]

Hi @joelostblom
Thanks! I'm glad you like this app and find it useful!

Currently (this might change in the future), the application fetches the most recent 200 entries from the server -- no matter if they are unread or archived. There is no other filtering done by the application to exclude any kind of entry.

Do you have, on wallabag.it, more than 200 entries (including unread, starred and archived ones) that are more recent than the ones imported from pocket ?

If so, that's why the ones from Pocket are not fetched by the app.
Else... that's interesting... Maybe wallabag flags the imported entry some way and doesn't return them in the API or something -- I'll have to try importing some entries to find out (or ask wallabag's developers their opinion)

[joelostblom]

Thanks for the quick reply! I only tried with one item that is more recent than the items imported from Pocket. So the 200 limit should not be the issue.

I just tried fiddling around some more and noticed that if I mark a wallabag item originally imported from pocket as read/archived and then as unread again, it is discovered and synced by plop-reader. The same thing happens if I star an item. For some reason they are not discovered by plop-reader sync before this....

[pmartin]

@joelostblom thanks for this additional information!

From plop-reader, I request entries from wallabag using its API, specifying I want entries with sort=updated and order=desc
-> This can be seen in this piece of code

I believe the API endpoint in wallabag is this one.
Not much to note there, except I suppose there is some kind of updated_at field on each entry (there is), that's used when the client asks for entries sorted by updated, like I do from plop-reader.

Taking a look at the parseEntry method in import from pocket, I see a call to setCreatedAt in some case, but I don't see any call to any kind of setUpdatedAt method...

So... Maybe entries imported from Pocket do not get an updated_at date set, and maybe this causes them not to be returned by the API? Or to be farther than the 200 mark.
Which means a solution could be to always set updated_at to now when importing from Pocket ?

It would fit quite well with your observation that the entry is returned by the API once you have marked it as archived/starred from wallabag, as marking it one of these ways most likely updates (or sets, in this case) the updated_at field to now.

Judging from the history of PocketImport.php, maybe @j0k3r could help on this, at least to confirm what I'm thinking?

[j0k3r]

The updated_at field is always defined when an entry is saved: https://github.com/wallabag/wallabag/blob/master/src/Wallabag/CoreBundle/Entity/Entry.php#L457-L468

And by default defined at the current date and every save.
So the updated_at field should be defined.

I've to check locally what's going on because right now I don't know what could be wrong.

[pmartin]

Oh :'( I was so hoping this would explain that ^^
Thanks for your answer @j0k3r!
If I ever get to meet you IRL, I owe you a drink ;-)

In the end, if it comes to that, I'll end up creating a test account on pocket, with only a few entries -- and then import them, checking how things are looking in the database, from the API's point of view, and finally on plop-reader's side (and if I messed up on my side... #shame )

[joelostblom]

@pmartin @j0k3r Great, thank you for looking into this! It would indeed be cool to have this feature in plop-reader, but I don't think it is essential. It works quite well to just go through items individually and update the ones I want to send to my pocketbook. So if this is estimated to be a very time consuming addition rather than a quick fix, maybe it is not worthwhile at the moment.

[wxcafe]

I have that same issue on my self-hosted instance. Here's a debug log from plop-reader (you can see I like reading Making Magic lol) https://pub.wxcafe.net/plop-reader-logs.html

[pmartin]

@wxcafe Thanks for the log! I only quickly took a look at it for now, but didn't see anything odd.

I've done a first quick-test, trying to reproduce this problem:

• Created a new pocket account (I had one many mongth ago, but there are still thousands of entries in it and I don't want to import them to my real wallabag instance ^^ )
• Added an entry to it (via pocket's web interface)
• Configured my self-hosted wallabag instance to import from pocket (after creating an API key on pocket's side)
• Ran the "import from pocket" on my wallabag -- synchronously in the foreground (I don't have a rabbitmq on my server ; and for 1 entry, that'd be a bit overkill) and without checking "mark as read"

After that, the entry seems correct on wallabag's web interface (but I didn't do anything on it via wallabag's web interface), so I start a "sync" on my TL3.
=> That's what I was expecting to fail.

And it didn't fail :-(
The entry imported from pocket is visible in the "unread" list of entries in plop-reader on my TL3 ;-(

I did the same test with a couple of other entries (from other domains, just in case) ; same thing, they are visible in plop-reader ;-(

So, for now, I failed to reproduce this.

@wxcafe and @joelostblom : are the steps I wrote here (the steps are did) the same you are doing on your instances? Maybe I didn't do something the way you did?

[wxcafe]

basically same thing, but I have 700 entries on my Wallabag instance’s web interface, and when I sync to my PB Inkpad I only get 6 of the unread ones and about 150 of the archived ones. My import was done asynchronously through Redis, but apart from that I don’t see any differences with your steps.

…

On 1 Sep 2017, at 13:44, Pascal MARTIN ***@***.***> wrote: @wxcafe <https://github.com/wxcafe> Thanks for the log! I only quickly took a look at it for now, but didn't see anything odd. I've done a first quick-test, trying to reproduce this problem: Created a new pocket account (I had one many mongth ago, but there are still thousands of entries in it and I don't want to import them to my real wallabag instance ^^ ) Added an entry to it (via pocket's web interface) Configured my self-hosted wallabag instance to import from pocket (after creating an API key on pocket's side) Ran the "import from pocket" on my wallabag -- synchronously in the foreground (I don't have a rabbitmq on my server ; and for 1 entry, that'd be a bit overkill) and without checking "mark as read" After that, the entry seems correct on wallabag's web interface (but I didn't do anything on it via wallabag's web interface), so I start a "sync" on my TL3. => That's what I was expecting to fail. And it didn't fail :-( The entry imported from pocket is visible in the "unread" list of entries in plop-reader on my TL3 ;-( I did the same test with a couple of other entries (from other domains, just in case) ; same thing, they are visible in plop-reader ;-( So, for now, I failed to reproduce this. @wxcafe <https://github.com/wxcafe> and @joelostblom <https://github.com/joelostblom> : are the steps I wrote here (the steps are did) the same you are doing on your instances? Maybe I didn't do something the way you did? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AC3rL-ZAyjok8cxzqHu12g34Oq_7q31hks5sd-4ugaJpZM4OwTJN>.

[joelostblom]

Thanks for spending time on this!
Yes, the procedure is the same except that I did it via wallabag.it, so I used RabbitMQ for the import. I imported about 1500 entries and none of them was visible before I did the mark read/unread manually as described in my second post above. I have a TL3 pocketbook. (I just clicked the import from pocket button again because I thought wallabag would ask me for the API key once more, but instead it started importing the 1500 entries again... noooooooo..... =/ )