User media response format issue

[pj-99]

hello,

thank you for this awesome package!

found an issue related to user media (with login). i use the code for getting user media with pagination:

import json
import time, random
from tweeterpy import TweeterPy
from tweeterpy import config
from tweeterpy.util import RateLimitError

twitter = TweeterPy()

# have logged in
twitter.login("user name","some password") 

user_tweets = []
has_more = True
cursor = None
while has_more:
    try:
        response = None
        response = twitter.get_user_media('elonmusk', end_cursor=cursor, pagination=False)
        user_tweets.extend(response['data'])
        has_more = response.get('has_next_page')
        api_rate_limits = response.get('api_rate_limit')
        limit_exhausted = api_rate_limits.get('rate_limit_exhausted')
        if has_more:
            cursor = response.get('cursor_endpoint')
        ## YOUR CUSTOM CODE HERE (DATA HANDLING, REQUEST DELAYS, SESSION SHUFFLING ETC.)
        time.sleep(random.uniform(7,10))
        if limit_exhausted:
            raise RateLimitError
    except Exception as error:
        print(error)
        break

with open('user_tweets.json', 'w') as f:
    f.write(json.dumps(user_tweets, indent=2))

and then count how many tweets are in the response

cat user_tweets.json | grep profile-grid-0-tweet | wc -l

it may show around 50 to 80, which is less than the total media count (1xxx).

after printing some information, it seems the response format changes after the first pagination. in the second pagination, _handle_pagination() finds the cursors but doesn't find the tweets data, so it terminates.

here is the first pagination response for reference:

{
    "data": {
        "user": {
            "result": {
                "__typename": "User",
                "timeline_v2": {
                    "timeline": {
                        "instructions": [
                            {
                                "type": "TimelineClearCache"
                            },
                            {
                                "type": "TimelineTerminateTimeline",
                                "direction": "Top"
                            },
                            {
                                "type": "TimelineAddEntries",
                                "entries": [
                                    {
                                        "entryId": "profile-grid-0",
                                        "sortIndex": "1814743397961826304",
                                        "content": {
                                            "entryType": "TimelineTimelineModule",
                                            "__typename": "TimelineTimelineModule",
                                            "items": [
                                                {
                                                    "entryId": "profile-grid-0-tweet-1814662136330936643",
                                           ....

and this is the next response:

{
    "data": {
        "user": {
            "result": {
                "__typename": "User",
                "timeline_v2": {
                    "timeline": {
                        "instructions": [
                            {
                                "type": "TimelineAddToModule",
                                "moduleItems": [
                                    {
                                        "entryId": "profile-grid-0-tweet-1764602425930060039",
                                        "item": {
                                            "itemContent": {
                                                "itemType": "TimelineTweet",
                                                "__typename": "TimelineTweet",
                                                "tweet_results": {
                                                    "result": {
                                                        "__typename": "Tweet",
                                                        "rest_id": "1764602425930060039",
                                                        ....... 

it looks like the tweets are not in the TimelineAddEntries

thanks

[iSarabjitDhiman]

I am refactoring the code and will fix all these bugs.