update method of jwt token checking, output directory name and readme…

… file (#10)

* update readme file by @kinkwanchan

* update the method to check if jwt token is expired

* output video to a directory named after the channel name when not in channel mode.

---------

Co-authored-by: Kin Kwan Chan <10451800+kinkwanchan@users.noreply.github.com>

README.md

@@ -1,28 +1,38 @@
 `ncp-downloader` is a tool for network performance testing by downloading videos from a well-known online video platform. 
 
+<p align="left">
+  <a href="#installation">Installation</a>
+   • 
+  <a href="#usage">Usage</a>
+   • 
+  <a href="#disclaimer">Disclaimer</a>
+   • 
+  <a href="#license">License</a>
+</p>
+
 # Installation
 Download the latest release from the [Releases](https://github.com/sakkyoi/ncp-downloader/releases/latest) page
 and extract it to the directory of your choice on your local machine.
 
 # Usage
 `ncp QUERY [OUTPUT_DIR] [OPTIONS]`
 
-QUERY is a URL of video or channel.
+`QUERY`: URL of the video or channel.
 
 ```
--r RESOLUTION, --resolution RESOLUTION      Target resolution. Defaults to highest resolution.
+-r RESOLUTION, --resolution RESOLUTION      Target resolution. Defaults to highest resolution available.
 -R, --resume                                Resume download.
--t, --transcode                             Transcode video.
---ffmpeg FFMPEG                             Path to ffmpeg. Defaults to ffmpeg in PATH.
---vcodec VCODEC                             Video codec for transcoding.
---acodec ACODEC                             Audio codec for transcoding.
+-t, --transcode                             Transcode downloaded videos.
+--ffmpeg /PATH/TO/FFMPEG                    Path to ffmpeg. Defaults to ffmpeg stored in PATH.
+--vcodec VCODEC                             Video codec for ffmpeg transcoding.
+--acodec ACODEC                             Audio codec for ffmpeg transcoding.
 --ffmpeg-options FFMPEG_OPTIONS             Additional ffmpeg options. (e.g. --ffmpeg-options "-acodec copy -vcodec copy")
---thread THREAD                             Number of threads for downloading. Defaults to 1. (highly not recommended)
---select-manually                           Select video manually. This option only works with channel.
+--thread THREAD                             Number of threads for downloading. Defaults to 1. (NOT RECOMMENDED TO EDIT)
+--select-manually                           Manually select videos to download. Only works when downloading the whole channel.
 --username USERNAME                         Username for login.
 --password PASSWORD                         Password for login.
---debug                                     Enable debug mode.
---help                                      Show help message.
+--debug                                     Enable debug mode (displays debug messages).
+--help                                      Show help menu.
 ```
 
 **If login credentials are provided, a session token will be generated and saved locally.
@@ -31,41 +41,42 @@ Sometimes this tool may not function properly, delete temp files and folder to m
 (Feel free to modify the .json file if you understand what you are doing.)
 
 ## --select-manually
-When downloading a channel, you can use this option to select the video manually.
-- `arrow up`/`arrow down`, `arrow left`/`arrow right`: Navigate
-- `space`: Select/deselect
-- `enter`: Start downloading
-- `ctrl+r`: Deselct all
-- `ctrl+a`: Select all
-- `ctrl+w`: Use filter
-
-The filter is case-insensitive and supports lambda expressions. Video will be selected if matched.
-- `/only <keyword>`: Leave only the video that contains the keyword.
-- `/add <keyword>`: Add the video that contains the keyword. (keep the original selection)
-- `/remove <keyword>`: Remove the video that contains the keyword. (keep the original selection)
-- `/lambda <lambda expression>`: Use lambda expression to filter the video.
+When downloading the whole channel, you can use this option to manually select the videos to be downloaded.
+- `Arrow Up`/`Arrow Down`, `Arrow Left`/`Arrow Right`: Navigate
+- `Space`: Select / Deselect
+- `Enter`: Start download
+- `Ctrl + A`: Select all
+- `Ctrl + R`: Deselct all
+- `Ctrl + W`: Use filter
+
+The filter supports either with noarmal keyword-filtering or lambda expression (**Case-Sensitive**). <br>
+Videos will be selected if they match the conditions.
+- `/only <keyword>`: Select videos that contain the keyword only.
+- `/add <keyword>`: Add the videos that contain the keyword to selection.
+- `/remove <keyword>`: Remove the videos that contains the keyword from selection.
+- `/lambda <lambda expression>`: Use lambda expression to filter the videos.
 
 The syntax of the lambda expression does not need to include the `lambda x:` part, and should return a boolean value. <br>
 The following is an example of a lambda expression that selects the video with 
-- title containing "ASMR", 
-- length of title is greater than 30, 
-- index is greater than 10, 
+- title containing "ASMR"; 
+- length of title is greater than 30; 
+- index is greater than 10; 
 - and the content code contains letter "R".
 
 `/lambda "ASMR" in x.title and len(x.title) > 30 and x.index > 10 and "R" in x.content_code`
 
-the x object has the following attributes:
+The x object has the following attributes:
 - `title`: Title of the video.
 - `index`: Index of the video.
 - `content_code`: Content code of the video.
 
 **NOTE: lambda expression is case-sensitive. You can use `.lower()` to make it all lowercase.**<br>
-**python built-in functions and variables are available in the lambda expression.**
+**Python built-in functions and variables are supported in the lambda expression.**
 
+# Disclaimer
 
-## `Using this tool may lead to account suspension or ban. Use it at your own discretion.`
+**`Using this tool may lead to account suspension or ban. Use it at your own discretion.`**
 
-# Disclaimer
 Please do use this tool responsibly and respect the rights of the content creators.<br>
 You should **ONLY** use this tool for network performance testing under any circumstances.<br>
 Data downloaded using this tool should be for personal use only. Obtain permission from the original creator before use.

api/api.py

@@ -64,7 +64,7 @@ def __init__(self, site_base: str, username: Optional[str], password: Optional[s
 
         # initial auth
         if username is not None and password is not None:
-            self.auth = NCPAuth(username, password, site_base,
+            self.auth = NCPAuth(username, password, site_base, self.fanclub_site_id,
                                 self.platform_id, self.auth_client_id, self.auth_base,
                                 urlparse(self.api_base).netloc)
         else:

api/auth.py

@@ -1,13 +1,20 @@
 from typing import Tuple
+from enum import Enum
 
 import random
 import base64
 import hashlib
 from requests import session
 import json
 from urllib.parse import urlencode, urlparse, parse_qs
-import jwt
-import time
+
+
+class Method(Enum):
+    """
+    Refresh Enum
+    """
+    LOGIN = 1
+    REFRESH = 2
 
 
 class NCPAuth(object):
@@ -23,18 +30,21 @@ class NCPAuth(object):
         auth0_domain (str): auth0 domain
         audience (str): audience
     """
-    def __init__(self, username: str, password: str,
-                 site_base: str, platform_id: str, client_id: str, auth0_domain: str, audience: str) -> None:
+    def __init__(self, username: str, password: str, site_base: str, site_id: str,
+                 platform_id: str, client_id: str, auth0_domain: str, audience: str) -> None:
         self.session = session()
 
         self.username = username
         self.password = password
         self.site_base = site_base
+        self.site_id = site_id
         self.platform_id = platform_id
         self.client_id = client_id
         self.auth0_domain = auth0_domain
         self.audience = audience
 
+        self.api_user_info = f'https://{self.site_base}/fc/fanclub_sites/{self.site_id}/user_info'
+
         # this is used to get openid configuration like authorization_endpoint, token_endpoint
         self.openid_configuration = f'https://{self.auth0_domain}/.well-known/openid-configuration'
         self.authorization_endpoint, self.token_endpoint = self.__initial_openid()
@@ -58,7 +68,20 @@ def __init__(self, username: str, password: str,
         self.access_token, self.refresh_token = self.__initial_token()
 
     def __str__(self) -> str:
-        self.__auto_refresh()
+        """
+        Auto refresh access token if expired, and return it
+        If access token is expired, refresh it
+        If refresh token is expired, login again
+        If login failed, raise RuntimeError
+        """
+        # check if access token is expired
+        if not self.__check_status():
+            try:
+                self.access_token, self.refresh_token = self.__request_token(Method.REFRESH)
+            except RuntimeError:
+                # if failed to refresh, login again
+                self.access_token, self.refresh_token = self.__request_token(Method.LOGIN)
+
         return self.access_token
 
     def __initial_openid(self) -> Tuple[str, str]:
@@ -71,12 +94,17 @@ def __initial_openid(self) -> Tuple[str, str]:
         return openid_configuration['authorization_endpoint'], openid_configuration['token_endpoint']
 
     def __initial_token(self) -> Tuple[str, str]:
+        """
+        Initial token
+
+        Try to get tokens from file, if not found, login
+        """
         try:
             with open(f'tokens_{hashlib.md5(self.username.encode()).hexdigest()}.json', 'r') as f:
                 tokens = json.load(f)
                 return tokens['access_token'], tokens['refresh_token']
         except FileNotFoundError:
-            return self.__login()
+            return self.__request_token(Method.LOGIN)
 
     def __prepare_authorize_url(self) -> str:
         """
@@ -106,64 +134,52 @@ def __prepare_authorize_url(self) -> str:
 
         return f'{self.authorization_endpoint}?{urlencode(params)}'
 
-    def __login(self) -> Tuple[str, str]:
-        """
-        Login, or Initial from stored token
-
-        workflow:
-        - get authorize url -> this will redirect to login page
-        - post login page with username and password -> this will redirect to redirect uri with code
-        - post token endpoint with code -> this will return access token and refresh token
-        """
-        r_login_page = self.session.get(self.__prepare_authorize_url())
-        if r_login_page.status_code != 200:
-            raise RuntimeError('Failed to get login page')
-
-        r_redirect = self.session.post(r_login_page.url, {
-            'username': self.username,
-            'password': self.password,
-            'state': parse_qs(urlparse(r_login_page.url).query)['state'][0]
-        }, headers=self.headers)
-        if r_redirect.status_code != 404 and 'code' not in parse_qs(urlparse(r_redirect.url).query):
-            raise RuntimeError('Failed to login')
-
-        r_token = self.session.post(self.token_endpoint, {
-            'client_id': self.client_id,
-            'code_verifier': self.code_verifier,
-            'grant_type': 'authorization_code',
-            'code': parse_qs(urlparse(r_redirect.url).query)['code'][0],
-            'redirect_uri': self.redirect_uri
-        }, headers=self.headers)
-        if r_token.status_code != 200 or 'access_token' not in r_token.json() or 'refresh_token' not in r_token.json():
-            raise RuntimeError('Failed to get access token')
-
-        token = r_token.json()
-
-        # dump tokens to file
-        with open(f'tokens_{hashlib.md5(self.username.encode()).hexdigest()}.json', 'w') as f:
-            json.dump({
-                'access_token': token['access_token'],
-                'refresh_token': token['refresh_token']
-            }, f)
-
-        return token['access_token'], token['refresh_token']
-
-    def __refresh(self) -> Tuple[str, str]:
-        """
-        Refresh access token
-
-        workflow:
-        - post token endpoint with refresh token -> this will return access token and refresh token
-        """
-        r_token = self.session.post(self.token_endpoint, {
-            'client_id': self.client_id,
-            'redirect_uri': self.redirect_uri,
-            'grant_type': 'refresh_token',
-            'refresh_token': self.refresh_token
-        }, headers=self.headers)
-        if r_token.status_code != 200 or 'access_token' not in r_token.json() or 'refresh_token' not in r_token.json():
-            # failed to refresh access token, login again
-            raise RuntimeError('Failed to refresh access token')
+    def __request_token(self, method: Method) -> Tuple[str, str]:
+        match method:
+            case Method.LOGIN:
+                # login
+                # workflow:
+                # - get authorize url -> this will redirect to login page
+                # - post login page with username and password -> this will redirect to redirect uri with code
+                # - post token endpoint with code -> this will return access token and refresh token
+                r_login_page = self.session.get(self.__prepare_authorize_url())
+                if r_login_page.status_code != 200:
+                    raise RuntimeError('Failed to get login page')
+
+                r_redirect = self.session.post(r_login_page.url, {
+                    'username': self.username,
+                    'password': self.password,
+                    'state': parse_qs(urlparse(r_login_page.url).query)['state'][0]
+                }, headers=self.headers)
+                if r_redirect.status_code != 404 and 'code' not in parse_qs(urlparse(r_redirect.url).query):
+                    raise RuntimeError('Failed to login')
+
+                r_token = self.session.post(self.token_endpoint, {
+                    'client_id': self.client_id,
+                    'code_verifier': self.code_verifier,
+                    'grant_type': 'authorization_code',
+                    'code': parse_qs(urlparse(r_redirect.url).query)['code'][0],
+                    'redirect_uri': self.redirect_uri
+                }, headers=self.headers)
+                if r_token.status_code != 200 or \
+                        'access_token' not in r_token.json() or 'refresh_token' not in r_token.json():
+                    raise RuntimeError('Failed to get access token')
+            case Method.REFRESH:
+                # refresh access token
+                # workflow:
+                # - post token endpoint with refresh token -> this will return access token and refresh token
+                r_token = self.session.post(self.token_endpoint, {
+                    'client_id': self.client_id,
+                    'redirect_uri': self.redirect_uri,
+                    'grant_type': 'refresh_token',
+                    'refresh_token': self.refresh_token
+                }, headers=self.headers)
+                if r_token.status_code != 200 or \
+                        'access_token' not in r_token.json() or 'refresh_token' not in r_token.json():
+                    # failed to refresh access token, login again
+                    raise RuntimeError('Failed to refresh access token')
+            case _:
+                raise ValueError('Invalid refresh type')
 
         token = r_token.json()
 
@@ -176,22 +192,14 @@ def __refresh(self) -> Tuple[str, str]:
 
         return token['access_token'], token['refresh_token']
 
-    def __auto_refresh(self) -> None:
+    def __check_status(self) -> bool:
         """
-        Auto refresh access token
-        If access token is expired, refresh it
-        If refresh token is expired, login again
+        Check auth status(by user_info endpoint)
+        This is used to check if access token is expired
         """
-        # check if access token is expired
-        jwt_payload = jwt.decode(self.access_token, options={"verify_signature": False})
-        if jwt_payload['exp'] < int(time.time()):
-            try:
-                self.access_token, self.refresh_token = self.__refresh()
-            except RuntimeError:
-                # if failed to refresh, login again
-                self.access_token, self.refresh_token = self.__login()
+        r = self.session.post(self.api_user_info, headers=self.headers)
 
-        # if access token is not expired, do nothing
+        return r.status_code == 200
 
     @staticmethod
     def __rand(size=43):