+
+ 
+
+
+
+ 
+
+
@@ -41,23 +49,22 @@
English | 中文
KToolBox is a useful CLI tool for downloading posts content in - Kemono.party / Kemono.su + Kemono.su / Kemono.party
@@ -16,6 +16,14 @@
-
## Features
-- Support for **multi-thread** downloads (technically, coroutine)
-- **Retry** after download failed
-- Ability to download individual post as well as **all post** by a specified creator/artist
-- **Update downloaded** creator/artist directories to the latest status
-- Customize the **structure** of downloaded post/creator **directories**
-- Search for creators/artists and posts, and **export the results**
-- **Cross-platform** support & **iOS shortcuts** available
-- For Coomer.su / Coomer.party support, check document [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/) for more.
+- Supports concurrent downloads
+- Automatically retries API calls and downloads after failures
+- Allows downloading individual posts or **all posts** of a specified artist
+- Can **update downloaded** artist directories to the latest state
+- Supports customizable **file and directory naming formats** and **directory structures** for downloaded posts/artists
+- Enables excluding **specified file formats** or downloading only specified formats
+- Allows searching for artists and posts, with options to export results
+- Compatible with all platforms, with iOS shortcuts provided
+- For support related to _Coomer.su / Coomer.party_, please refer to the documentation: [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/)
## Dev Plan
- [ ] GUI
-- [x] Add uvloop support for Unix platform
+- [ ] Discord support
## Tutorial
diff --git a/README_zh-CN.md b/README_zh-CN.md
index 1caee4d..03774c9 100644
--- a/README_zh-CN.md
+++ b/README_zh-CN.md
@@ -8,7 +8,7 @@
KToolBox 是一个用于下载 - Kemono.party / Kemono.su + Kemono.su / Kemono.party 中作品内容的实用命令行工具
@@ -17,6 +17,14 @@
+
+
+
+
+
+
+
+
@@ -42,23 +50,22 @@
English | 中文
-
-
## 功能
-- 支持 **多线程** 下载（技术上是协程）
-- 下载失败后进行 **重试**
+- 支持并发下载
+- API 调用和下载失败后 **自动重试**
- 支持下载单个作品以及指定的画师的 **所有作品**
- 可 **更新已下载** 的画师目录至最新状态
-- 可自定义下载的作品/画师 **目录结构**
-- 可搜索画师和作品，并 **导出结果**
-- 支持全平台，并提供 **iOS 快捷指令**
-- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)。
+- 支持自定义下载的作品/画师 **文件和目录名格式**、**目录结构**
+- 支持排除 **指定格式** 的文件或仅下载指定格式的文件
+- 可搜索画师和作品，并导出结果
+- 支持全平台，并提供 iOS 快捷指令
+- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/)
## 开发计划
- [ ] GUI
-- [x] 对 Unix 平台增加 uvloop 支持
+- [ ] Discord 下载支持
## 使用方法
diff --git a/docs/en/index.md b/docs/en/index.md
index 61b6571..3e551af 100644
--- a/docs/en/index.md
+++ b/docs/en/index.md
@@ -39,13 +39,15 @@
## Features
-- Support for **multi-thread** downloads (technically, coroutine)
-- **Retry** after download failed
-- Ability to download individual post as well as **all post** by a specified creator/artist
-- **Update downloaded** creator/artist directories to the latest status
-- Customize the **structure** of downloaded post/creator **directories**
-- Search for creators/artists and posts, and **export the results**
-- **Cross-platform** support & **iOS shortcuts** available
+- Supports concurrent downloads
+- Automatically retries API calls and downloads after failures
+- Allows downloading individual posts or **all posts** of a specified artist
+- Can **update downloaded** artist directories to the latest state
+- Supports customizable **file and directory naming formats** and **directory structures** for downloaded posts/artists
+- Enables excluding **specified file formats** or downloading only specified formats
+- Allows searching for artists and posts, with options to export results
+- Compatible with all platforms, with iOS shortcuts provided
+- For support related to _Coomer.su / Coomer.party_, please refer to the documentation: [Coomer](https://ktoolbox.readthedocs.io/latest/coomer/)
## Tutorial
diff --git a/docs/zh/index.md b/docs/zh/index.md
index 4eabc3a..cb8f071 100644
--- a/docs/zh/index.md
+++ b/docs/zh/index.md
@@ -8,7 +8,7 @@
KToolBox 是一个用于下载 - Kemono.party / Kemono.su + Kemono.su / Kemono.party 中作品内容的实用命令行工具
@@ -40,13 +40,15 @@ ## 功能 -- 支持 **多线程** 下载（技术上是协程） -- 下载失败后进行 **重试** +- 支持并发下载 +- API 调用和下载失败后 **自动重试** - 支持下载单个作品以及指定的画师的 **所有作品** - 可 **更新已下载** 的画师目录至最新状态 -- 可自定义下载的作品/画师 **目录结构** -- 可搜索画师和作品，并 **导出结果** -- 支持全平台，并提供 **iOS 快捷指令** +- 支持自定义下载的作品/画师 **文件和目录名格式**、**目录结构** +- 支持排除 **指定格式** 的文件或仅下载指定格式的文件 +- 可搜索画师和作品，并导出结果 +- 支持全平台，并提供 iOS 快捷指令 +- 对于 Coomer.su / Coomer.party 的支持，请查看文档 [Coomer](https://ktoolbox.readthedocs.io/latest/zh/coomer/) ## 使用方法 diff --git a/ktoolbox/__init__.py b/ktoolbox/__init__.py index ec372d3..fead139 100644 --- a/ktoolbox/__init__.py +++ b/ktoolbox/__init__.py @@ -1,4 +1,4 @@ __title__ = "KToolBox" # noinspection SpellCheckingInspection __description__ = "A useful CLI tool for downloading posts in Kemono.party / .su" -__version__ = "v0.10.0" +__version__ = "v0.11.0" diff --git a/ktoolbox/action/search.py b/ktoolbox/action/search.py index 7b35162..fe5e097 100644 --- a/ktoolbox/action/search.py +++ b/ktoolbox/action/search.py @@ -19,23 +19,23 @@ async def search_creator(id: str = None, name: str = None, service: str = None) :param service: The service for the creator """ - async def inner(**kwargs): - def filter_func(creator: Creator): - """Filter creators with attributes""" - for key, value in kwargs.items(): - if value is None: - continue - elif creator.__getattribute__(key) != value: - return False - return True - - ret = await get_creators() - if not ret: - return ret - creators = ret.data - return ActionRet(data=iter(filter(filter_func, creators))) + def filter_func(creator: Creator): + """Filter creators with attributes""" + if id is not None and creator.id != id: + return False + if name is not None and name not in creator.name: + return False + if service is not None and creator.service != service: + return False + return True - return await inner(id=id, name=name, service=service) + ret = await get_creators() + if not ret: + base_ret = BaseRet.model_validate(ret.model_dump()) + base_ret.data = iter([]) + return base_ret + creators = ret.data + return ActionRet(data=iter(filter(filter_func, creators))) # noinspection PyShadowingBuiltins diff --git a/ktoolbox/api/base.py b/ktoolbox/api/base.py index d6e140e..91dc96b 100644 --- a/ktoolbox/api/base.py +++ b/ktoolbox/api/base.py @@ -69,6 +69,7 @@ class BaseAPI(ABC, Generic[_T]): path: str = "/" method: Literal["get", "post"] extra_validator: Optional[Callable[[str], BaseModel]] = None + client = httpx.AsyncClient(verify=config.ssl_verify) Response = BaseModel """API response model""" @@ -111,14 +112,13 @@ async def request(cls, path: str = None, **kwargs) -> APIRet[_T]: url_parts = [config.api.scheme, config.api.netloc, f"{config.api.path}{path}", '', '', ''] url = str(urlunparse(url_parts)) try: - async with httpx.AsyncClient(verify=config.ssl_verify) as client: - res = await client.request( - method=cls.method, - url=url, - timeout=config.api.timeout, - follow_redirects=True, - **kwargs - ) + res = await cls.client.request( + method=cls.method, + url=url, + timeout=config.api.timeout, + follow_redirects=True, + **kwargs + ) except Exception as e: return APIRet( code=RetCodeEnum.NetWorkError, diff --git a/ktoolbox/api/model/creator.py b/ktoolbox/api/model/creator.py index 69c268f..d9006b6 100644 --- a/ktoolbox/api/model/creator.py +++ b/ktoolbox/api/model/creator.py @@ -1,3 +1,5 @@ +from datetime import datetime + from pydantic import BaseModel __all__ = ["Creator"] @@ -10,11 +12,11 @@ class Creator(BaseModel): """The number of times this creator has been favorited""" id: str """The ID of the creator""" - indexed: float + indexed: datetime """Timestamp when the creator was indexed, Unix time as integer""" name: str """The name of the creator""" service: str """The service for the creator""" - updated: float + updated: datetime """Timestamp when the creator was last updated, Unix time as integer""" diff --git a/ktoolbox/downloader/downloader.py b/ktoolbox/downloader/downloader.py index dc88d34..7c68d7d 100644 --- a/ktoolbox/downloader/downloader.py +++ b/ktoolbox/downloader/downloader.py @@ -29,6 +29,7 @@ class Downloader: """ :ivar _save_filename: The actual filename for saving. """ + client = httpx.AsyncClient(verify=config.ssl_verify) def __init__( self, @@ -172,56 +173,55 @@ async def run( tqdm_class: Type[std_tqdm] = tqdm_class or tqdm.asyncio.tqdm async with self._lock: - async with httpx.AsyncClient(verify=config.ssl_verify) as client: - async with client.stream( - method="GET", - url=self._url, - follow_redirects=True, - timeout=config.downloader.timeout - ) as res: # type: httpx.Response - if res.status_code != httpx.codes.OK: - return DownloaderRet( - code=RetCodeEnum.GeneralFailure, - message=generate_msg( - "Download failed", - status_code=res.status_code, - filename=save_filepath - ) + async with self.client.stream( + method="GET", + url=self._url, + follow_redirects=True, + timeout=config.downloader.timeout + ) as res: # type: httpx.Response + if res.status_code != httpx.codes.OK: + return DownloaderRet( + code=RetCodeEnum.GeneralFailure, + message=generate_msg( + "Download failed", + status_code=res.status_code, + filename=save_filepath ) - - # Get filename for saving and check if file exists (Second-time duplicate file check) - # Priority order can be referenced from the constructor's documentation - self._save_filename = self._designated_filename or sanitize_filename( - filename_from_headers(res.headers) - ) or server_path_filename - save_filepath = self._path / self._save_filename - file_existed, ret_msg = duplicate_file_check(save_filepath, bucket_file_path) - if file_existed: - return DownloaderRet( - code=RetCodeEnum.FileExisted, - message=generate_msg( - ret_msg, - path=save_filepath - ) - ) - - # Download - temp_filepath = Path(f"{save_filepath}.{config.downloader.temp_suffix}") - total_size = int(length_str) if (length_str := res.headers.get("Content-Length")) else None - async with aiofiles.open(str(temp_filepath), "wb", self._buffer_size) as f: - chunk_iterator = res.aiter_bytes(self._chunk_size) - t = tqdm_class( - desc=self._save_filename, - total=total_size, - disable=not progress, - unit="B", - unit_scale=True + ) + + # Get filename for saving and check if file exists (Second-time duplicate file check) + # Priority order can be referenced from the constructor's documentation + self._save_filename = self._designated_filename or sanitize_filename( + filename_from_headers(res.headers) + ) or server_path_filename + save_filepath = self._path / self._save_filename + file_existed, ret_msg = duplicate_file_check(save_filepath, bucket_file_path) + if file_existed: + return DownloaderRet( + code=RetCodeEnum.FileExisted, + message=generate_msg( + ret_msg, + path=save_filepath ) - async for chunk in chunk_iterator: - if self._stop: - raise CancelledError - await f.write(chunk) - t.update(len(chunk)) # Update progress bar + ) + + # Download + temp_filepath = Path(f"{save_filepath}.{config.downloader.temp_suffix}") + total_size = int(length_str) if (length_str := res.headers.get("Content-Length")) else None + async with aiofiles.open(str(temp_filepath), "wb", self._buffer_size) as f: + chunk_iterator = res.aiter_bytes(self._chunk_size) + t = tqdm_class( + desc=self._save_filename, + total=total_size, + disable=not progress, + unit="B", + unit_scale=True + ) + async for chunk in chunk_iterator: + if self._stop: + raise CancelledError + await f.write(chunk) + t.update(len(chunk)) # Update progress bar # Download finished if config.downloader.use_bucket: diff --git a/pyproject.toml b/pyproject.toml index fc6e722..68b217e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,12 +1,13 @@ [tool.poetry] name = "ktoolbox-pure-py" -version = "v0.10.0" -description = "A useful CLI tool for downloading posts in Kemono.party / .su (Pure Python version)" +version = "v0.11.0" +description = "A useful CLI tool for downloading posts in Kemono.su / .party (Pure Python version)" authors = ["Ljzd-PRO