Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 23 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
root = true

[*]
end_of_line = lf
insert_final_newline = true
charset = utf-8
trim_trailing_whitespace = true

[*.{py,yml}]
indent_style = space
max_line_length = 79

[*.py]
indent_size = 4

[*.rst]
indent_size = 3

[Makefile]
indent_style = tab

[*.yml]
indent_size = 2
1 change: 1 addition & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,7 @@ jobs:
- name: tox
run: tox
env:
TOXENV: ${{ matrix.toxenv }}
COVERALLS_PARALLEL: 'true'
COVERALLS_SERVICE_NAME: github
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
Expand Down
72 changes: 70 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Features

* Since version 2.0.0 using `caio`_, which contains linux ``libaio`` and two
thread-based implementations (c-based and pure-python).
* AIOFile has no internal pointer. You should pass ``offset`` and
* ``AIOFile`` has no internal pointer. You should pass ``offset`` and
``chunk_size`` for each operation or use helpers (Reader or Writer).
The simples way is to use ``async_open`` for creating object with
file-like interface.
Expand All @@ -57,7 +57,7 @@ Limitations

* Linux native AIO implementation is not able to open special files.
Asynchronous operations against special fs like ``/proc/`` ``/sys/`` are not
supported by the kernel. It's not a `aiofile`s or `caio` issue.
supported by the kernel. It's not a ``aiofile`` or ``caio`` issue.
In this cases, you might switch to thread-based implementations
(see troubleshooting_ section).
However, when used on supported file systems, the linux implementation has a
Expand Down Expand Up @@ -262,6 +262,61 @@ these files using compatible context objects.

asyncio.run(main())


``clone`` helper
~~~~~~~~~~~~~~~~

Asynchronous context at a low level supports a limited number of concurrency operations, no matter how many
file descriptors are open. This means that you can make a second file-like object with its own offset stub
for one descriptor without opening the file several times.

.. code-block:: python

"""
This example counts multiple hash functions from the file passed as the first argument.
The hash functions are counted competitively, and the results are printed in the order of hashing completion.
"""
import asyncio
import hashlib
import sys

import aiofile


async def hasher(name, hash_func, afp):
loop = asyncio.get_running_loop()
async for chunk in afp.iter_chunked(2 ** 20):
await loop.run_in_executor(None, hash_func.update, chunk)
print(name, hash_func.hexdigest())


async def main():
async with aiofile.async_open(sys.argv[1], "rb") as source:
hashers = [
("MD5", hashlib.md5()),
("SHA1", hashlib.sha1()),
("SHA256", hashlib.sha256()),
("SHA512", hashlib.sha512()),
("SHA3 224", hashlib.sha3_224()),
("SHA3 256", hashlib.sha3_256()),
("SHA3 512", hashlib.sha3_512()),
("Blake2b", hashlib.blake2b()),
]

await asyncio.gather(*[
hasher(name, hash_func, await aiofile.clone(source))
for name, hash_func in hashers
])


asyncio.run(main())


.. note::

In fact this will most likely perform very bad under windows, so if that's your target platform it's not
worth it to apply this optimization.

Low-level API
++++++++++++++

Expand Down Expand Up @@ -462,6 +517,19 @@ Async CSV Dict Reader

asyncio.run(main())

Limitations
-----------

The underlying library ``caio`` uses multiple implementations of asynchronous
IO to ensure that it works across operating systems.

This imposes a limitation on the mode of working with files; files are
always opened in binary mode, and string encoding and decoding takes place
in the library, not in the operating system.

Thus the presence of ``b`` in the file opening mode determines the behavior
of the ``read`` and ``write`` functions, while the file descriptor will
always be opened in binary mode.

.. _troubleshooting:

Expand Down
6 changes: 4 additions & 2 deletions aiofile/__init__.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from .aio import AIOFile
from .utils import (
BinaryFileWrapper, FileIOWrapperBase, LineReader, Reader, TextFileWrapper,
Writer, async_open,
BinaryFileWrapper, FileIOCloner, FileIOWrapperBase, LineReader, Reader,
TextFileWrapper, Writer, async_open, clone,
)
from .version import (
__author__, __version__, author_info, package_info, package_license,
Expand All @@ -13,6 +13,7 @@
"AIOFile",
"BinaryFileWrapper",
"FileIOWrapperBase",
"FileIOCloner",
"LineReader",
"Reader",
"TextFileWrapper",
Expand All @@ -21,6 +22,7 @@
"__version__",
"async_open",
"author_info",
"clone",
"package_info",
"package_license",
"project_home",
Expand Down
Loading