Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C++ Rewrite #74

Merged
merged 121 commits into from
Jul 15, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
121 commits
Select commit Hold shift + click to select a range
03d9745
Replace cython requirement with pybind11
gentlegiantJGC Jun 21, 2024
09e5ece
Refactor setup.py to work with pybind11 module
gentlegiantJGC Jun 21, 2024
e6b6d16
Clean up stub file
gentlegiantJGC Jun 21, 2024
d113ef7
Don't ignore C++ source files
gentlegiantJGC Jun 21, 2024
a457469
Move and refactor C++ header files
gentlegiantJGC Jun 21, 2024
943068b
Add nbt wrapper classes
gentlegiantJGC Jun 21, 2024
5277bc6
Add abstract base pybind11 classes
gentlegiantJGC Jun 21, 2024
708f7ab
Add int pybind11 classes
gentlegiantJGC Jun 21, 2024
fdc7717
Add float pybind11 classes
gentlegiantJGC Jun 21, 2024
2d0e75d
Add string pybind11 classes
gentlegiantJGC Jun 21, 2024
d7cb085
Add pybind11 array classes
gentlegiantJGC Jun 21, 2024
018e8dc
Add pybind11 module
gentlegiantJGC Jun 21, 2024
29b7353
Add common header
gentlegiantJGC Jun 21, 2024
bae8f2c
Renamed list tag types
gentlegiantJGC Jun 21, 2024
6e2fa1b
Remove cython files from manifest.in
gentlegiantJGC Jun 28, 2024
2dec652
Refactored setup.py
gentlegiantJGC Jun 28, 2024
1cf80e9
Move cython implementation
gentlegiantJGC Jun 28, 2024
8b07d0d
Added generation macros
gentlegiantJGC Jun 28, 2024
c407e4a
Fix variant functions
gentlegiantJGC Jun 28, 2024
74cc227
Move TagWrapper base class out of template
gentlegiantJGC Jun 28, 2024
0f21ded
Added a wrapper variant
gentlegiantJGC Jun 28, 2024
6b4ae1a
Renamed the pybind directory
gentlegiantJGC Jun 28, 2024
483f3e4
Added __init__.py
gentlegiantJGC Jun 28, 2024
9c14884
Added equals functions
gentlegiantJGC Jun 28, 2024
4ac488b
Added wrap node function
gentlegiantJGC Jun 28, 2024
52b1b7c
Added list functions
gentlegiantJGC Jun 28, 2024
5a04c50
Added a template function to find the variant index
gentlegiantJGC Jun 28, 2024
505604c
Added compound and named tag classes
gentlegiantJGC Jun 28, 2024
c6957c8
Added most of the list tag implementation
gentlegiantJGC Jun 28, 2024
1476d5c
Fix ListTag_set
gentlegiantJGC Jun 28, 2024
7fceafb
Fix py_getter
gentlegiantJGC Jun 28, 2024
e402053
Reformat
gentlegiantJGC Jun 28, 2024
549719c
Improve repr output
gentlegiantJGC Jun 28, 2024
e067e50
Switch to Py_ssize_t
gentlegiantJGC Jun 28, 2024
432926d
Added setitem and delitem
gentlegiantJGC Jun 28, 2024
707f608
Refactor abstract method
gentlegiantJGC Jun 28, 2024
47487e1
Fix array constructor
gentlegiantJGC Jun 28, 2024
af271c6
Add tag_id value
gentlegiantJGC Jun 28, 2024
a62d500
Reformat
gentlegiantJGC Jun 28, 2024
ac873c5
Extended __init__.py
gentlegiantJGC Jul 2, 2024
16654db
Added string encoding and presets
gentlegiantJGC Jul 2, 2024
5ed7b2d
Move is_shared_ptr into common
gentlegiantJGC Jul 3, 2024
41e4dec
Rename module_
gentlegiantJGC Jul 3, 2024
b52a1e6
Fix numerical constructors
gentlegiantJGC Jul 3, 2024
7592da8
Fix macro
gentlegiantJGC Jul 3, 2024
cbcfef0
Added function to unwrap node
gentlegiantJGC Jul 3, 2024
5c9a101
Added compound tag
gentlegiantJGC Jul 3, 2024
6523f67
Added NamedTag
gentlegiantJGC Jul 4, 2024
537793e
Added binary load functions
gentlegiantJGC Jul 4, 2024
03b83ce
Added load overloads
gentlegiantJGC Jul 4, 2024
029ce6f
Bumped to numpy 2.0
gentlegiantJGC Jul 4, 2024
7761eff
Fix list equals
gentlegiantJGC Jul 4, 2024
7ec0ecf
Moved the encoding data classes
gentlegiantJGC Jul 4, 2024
c05be93
Added binary nbt writing
gentlegiantJGC Jul 4, 2024
7f0cc19
Improved read and write type hints
gentlegiantJGC Jul 5, 2024
375e43e
Refactored read and write
gentlegiantJGC Jul 8, 2024
8335180
Added error message macro
gentlegiantJGC Jul 8, 2024
7ac71ec
Added copy and deepcopy
gentlegiantJGC Jul 8, 2024
3240bb0
Implemented save_to
gentlegiantJGC Jul 8, 2024
c23b5bb
Removed old components
gentlegiantJGC Jul 8, 2024
3968aaa
Added pickle support
gentlegiantJGC Jul 8, 2024
994c6dc
Added conversion to snbt and refactored
gentlegiantJGC Jul 8, 2024
d8ec492
Expose lower level encoding functions
gentlegiantJGC Jul 9, 2024
4d3795c
Fixed function calls
gentlegiantJGC Jul 9, 2024
022c16c
Added easier to call string encoding functions
gentlegiantJGC Jul 10, 2024
2635849
Partially implemented snbt parsing
gentlegiantJGC Jul 10, 2024
ebde4b8
Added from_snbt
gentlegiantJGC Jul 10, 2024
98cc246
Renamed load functions
gentlegiantJGC Jul 10, 2024
d3306bf
Removed exceptions
gentlegiantJGC Jul 10, 2024
54ba1be
Added missing return
gentlegiantJGC Jul 11, 2024
9af51ae
Fixed NamedTag repr
gentlegiantJGC Jul 11, 2024
340061d
Added more float snbt parse tests
gentlegiantJGC Jul 11, 2024
774763a
Added int, float and string SNBT parsing
gentlegiantJGC Jul 11, 2024
2827e77
Removed numpy build-time dependency
gentlegiantJGC Jul 11, 2024
e0ce2cf
Modified numpy requirement
gentlegiantJGC Jul 11, 2024
4909ef6
Remove python errors
gentlegiantJGC Jul 11, 2024
2c3c3db
Remove template test
gentlegiantJGC Jul 11, 2024
a56b334
Added NamedTag serialisation methods
gentlegiantJGC Jul 11, 2024
c9240ce
Fixed some snbt read and write issues
gentlegiantJGC Jul 11, 2024
be53fa9
Revert lower bounds check
gentlegiantJGC Jul 11, 2024
4a55f85
Improved constructors
gentlegiantJGC Jul 12, 2024
85ebd97
Convert cast_error to type_error
gentlegiantJGC Jul 12, 2024
5affa66
Fixed getitem
gentlegiantJGC Jul 12, 2024
2699746
Fixed errors
gentlegiantJGC Jul 12, 2024
84d684d
Added missing import
gentlegiantJGC Jul 12, 2024
b3533cd
Added snbt list parsing
gentlegiantJGC Jul 12, 2024
cc8bec6
Fixed float repr and string
gentlegiantJGC Jul 12, 2024
1a24886
Fixed int construction
gentlegiantJGC Jul 12, 2024
d9ac1c1
Improved int tests
gentlegiantJGC Jul 12, 2024
89e35bd
Added type_error exception
gentlegiantJGC Jul 12, 2024
587ff8f
Modified count behaviour
gentlegiantJGC Jul 12, 2024
660c1c1
Fixed ListTag_append
gentlegiantJGC Jul 12, 2024
0d4bf77
Updated tests
gentlegiantJGC Jul 12, 2024
41f4478
Added sanity check exception
gentlegiantJGC Jul 12, 2024
00d6f3a
Renamed files to give unique names
gentlegiantJGC Jul 12, 2024
fcc5cc9
Handle null case in ListTag.__contains__
gentlegiantJGC Jul 12, 2024
1e56a9d
Changed exception type
gentlegiantJGC Jul 12, 2024
0476489
Seek past closing bracket
gentlegiantJGC Jul 12, 2024
f9618ab
Updated tests
gentlegiantJGC Jul 12, 2024
01b6c01
Improved string construction
gentlegiantJGC Jul 12, 2024
27e8047
Added filepath_or_writable default argument
gentlegiantJGC Jul 12, 2024
15eaa01
Fixed list iterator
gentlegiantJGC Jul 12, 2024
d3d02e9
Fixed list methods
gentlegiantJGC Jul 12, 2024
c368ea6
Fixed list tag setitem
gentlegiantJGC Jul 15, 2024
8ea927b
Reformatted
gentlegiantJGC Jul 15, 2024
ffb1255
Remove pragma once from source file
gentlegiantJGC Jul 15, 2024
4fb276d
Added brackets to binary operators
gentlegiantJGC Jul 15, 2024
314e23d
Added limits import
gentlegiantJGC Jul 15, 2024
87d9bd4
Added and moved pragma once
gentlegiantJGC Jul 15, 2024
87d8ba7
Change header
gentlegiantJGC Jul 15, 2024
495ae04
Add std prefix
gentlegiantJGC Jul 15, 2024
ead15c4
Added missing imports
gentlegiantJGC Jul 15, 2024
a234d4a
Changed set to unordered_set
gentlegiantJGC Jul 15, 2024
1343cd9
Change exception to runtime_error
gentlegiantJGC Jul 15, 2024
525976d
Added virtual destructors to ABCs
gentlegiantJGC Jul 15, 2024
0705a06
Added space
gentlegiantJGC Jul 15, 2024
02ac0a7
Moved tag creation into template function
gentlegiantJGC Jul 15, 2024
f910fec
Removed cython version
gentlegiantJGC Jul 15, 2024
151c6a6
Remove tests from sdist
gentlegiantJGC Jul 15, 2024
5689dd3
Changed package finding
gentlegiantJGC Jul 15, 2024
8dbe6f4
Include source code release
gentlegiantJGC Jul 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 0 additions & 6 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,3 @@ temp
*.vs/
*.vcxproj*
*.sln

# C++ files
*.c
!**/_cpp/**/*.c
*.cpp
!**/_cpp/**/*.cpp
3 changes: 2 additions & 1 deletion MANIFEST.in
Original file line number Diff line number Diff line change
@@ -1 +1,2 @@
recursive-include src *.pyi py.typed *.py *.pyx *.pxd *.cpp *.hpp
prune tests
recursive-include src *.pyi py.typed *.py *.cpp *.hpp
14 changes: 7 additions & 7 deletions docs_source/getting_started.rst
Original file line number Diff line number Diff line change
Expand Up @@ -52,32 +52,32 @@
bedrock_named_tag: NamedTag

# Load binary NBT
named_tag = amulet_nbt.load(
named_tag = amulet_nbt.read_nbt(
"the/path/to/your/binary/nbt/file",
preset=java_encoding,
compressed=True, # These inputs must be specified as keyword inputs like this.
) # from a file
named_tag = amulet_nbt.load(
named_tag = amulet_nbt.read_nbt(
"the/path/to/your/binary/nbt/file",
compressed=True, # These inputs must be specified as keyword inputs like this.
little_endian=False, # If you do not define them they will default to these values
string_encoding=mutf8_encoding
) # from a file
named_tag = amulet_nbt.load(b'<nbt file bytes>') # from a bytes object
named_tag = amulet_nbt.read_nbt(b'<nbt file bytes>') # from a bytes object

# Note that Java Edition usually uses compressed modified UTF-8.
java_named_tag = amulet_nbt.load(
java_named_tag = amulet_nbt.read_nbt(
"the/path/to/your/binary/java/nbt/file",
string_encoding=mutf8_encoding
)

# Bedrock edition data is stored in little endian format and uses non-compressed UTF-8 but can also have arbitrary bytes.
bedrock_named_tag = amulet_nbt.load(
bedrock_named_tag = amulet_nbt.read_nbt(
"the/path/to/your/binary/bedrock/nbt/file",
preset=bedrock_encoding,
compressed=False,
)
bedrock_named_tag = amulet_nbt.load(
bedrock_named_tag = amulet_nbt.read_nbt(
"the/path/to/your/binary/bedrock/nbt/file",
compressed=False,
little_endian=True,
Expand Down Expand Up @@ -117,7 +117,7 @@


# You can also parse the stringified NBT format used in Java commands.
tag = amulet_nbt.from_snbt('{key1: "value", key2: 0b, key3: 0.0f}')
tag = amulet_nbt.read_snbt('{key1: "value", key2: 0b, key3: 0.0f}')
# tag should look like this
# TAG_Compound(
# key1: TAG_String("value"),
Expand Down
23 changes: 5 additions & 18 deletions docs_source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -186,24 +186,11 @@

These are functions to load the binary and stringified NBT formats.

.. autofunction:: amulet_nbt.load
.. autofunction:: amulet_nbt.load_array
.. autofunction:: amulet_nbt.read_nbt
.. autofunction:: amulet_nbt.read_nbt_array
.. autoclass:: amulet_nbt.ReadOffset
:members:
.. autofunction:: amulet_nbt.from_snbt


############
Exceptions
############

.. autoexception:: amulet_nbt.NBTError
.. autoexception:: amulet_nbt.NBTLoadError
:show-inheritance:
.. autoexception:: amulet_nbt.NBTFormatError
:show-inheritance:
.. autoexception:: amulet_nbt.SNBTParseError
:show-inheritance:
.. autofunction:: amulet_nbt.read_snbt


#################
Expand All @@ -217,7 +204,7 @@ These are instances of a class storing C++ functions to encode and decode string
:inherited-members:
:undoc-members:

They can be passed to the string_encoding argument in to_nbt, save_to, load and load_array to control the string encoding behaviour.
They can be passed to the string_encoding argument in to_nbt, save_to, read_nbt and read_nbt_array to control the string encoding behaviour.

The usual string encoding scheme is called UTF-8.

Expand All @@ -237,7 +224,7 @@ Java Edition uses a modified version of UTF-8 implemented by the Java programmin
Encoding Presets
##################

The string encoding and endianness can be defined separatly but for simplicity the following presets have been defined.
The string encoding and endianness can be defined separately but for simplicity the following presets have been defined.

.. autodata:: amulet_nbt.java_encoding
.. autodata:: amulet_nbt.bedrock_encoding
Expand Down
3 changes: 1 addition & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,7 @@
requires = [
"setuptools >= 42",
"wheel",
"cython ~= 3.0",
"versioneer",
"numpy ~= 1.17"
"pybind11 ~= 2.12",
]
build-backend = "setuptools.build_meta"
7 changes: 5 additions & 2 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ platforms = any
[options]
package_dir=
=src
packages = find_namespace:
packages = find:
zip_safe = False
python_requires = ~=3.11
install_requires =
numpy ~= 1.17
numpy >= 1.17, < 3.0

[options.packages.find]
where=src
Expand All @@ -30,6 +30,9 @@ amulet_nbt =
py.typed
*.pyd
*.so
*.dylib
**/*.hpp
cpp/**/*.cpp


[options.extras_require]
Expand Down
30 changes: 23 additions & 7 deletions setup.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
from setuptools import setup
from Cython.Build import cythonize
from setuptools import setup, Extension
import versioneer
import numpy
import sysconfig
from distutils import ccompiler
import sys
import pybind11
import glob

if (sysconfig.get_config_var("CXX") or ccompiler.get_default_compiler()).split()[
0
Expand All @@ -20,8 +20,24 @@
setup(
version=versioneer.get_version(),
cmdclass=versioneer.get_cmdclass(),
include_dirs=[numpy.get_include()],
ext_modules=cythonize(
f"src/**/*.pyx", language_level=3, aliases={"CPPCARGS": CompileArgs}
),
libraries=[
(
"amulet_nbt",
dict(
sources=glob.glob("src/amulet_nbt/cpp/**/*.cpp", recursive=True),
include_dirs=["src/amulet_nbt/include"],
cflags=CompileArgs,
),
)
],
ext_modules=[
Extension(
name="amulet_nbt._nbt",
sources=glob.glob("src/amulet_nbt/pybind/**/*.cpp", recursive=True),
include_dirs=["src/amulet_nbt/include", pybind11.get_include()],
libraries=["amulet_nbt"],
define_macros=[("PYBIND11_DETAILED_ERROR_MESSAGES", None)],
extra_compile_args=CompileArgs,
)
],
)
193 changes: 89 additions & 104 deletions src/amulet_nbt/__init__.pyx → src/amulet_nbt/__init__.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,94 @@
import re
from typing import Union

from typing import TypeAlias
from . import _version

__version__ = _version.get_versions()["version"]
__major__ = int(re.match(r"\d+", __version__)[0])
__major__ = int(re.match(r"\d+", __version__)[0]) # type: ignore

del re
del _version


def get_include() -> str:
import os

return os.path.join(__path__[0], "include")


from ._nbt import (
# Abstract classes
AbstractBaseTag,
AbstractBaseImmutableTag,
AbstractBaseMutableTag,
AbstractBaseNumericTag,
AbstractBaseIntTag,
AbstractBaseFloatTag,
AbstractBaseArrayTag,
# Tag classes
ByteTag,
ShortTag,
IntTag,
LongTag,
FloatTag,
DoubleTag,
ByteArrayTag,
StringTag,
ListTag,
CompoundTag,
IntArrayTag,
LongArrayTag,
# Tag class aliases
ByteTag as TAG_Byte,
ShortTag as TAG_Short,
IntTag as TAG_Int,
LongTag as TAG_Long,
FloatTag as TAG_Float,
DoubleTag as TAG_Double,
ByteArrayTag as TAG_Byte_Array,
StringTag as TAG_String,
ListTag as TAG_List,
CompoundTag as TAG_Compound,
IntArrayTag as TAG_Int_Array,
LongArrayTag as TAG_Long_Array,
NamedTag,
read_nbt,
read_nbt_array,
ReadOffset,
read_snbt,
StringEncoding,
mutf8_encoding,
utf8_encoding,
utf8_escape_encoding,
EncodingPreset,
java_encoding,
bedrock_encoding,
)

SNBTType: TypeAlias = str
IntType: TypeAlias = ByteTag | ShortTag | IntTag | LongTag
FloatType: TypeAlias = FloatTag | DoubleTag
NumberType: TypeAlias = ByteTag | ShortTag | IntTag | LongTag | FloatTag | DoubleTag
ArrayType: TypeAlias = ByteArrayTag | IntArrayTag | LongArrayTag
AnyNBT: TypeAlias = (
ByteTag
| ShortTag
| IntTag
| LongTag
| FloatTag
| DoubleTag
| ByteArrayTag
| StringTag
| ListTag
| CompoundTag
| IntArrayTag
| LongArrayTag
)


__all__ = [
"__version__",
"__major__",
"get_include",
"AbstractBaseTag",
"AbstractBaseImmutableTag",
"AbstractBaseMutableTag",
Expand Down Expand Up @@ -38,14 +121,10 @@
"CompoundTag",
"TAG_Compound",
"NamedTag",
"load",
"load_array",
"read_nbt",
"read_nbt_array",
"ReadOffset",
"from_snbt",
"NBTError",
"NBTLoadError",
"NBTFormatError",
"SNBTParseError",
"read_snbt",
"SNBTType",
"IntType",
"FloatType",
Expand All @@ -60,97 +139,3 @@
"java_encoding",
"bedrock_encoding",
]

from ._tag.abc import (
AbstractBaseTag,
AbstractBaseImmutableTag,
AbstractBaseMutableTag,
)
from ._tag.numeric import AbstractBaseNumericTag

# Types
from ._tag.int import (
AbstractBaseIntTag,
ByteTag,
ShortTag,
IntTag,
LongTag,
ByteTag as TAG_Byte,
ShortTag as TAG_Short,
IntTag as TAG_Int,
LongTag as TAG_Long,
)
from ._tag.float import (
AbstractBaseFloatTag,
FloatTag,
DoubleTag,
FloatTag as TAG_Float,
DoubleTag as TAG_Double,
)
from ._tag.array import (
AbstractBaseArrayTag,
ByteArrayTag,
IntArrayTag,
LongArrayTag,
ByteArrayTag as TAG_Byte_Array,
IntArrayTag as TAG_Int_Array,
LongArrayTag as TAG_Long_Array,
)
from ._tag.string import (
StringTag,
StringTag as TAG_String,
)

from ._tag.named_tag import NamedTag

from ._tag.list import (
ListTag,
ListTag as TAG_List,
)
from ._tag.compound import (
CompoundTag,
CompoundTag as TAG_Compound,
)

# Load functions
from amulet_nbt._nbt_encoding._binary import load, load_array, ReadOffset
from amulet_nbt._nbt_encoding._string import from_snbt

from ._errors import NBTError, NBTLoadError, NBTFormatError, SNBTParseError

from ._string_encoding.encoding import (
mutf8_encoding,
utf8_encoding,
utf8_escape_encoding,
StringEncoding,
)
from ._nbt_encoding._binary.encoding_preset import (
java_encoding,
bedrock_encoding,
EncodingPreset,
)

SNBTType = str

IntType = Union[ByteTag, ShortTag, IntTag, LongTag]

FloatType = Union[FloatTag, DoubleTag]

NumberType = Union[ByteTag, ShortTag, IntTag, LongTag, FloatTag, DoubleTag]

ArrayType = Union[ByteArrayTag, IntArrayTag, LongArrayTag]

AnyNBT = Union[
ByteTag,
ShortTag,
IntTag,
LongTag,
FloatTag,
DoubleTag,
ByteArrayTag,
StringTag,
ListTag,
CompoundTag,
IntArrayTag,
LongArrayTag,
]
Loading
Loading