-
Notifications
You must be signed in to change notification settings - Fork 1
Docs
This time docs for kefir. May be soon on the RTD...
Here you can find useful information about Kefir: using, philosophy, pros and cons
Kefir - new framework for (de)serialization of SQLAlchemy models and complex objects. It's easy to learn and setup.
Yes, but there are some differences in my framework (whether this is a plus or a minus - it's up to you!).
- It is ~ 3 times faster than marshmallow! See benchmarks.
- Its syntax is clearer, simplifies routine work in many ways. Sometimes you don't even need to do anything extra
- It uses OOP. More here
- There is support for Flask/FastAPI out of the box
- It is written in pure Python and has no dependencies, which is very convenient for projects of any level of complexity.
A small clarification: Kefir has dependencies for development and support of the project, but does not have dependencies in the code itself.
for installing follow this commands:
git clone https://github.com/Yourun-proger/kefir.gitThis command just copy my repo to your project folder for installing kefir. Also you can upload a archive and continue work on plan
Windows
> py -m venv env
> env\Scripts\activateLinux/MacOS
$ python3 -m venv env
$ . env/bin/activatecd kefirpip install -e .So far, there is no kefir on PyPI, so you need to install it in editable mode using the -e option, which installs my module into your virtual environment through its setup.py file.
Also i see that the name kefir is occupied in the PyPI space, so it will be set through a different name, but directly in the code you can refer to it as kefir.
Im glad what you installed my kefir
- Import
Kefirclass from kefir - Create
Kefirobject. Please give a name kef for this object! (really don’t give another name, please) - Use
dumpfunction where argument is your object.
from kefir import Kefir
kef = Kefir()
class A:
def __init__(self, arg1):
self.arg1 = arg1
class B:
def __init__(self, arg1, a):
self.arg1 = arg1
self.a = a
a = A('kefir is not bad!')
b = B('so true', a)
dumped_a = kef.dump(a) # {'arg1': 'kefir is not bad!'}
dumped_b = kef.dump(b) # {'arg1': 'so true', 'a': {'arg1': 'kefir is not bad!'}}Kefir can serialize custom object, but more often you need to serialize SQLAlchemy models. It's easy. See this example:
from my_models import User
from kefir import Kefir
kef = Kefir()
user = User.query.get(1) # User(id=42, nickname='kefir_fun', mail='kefir@fun.mars')
dumped_user = kef.dump(user)dumped_user will be like this:
{
'id': 42,
'nickname': 'kefir_fun',
'mail':'kefir@fun.mars',
'orders': [
{
'id':993,
'title' 'empty box'
},
{
'id':1234,
'title': 'another empty box'
}
]
}It works good with pure alchemy models and with flask-sqlalchemy models too
The fact that kefir can do it conveniently and quickly. But still, in most cases, it is necessary to have a dict representation that is different from the original model, its customized version. To do this, you need to create your own view using Repr (a bit like Schema).
- Declare fields to be ignored. This really may be needed, for example, to hide some data from the client.
- Change the names of the fields in the summary view.
This can be very convenient and time-saving. Imagine you have a
Noteclass that has adescattribute. You would like the field to be named `description. If not for kefir, then in this case you would have to rename the attribute itself in the class and in ALL places where it is used, which can be a long and tedious process. - Change field values. Customize them in some way, in order to make the conclusion more convenient, more accurate, etc.
- Validation. A really important aspect that helps the developer understand the problem in his data, as well as show the client that his input is incorrect.
Okay. We have a sqlalchemy model User:
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
mail = Column(String)
password_hash = Column(String)
orders = relationship('Order', back_populates='user')We work on some API. We need to make GET /users/<int:user_id> endpoint, that return an User object like this:
{
"email": "liam.bolb@bob",
"orders": [{"address": "la rue",
"date": "26.01.2021",
"price": 42,
},
{"address": "la rue number two",
"date": "01.01.2020",
"price": 42,
}]
}This is UserRepr code:
from kefir import Repr
class UserRepr(Repr):
ignore = ['password_hash']
names_map = {
'mail': 'email'
}
look = ['email']
validate = ['email']
def look_email(mail_addr):
return mail_addr[::-1]
def validate_email(mail_addr):
assert '@' in mail_addr, "invalid email address!"Yes, that's all! Let's take a closer look...
-
ignore- list of attrs names for ignoring. -
names_map- dict, where key - attrname, value - new name in representation.
When you change name in names_map, you need use new name in validate and look fields and functions!!
-
look- list of attrs names for format it on representation -
validate- list of attrs names for validation
For all in look and validate, you need to create function like this format: validate_<name> or look_<name>!
For validation you always write function with assert statement. If you want custom message about not valide result, you need to add after this statement string with your message
That's easy. Just write:
kef = Kefir(represents={User:UserRepr})Kefir does not convert objects and models to json, it converts them to dictionary. Further conversion depends on the user using regular python tools - the json module or external new libraries. It is worth noting that specifically for Flask, there is a decorator that converts a raw object or model to json, but only for a web view, since Flask does not allow the dictionary to be passed as a return object of view-function. Although it is worth noting that thanks to this patch in 1.1.0 version, this is possible and you can write like this:
@app.route(...)
def view_func(...):
return {'ping': 'pong'}and hence like this:
@app.route(...)
def view_func(...):
return kef.dump(some)Although under the hood here the flask simply uses jsonify, which in this case is similar to using json.dumps in the kefir decorator.
I really love Flask, so I made a special decorator - @dump_route, which converts the returned result to json. This can be useful for building APIs. Look at this example of using, what i copy-pasted from docstring 😏:
from models import User
kef = Kefir(...)
app = Flask(...)
@app.route('/users/<int:user_id>')
@kef.dump_route
def user_view(user_id):
return User.query.get(user_id)What we see here? Function returns a raw sqlalchemy model, but this decorator changes it and dump to json this model.
@kef.dump_route must be between @app.route and view-function for correct work!
This decorator also works for fastapi. BUT in order for everything to work correctly, you need to correctly configure the Kefir object:
from fastapi import FastAPI
from kefir import Kefir
app = FastAPI(...)
kef = Kefir(used='fastapi')
@app.get('/')
@kef.dump_route
def main():
return SomeObject(...)In short, when initializing, you need to specify used='fastapi'. You don't need to do this for flask, since used='flask' is the default
Now kefir does not support FastAPI and Flask (since 2.0)asynchronous views. I want to better understand and understand asynchrony in principle and how to work with it in Python in particular. I could mindlessly copy-paste the code i need from any source, but i don't want to end up like this.
It's a bit tricky for me... I just want to clarify that the kefir will have a mode parameter, which indicates which view functions the kefir should work with in this decorator. It does not mean that the dump and load methods will be asynchronous. However, I have gained experience and can make kefir completely asynchronous too. Just tell me your opinion on the github discussion so that I know how to prioritize.
UPD: 10.08.2022
Now everything is different. Kefir doesn't need to know this. Kefir focuses on a framework plugin that...
Today, YES. I realized that this is inconvenient and stupid. I'll create a something like DumpMiddleware for simplify work.
UPD: 28.03.22
Yeeah. Now you can use ASGIDumpMiddleware for your fasapi application.
Example:
from kefir.middlewares import ASGIDumpMiddleware
app = FastAPI(...)
app.add_middleware(ASGIDumpMiddleware)That's all
Also, using kefir, you can turn a dictionary into an object (deserialization). It's not too hard
To do this, you need to use kef.load function like this:
kef.load(some_dict, some_cls)See simple example:
from kefir import Kefir
class A:
def __init__(self, x, y):
self.x = x
self.y = y
kef = Kefir()
new_a_object = kef.load({'x':42,'y':'kefir'}, A) # A(x=42, y='kefir')In this case this is similarly with:
dct = {'x':42, 'y':'kefir'}
new_a_object = A(**dct)Or with this:
values = [42, 'kefir']
new_a_object = A(*values)This also works for alchemy models:
@app.route('/create_new/', methods=['POST'])
def create_model():
dct = request.json
model = kef.load(dct, Model) # similar with Model(**dct)
session.add(model)
session.commit()
return {"Created": "OK"}, 201Things get more complicated if there are nested dictionaries inside a given dictionary. Kefir perceives this as some description of another nested object. We can try to turn such a dictionary into an object:
class A:
def __init__(self, attr):
self.attr = attr
class B:
def __init__(self, attr, a_obj):
self.attr = attr
self.a_obj = a_obj
print(kef.load({'attr':42, 'a_obj':{'attr':1000}}, B))
>>>
Traceback (most recent call last):
File "xxx.py", line 13, in <module>
print(kef.load({'attr':4, 'a':{'attr':2}}, B))
File "x/x/x/xxx.py", line 138, in load
raise NeedReprException(
kefir.NeedReprException:
This object with the nested data.
Add Repr for `B` class!
In Repr, `a` field must be added to `loads` dictKefir threw an exception! What we need to do?
Actually what is written in the exception)
Let's clarify.
I thought for a long time about implementing this in kefir, and what is described below happened. I had a lot of thoughts about this, i wanted to make my implementation smart, i had another idea: check globals() or kef.represents and look for classes with a name similar to the name, otherwise raise an exception. I don’t know, I thought this was a bad idea and the explicit designation in loads seemed to me the best option. What we have now is perhaps not bad, but i'll try to find a better way, or i'll leave it as it is.
If you have not read the section about Reprs, read it for understanding.
To enable the kefir to correctly transform nested dictionaries, you need to create a Repr class, in which it is necessary to create a field loads - a dictionary, where the key is the name of the field in the class, and the value is the class itself, into which you need to convert the nested dictionary.
The updated code will look like this:
class A:
def __init__(self, attr):
self.attr = attr
class B:
def __init__(self, attr, a_obj):
self.attr = attr
self.a_obj = a_obj
+ class BRepr(Repr):
+ loads = {'a_obj': A}
+
print(kef.load({'attr':42, 'a_obj':{'attr':1000}}, B)) # B(42, A(1000))It's simple, but you need to make a few clarifications.
- Exactly the same behavior with
dumpmethod, whennames_mapis declared. This is only important if you are using sqlalchemy models where correct naming is important. - Validation occurs before the final creation of the object. If any of the passed values is invalid, a
DeserializationExceptionexception is thrown.
I don't know who really needs it. I needed it here (which was otherwise optional), which may not make sense, but I'm experimenting I don't care!!1
new = kef.load({'my_dict':{'x':3, 'y':4}, 'arg': 42}, SomeCLS, allow_dict=True)Yes, you can. Just type something like this:
some = kef.load('your_file.json', SomeCLS)Do not forget to specify the .json extension, otherwise the kefir will get angry:
wrong = kef.load('your_file', SomeCLS)
>>>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "x/x/x/xxx.py", line 137, in load
raise ValueError(
ValueError:
If you want to feed me a json file,
please change/add the .json extension.And do not forget to create your json file:
not_found = kef.load('not_to_be.json', SomeCLS)
>>>
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "x/x/x/xxx.py", line 132, in load
raise ValueError(
ValueError:
Where is json file?!
I can't found it here: "3.json"I don’t know if anyone needs it, but let it be a killer-feature)
This is a very common type for working with date and time, but its (de)serialization is a separate item for documentation, since there are fundamental differences from the other
During serialization, Kefir automatically formats the datetime.datetime object in accordance with the variable in the datetime_format of the Repr class, or in the absence of Repr in the class, the same variable of the Kefir class object. In both cases, the default is % d.% M.% Y
That is:
kefir at the beginning tries to find repr.datetime_format, otherwise it gives the value kef.datetime_format
For example:
class A:
def __init__(self, x, date):
self.x = x
self.date = date
a = A(42, datetime.datetime.now())
print(kef.dump(a))
>>> {'x': 42, 'date': '20.11.2011'}Or:
class B:
def __init__(self, arg, date):
self.arg = arg
self.date = date
class BRepr(Repr):
names_map = {'date': 'year'}
loads = {'year': datetime.datetime}
datetime_format = '%Y'
b = B('string', datetime.datetime.today())
print(kef.dump(b))
>>> {'arg': 'string', 'year': 2021}It is worth noting that here we essentially replaced the look function for the "date" field, which looks simpler
It's no more complicated than deserializing complex objects. In the Repr class in the dictionary loads, you need to create a key-value pair, where the key is the name of the field that needs to be converted to datetime.datetime, and the value is, in fact, the datetime.datetime class. Since Repr is used, its value datetime_format is also used.
Demonstration:
class A:
def __init__(self, i_have_only_date_for_demo):
self.date = i_have_only_date_for_demo
class ARepr(Repr):
loads = {'date': datetime.datetime}
print(kef.load({'date':'21.12.2112'}, A))
>>> A(datetime.datetime(2112, 12, 21, 0, 0))Not bad, right?
This section of the documentation will provide a little note for users of kefir and for its creator, because it's all just a rubber duck method?
- Don't panic about
from kefir import *I registered the variable__all__, where only what you need is indicated: the Kefir and Repr classes, so the entire kefir namespace, along with everything you don’t need, will not be imported. - How it works?
Yes, it's actually pretty easy. It's so simple that I encourage you to make your own kefir - better and stronger. Well, or send PR;)
When kefir is created, nothing in it is initialized. Its
__new__method just twitches, inside which the kefir factory is launched. If the mode parameter == "sync", then the factory will make synchronous kefir according to the specified parameters. If async, well, guess yourself). Right now, make an instance of the Kefir class and check its type. You will not seekefir.Kefir, you will seekefir.SyncKefir(and in the future you will see AsyncKefir). So, well, the object itself seems to have been sorted out. To be continued.... - I don’t know, maybe i shouldn’t write about it in such detail, everything seems to be clear from the code ... But okay, let it be
- Is kefir a gun for ants?
Well, it didn't. Well, as a suggestion, you can create a
to_dictmethod inside your class that describes a model or object, where you can write the serialization code in a format convenient for you. Perhaps you just needobject.__dict__orvars(object). Frankly speaking, maybe kefir is a bad solution for projects? Will the developers using it write bad code or make a bad project structure to insert kefir? I do not know. I have no commercial experience in software development, but I just study development on my own for 2-3 years, which is not enough. Maybe the project turned out to be immature, because I myself am still green? Maybe. But I've gone too far. The project will develop and mature, just like me in the IT world. I hope I don't wake up with the feeling that the project is "stillborn"...
Now this topic is quite popular and widely discussed. Most projects use them. I'm not going to do this because:
- Silly mistakes with type annotations that hinder the development of the project. Sometimes it can be very, very difficult to fix.
- Type-hints are needed to make it easier for any programmer to understand what's going on. There are no crazy abstractions and complex code in the code of my project. Sounds presumptuous, but look at the code for yourself. No annotations are needed here.
Conclusion, i have nothing against PEP 484 and its ideas in principle, they have a right to exist. But not here)
This project uses pure Python and no more. There is no sqlalchemy in the dependencies, since you may not need it. If you dump alchemy models, it is assumed that it is already installed. An exception is thrown if you forgot to install Flask, but you are using @dump_route decorator.
Why not just make an only dump function in the module, for example? It's easier! Initially (as you can see from the commits) I thought so too.
Now i understand that it will be inconvenient. In the future, kefir will have several specials. parameters to intelligently configure it for something specific or to simplify some routine things. It is worth adding that you may need to have several differently customized Kefir objects. Also, inside the class, i can more conveniently hide various small operations using private methods. Finally, it’s more convenient to structure all my work and expand the capabilities of kefir when i work on Kefir class.
Why? Well, simply because I like this ORM more than others with its architecture and philosophy. It is used by most Python programmers. Well, as a last resort, I will come up with some kind of ORM_Adapter class so that you can safely apply (de)serialization in kefir to a specific ORM.
I want to talk a little about what I don't like about kefir itself.
-
All in one file. It was convenient until it began to take 200+ lines of code. Everything that interferes is poked here and it is better to create a package divided into logical parts (reprs, exceptions, main, etc.).Okay now, i fixed it. - A lot of similar code. Just insanely many repetitions! This violates the DRY and even if you don't know about it, the code looks bad :/
- Low level of readability. This does not contradict this (it is possible to read the code in principle, but it takes a little work on it) and not a duplicate of 2. Code just needs refactoring with more understandable names of variables and functions. It is also worth taking out some code in a function to improve readability. Also
pre-commithooks can be added in a future. - Still need to make (de)serialization more flexible for
datetime.datetimeobjects. UPD: Now i added basic support for datetime objects, but not for SQLAlchemy models. -
extraRepr field. I have removed it for now. I need to better define what it will be and how it will be implemented.
Have a small note about kefir problems or a bug in the behavior or code of kefir? Write in the discussion of kefir! More serious notes are best sent to issues. Don't be shy, kefir needs feedback!
I tried very hard while writing this, but I find the documentation very illogical and difficult to understand. I hope you leave your feedback on the documentation here to help me improve it or even rewrite some of its fragments. Also, in the near future i want to write a separate tutorial on kefir, in which I will touch upon all aspects of its use (01/03/2022) When i do it, i will attach a link. Soon!