Support the SQLAlchemy 2.0 DeclarativeBase models #1215

pamelafox · 2023-06-01T20:38:27Z

This PR adds support to Flask-SQLAlchemy for using the new SQLAlchemy 2.0 style of subclassing DeclarativeBase or DeclarativeBaseNoMeta (and optionally MappedAsDataclass).

Here's a simple usage example:

class Base(sa.orm.DeclarativeBase):
    pass

db = SQLAlchemy(app, model_class=Base)

class Quiz(db.Model):
    __bind_key__ = "other"
    id_column: Mapped[int] = mapped_column(primary_key=True, autoincrement=True)
    title: Mapped[str] = mapped_column(db.String(255), nullable=False)

The tests check for many combinations of subclasses of base, see conftest.py for the parameterized fixture.

This approach has the advantage of letting developers follow the SQLAlchemy 2.0 docs for defining models while still being able to benefit from the Bind and Name mixin functionality, as well as all the standard functionality offered by Flask-SQLAlchemy.

There's one way that this approach is less flexible than the 1.x integration: it isn't possible to opt out of the Bind or Name mixin. Opting out of bind doesn't seem necessary since it doesn't really affect code that doesn't use it. If opting out of name is a big need, then I think the simplest way to support that is to add a disable_autonaming parameter to SQLAlchemy.

Other potential drawbacks of this PR:

The documentation now has some "forks" for 1.x vs 2.x SQLAlchemy APIs. I imagine not everyone is ready to move to 2.x API so Flask-SA may want to support both for a while.
Similarly, there is some code redundancy in model.py, since I copied the mixins to make them into base classes using init_subclass (versus metaclasses). I could probably refactor redundant code into helper functions if you felt it was necessary.
The typing situation got a bit hairy, since there are so many types of classes accepted for model_class. My approach resulted in adding some ignores but removing some previous ignores, so I think it's net neutral.

Checklist:

Add tests that demonstrate the correct behavior of the change. Tests should fail without the change.
Add or update relevant docs, in the docs folder and in code.
Add an entry in CHANGES.rst summarizing the change and linking to the issue.
Add .. versionchanged:: entries in any relevant code docs.
Run pre-commit hooks and fix any issues.
Run pytest and tox, no tests failed.

pamelafox · 2023-06-08T13:44:01Z

@davidism Ah, tests are failing due to a SQLAlchemy bug that they have since fixed in main, but is not yet released. I manually made the fix locally.

sqlalchemy/sqlalchemy#9862

The fix is merged but there hasn't been a release yet. I'll watch for the release.

pamelafox · 2023-06-13T18:10:03Z

Update: SQLAlchemy released the fix in 2.0.16, so this branch now requires that version. I don't know if it's okay that we require such a recent version of SQLAlchemy? The old Flask-SQLAlchemy approach still works with older SQLAlchemy versions, but the parameterized tests all check for the new SQLAlchemy 2.0 way.

All checks are now passing on this branch.

I also changed this PR to compare against main, instead of 3.0.x, as I re-read the contributing guidelines which says that features should be against main, and this feels like a feature.

jbhanks · 2023-06-16T01:46:20Z

Awesome, can't wait for it to be in the official release. I've been using your branch without issue for the past week. Mapped dataclasses are so much nicer to work with!

…

On Tue, Jun 13, 2023 at 2:10 PM Pamela Fox ***@***.***> wrote: Update: SQLAlchemy released the fix in 2.0.16, so this branch now requires that version. I don't know if it's okay that we require such a recent version of SQLAlchemy? The old Flask-SQLAlchemy approach still works with older SQLAlchemy versions, but the parameterized tests all check for the new SQLAlchemy 2.0 way. All checks are now passing on this branch. I also changed this PR to compare against main, instead of 3.0.x, as I re-read the contributing guidelines which says that features should be against main, and this feels like a feature. — Reply to this email directly, view it on GitHub <#1215 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADCXX3XTDZH6GXPMAVXOBULXLCUINANCNFSM6AAAAAAYXOGIDY> . You are receiving this because you commented.Message ID: ***@***.***>

tonybaloney · 2023-06-21T21:32:36Z

docs/quickstart.rst

+    db = SQLAlchemy()
+
+
+By default, this extension assumes that you are using the SQLAlchemy 1.x API for defining models.


maybe reword to "this extension supports both SQLAlchemy 1 and 2 APIs. By default models are defined using the SQLAlchemy 1 API"

I've reworded to use the first sentence, but I dislike the passive voice in the second sentence, since that makes it unclear who exactly is doing the model defining.

tonybaloney · 2023-06-21T21:32:44Z

docs/quickstart.rst

+
+By default, this extension assumes that you are using the SQLAlchemy 1.x API for defining models.
+
+To use the new SQLAlchemy 2.x API, pass a subclass of either ``DeclarativeBase`` or ``DeclarativeBaseNoMeta``


Provide a link to the DeclarativeBase and DeclarativeBaseNoMeta documentation

Good idea, doing.

tonybaloney · 2023-06-22T00:21:33Z

docs/quickstart.rst

+    from sqlalchemy.orm import DeclarativeBase
+
+    class Base(DeclarativeBase):
+      pass


I think it needs to be clearer what adding a Base class does here instead of just doing SQLAlchemy(model_class=DeclarativeBase). Maybe put an example attribute in or a comment that your user defined attributes should go there.

Hm, my earlier version of this feature had you call SQLAlchemy(model_class=DeclarativeBase) but I wanted to support someone also subclassing MappedAsDataclass, so that's why I now ask developers to create the Base class. They can then choose whether to subclass DeclarativeBase, DeclarativeBaseNoMeta, MappedAsDataclass, etc.

However, I wouldn't typically expect them to put user-defined attributes in Base - those go in the table-specific models.

I'm debating what to write here that doesn't become an essay / repeat SQLAlchemy docs too much.

I've added more detail and an additional example, let me know what you think of the change.

tonybaloney · 2023-06-22T00:23:14Z

pyproject.toml

@@ -16,7 +16,7 @@ classifiers = [
 requires-python = ">=3.7"
 dependencies = [
    "flask>=2.2.5",
-    "sqlalchemy>=1.4.18",
+    "sqlalchemy>=2.0.16",


Do we need to set a max-version as well? Just incase SQL Alchemy 2.1+ introduces breaking changes.

Also this would mean that this extension is no longer compatible with SQL Alchemy 1. Users will have to install an older version (which is fine, but worth noting in the release notes)

That's reasonable, I just got bit indirectly by Pillow 10.0 change in another repo, so it seems like the responsible thing for a package is to explicitly check compatibility with new major versions.

I've just changed it to:
"sqlalchemy>=2.0.16,<2.1",

Added note to the changelog

Libraries must never set upper bounds on dependencies, it makes it very difficult for installers to resolve dependency trees. We don't have any reason to expect SQLAlchemy 2.1 to break what we do, or to do so in a way that we won't address, so the upper bound would not be correct. See https://iscinumpy.dev/post/bound-version-constraints/

Ah okay, I've removed the version cap, still in process of reading the post, thanks for letting me know.

Update: Going to revert this, and make the tests run in both SQLAlchemy 1.4 and SQLAlchemy 2 (see my note in conftest.py)

"this" includes the >=2 requirement I assume? (i think it'll be a LONG time until everyone upgraded to SA2, for small apps it might be easy but for larger one, in particular if there's no perfect test coverage, it's not likely to happen quickly, so keeping support for SA 1.4 until upstream discontinues it is much better IMHO)

edit: ah i see you already removed the upper bound, so i guess this is indeed about the >=2

Update: I started along the path of removing the upper bound of 2, but realized that would require a lot more checks inside our codebase, since we wouldn't even be able to import the new classes without a version check.

After conferring with @davidism, we decided to stick with the upper bound of >=2.

There seems to be good adoption of SA2, and this version still supports all the 1.4 syntax, so it shouldn't break 1.4 code (if all goes as planned).

We wouldn't remove 1.4 support entirely until a major bump of this package, and we'd put deprecation warnings in before that happens.

SA version download charts: https://www.pepy.tech/projects/sqlalchemy?versions=2.*&versions=1.*

tonybaloney · 2023-06-22T00:56:05Z

src/flask_sqlalchemy/extension.py

 import typing as t
 from weakref import WeakKeyDictionary

 import sqlalchemy as sa
-import sqlalchemy.event
 import sqlalchemy.exc


sqlalchemy.exc is being imported but never used, however sa.exc is used. I think the import statement can be dropped. sa.exc is an attribute of the module import that's added at runtime.

tonybaloney · 2023-06-22T01:02:05Z

docs/quickstart.rst

@@ -90,6 +120,20 @@ The model will generate a table name by converting the ``CamelCase`` class name

 The table name ``"user"`` will automatically be assigned to the model's table.

+It's also possible to use the SQLAlchemy 2.x style of defining models,
+as long as you initialized the extension with an appropriate 2.x model base class
+as described above.


"above" should link to the section you mean

Rydberg95 · 2023-06-28T15:01:05Z

Just wanted to jump in and also say that I've been using this branch for the last couple of days, and it's running great! Thanks a bunch @pamelafox!

ThiefMaster · 2023-07-05T20:31:39Z

I really want to keep a way to opt out from auto-naming. If you use explicit names everywhere, getting auto-naming is counterproductive (it hides mistakes such as someone forgetting to put an explicit name).

pamelafox · 2023-07-05T20:47:02Z

@ThiefMaster I see, would a disable_autonaming parameter work for you, or do you have a different idea for how to disable it?

ThiefMaster · 2023-07-05T20:47:55Z

yeah, seems like a straightforward option to me

davidism · 2023-07-05T22:55:15Z

Since this now only uses __init_subclass__, bind_key and tablename could be class parameters instead of class attributes. I played around with this before and it made things nicer overall, but I couldn't do it when we still needed metaclass.__new__ for some parts.

pamelafox · 2023-07-11T21:20:13Z

@ThiefMaster I've added disable_autonaming as a SQLAlchemy constructor parameter, and supported it for both old and new SQLAlchemy users. Take a look at the test cases and let me know if that looks good.

pamelafox · 2023-07-11T21:23:22Z

@davidism To clarify, when you mean class parameter, do you mean like...

class User(db.Model, bind_key="auth"):
    id = db.Column(db.Integer, primary_key=True)

Or some other syntax? Searching the web for "class parameters" yields a few different interpretations.

If we introduced that syntax, then we could only do it for SQLAlchemy 2.x users, which would further fork the documentation/examples. I'm okay with that if you think it's worth it, I've just been trying to make the minimal change API-wise. (Admittedly there are already some forks in the docs in this PR).

pamelafox · 2023-07-30T19:12:38Z

docs/api.rst

+
+.. autoclass:: BindMixin
+
+.. autoclass:: NameMixin


Maybe we dont put this in the reference?

pamelafox · 2023-07-30T19:14:06Z

src/flask_sqlalchemy/model.py

+    ignored. If the ``metadata`` is the same as the parent model, it will not be set
+    directly on the child model.
+
+    .. versionchanged:: 3.0.4


Update this version number

pamelafox · 2023-07-30T19:17:38Z

src/flask_sqlalchemy/model.py

+    that do not otherwise define ``__tablename__``. If a model does not define a primary
+    key, it will not generate a name or ``__table__``, for single-table inheritance.
+
+    .. versionchanged:: 3.0.4


Change the version number here

pamelafox · 2023-07-30T19:22:36Z

docs/quickstart.rst

+
+.. _sqlalchemy1-initialization:
+
+Using the SQLAlchemy 1 API


Defer all the SQLAlchemy 1 stuff into its own page, with a note at the top.

pamelafox · 2023-07-30T19:56:22Z

tests/test_view_query.py

    with pytest.raises(NotFound):
-        assert db.get_or_404(Quiz, 2)
+        assert db.get_or_404(Todo, 2)


Note: click and flask have test_typing examples for mypy/pyright

src/flask_sqlalchemy/model.py

pamelafox · 2023-07-30T20:15:49Z

tests/conftest.py

-def db(app: Flask) -> SQLAlchemy:
-    return SQLAlchemy(app)
+test_classes = [
+    None,


Add a comment: None means 1.0 will be used

tests/conftest.py

… 2.x

pamelafox · 2023-08-22T23:17:58Z

I've made a number of changes based off feedback and on running additional tests. Specifically:

I went through every test and parameterized them in some way to run against both the 1.x and 2.x base classes. That revealed a few issues with metadata/bind key.
In SQLAlchemy 2.0, they create a metadata automatically when the user creates a Base class subclassing either DeclarativeBase/DeclarativeBaseNoMeta. That means that most of the time, our extension doesn't need to make metadata, and can just reuse that metadata. I made a slight change to make_declarative_base and BindMixin to reflect this. Related tests pass.
In SQLAlchemy 2.0, a user can override the default metadata via a class attribute on their Base class. Given that, it seems redundant/confusing for our extension to also accept a metadata parameter in the constructor. So I added a deprecation warning for anyone who passes in both a 2.x base class and metadata, with a note they should just put that metadata in the base class. I added a test for that and versionchanged note.
Per suggestion of @davidism, I reworked the documentation to make 2.x the default style used, and put 1.x instructions in a legacy-quickstart.md guide

So there are now quite a few more files changed than before: tests for parameterization changes, and docs for 1.x/2.x moves.

pamelafox · 2023-08-23T02:20:55Z

I've discovered a situation in customizing.rst that is not compatible with my current approach, so I will need to revise it and add a test for that.

pamelafox · 2023-08-25T23:34:37Z

I've now addressed that issue and added additional tests based on the examples in customizing.rst. The make_declarative_base function will now base the body of the new class on the dict of the passed in model_class. That allows it to keep class attributes like declared_attr.

pamelafox · 2023-08-25T23:35:48Z

This is ready for re-review.

pamelafox mentioned this pull request Jun 5, 2023

Support SQLAlchemy 2's DeclarativeBase and MappedAsDataclass #1140

Closed

pallets-eco deleted a comment from jbhanks Jun 8, 2023

pamelafox changed the base branch from 3.0.x to main June 13, 2023 13:43

pamelafox requested a review from davidism June 14, 2023 18:13

tonybaloney reviewed Jun 22, 2023

View reviewed changes

pallets-eco deleted a comment from Dnncha Jun 30, 2023

pamelafox added 11 commits July 2, 2023 14:22

Possible way to mixin with DeclarativeBase

0150152

Remove pdm-python file

7d0f5e5

Try different approach to mixins for 2.0

d073fa6

Support MappedAsDataClass

5ff15bc

Update should check

8d78e37

Parameterizing all the tests

41fdd5d

Remove redundant test

31e8e2c

Updates to docs

b536d28

Getting types to work (mostly)

af5096d

Update sqlalchemy version

a8da619

Mark the type classes as internal with an underscore in front

d3a30c5

pamelafox force-pushed the mixin-three branch from de9b105 to d3a30c5 Compare July 4, 2023 00:21

Fix tests

73e0206

pamelafox added 2 commits July 5, 2023 13:49

Addressing docs note

2c3650a

Add constraint of les than 2.1

7610ea7

Remove version cap per Davids comment

8fdc47a

pamelafox commented Jul 30, 2023

View reviewed changes

docs/api.rst Outdated

.. autoclass:: BindMixin

.. autoclass:: NameMixin

Copy link

Contributor Author

pamelafox Jul 30, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we dont put this in the reference?

pamelafox commented Jul 30, 2023

View reviewed changes

src/flask_sqlalchemy/model.py Show resolved Hide resolved

pamelafox commented Jul 30, 2023

View reviewed changes

tests/conftest.py Show resolved Hide resolved

pamelafox added 7 commits August 20, 2023 14:30

Update versionchanged for new mixins

b052dd6

Document refactor to emphasize 2.x, fix for bind key and metadata for…

d8c31ca

… 2.x

Parameterize more tests for commit

a89485b

Parameterize more tests to use 2.x

6f4c180

Update README

caee46a

Style and sphinx checks

6d7a7e3

Documentation updates- remove new mixins from ref, add some comments

1d877e8

Update abstract models and mixins

a2db216

Add test for declared_attr

cf4170a

pamelafox requested a review from tonybaloney August 25, 2023 23:35

pamelafox added 2 commits August 26, 2023 06:24

Make suggested typing fix

fe9e4de

Adjusted to work in 3.8

fdeec1d

davidism approved these changes Aug 31, 2023

View reviewed changes

davidism merged commit 9f3bf94 into pallets-eco:main Aug 31, 2023
7 checks passed

TaiSakuma mentioned this pull request Aug 31, 2023

Upgrade SQLAlchemy to version 2 simonsobs/acondbs#41

Closed

github-actions bot locked as resolved and limited conversation to collaborators Sep 15, 2023

		db = SQLAlchemy()


		By default, this extension assumes that you are using the SQLAlchemy 1.x API for defining models.


		By default, this extension assumes that you are using the SQLAlchemy 1.x API for defining models.

		To use the new SQLAlchemy 2.x API, pass a subclass of either ``DeclarativeBase`` or ``DeclarativeBaseNoMeta``

Support the SQLAlchemy 2.0 DeclarativeBase models #1215

Support the SQLAlchemy 2.0 DeclarativeBase models #1215

Conversation

pamelafox commented Jun 1, 2023

pamelafox commented Jun 8, 2023

pamelafox commented Jun 13, 2023

jbhanks commented Jun 16, 2023 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pamelafox Jul 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidism Jul 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ThiefMaster Jul 30, 2023 • edited Loading

Choose a reason for hiding this comment

pamelafox Aug 22, 2023 • edited by ThiefMaster Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Rydberg95 commented Jun 28, 2023

ThiefMaster commented Jul 5, 2023

pamelafox commented Jul 5, 2023

ThiefMaster commented Jul 5, 2023

davidism commented Jul 5, 2023

pamelafox commented Jul 11, 2023

pamelafox commented Jul 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pamelafox commented Aug 22, 2023

pamelafox commented Aug 23, 2023

pamelafox commented Aug 25, 2023

pamelafox commented Aug 25, 2023

pamelafox Jul 5, 2023 •

edited

Loading

davidism Jul 5, 2023 •

edited

Loading

ThiefMaster Jul 30, 2023 •

edited

Loading

pamelafox Aug 22, 2023 •

edited by ThiefMaster

Loading