Sometimes having to pip install after doing Poetry Add

[TerminatedProcess]

I did a search for a similar post before I posted this

I just started work on a Django project using wsl2, ubuntu, poetry, pyenv, python 3.10.6. Anytime I tried to do something with the project (check it, migrate it to create tables, etc), I would get a message that module django isn't found. I have used poetry for a while now, so I'm fairly familiar with it. I figured out that I could do a poetry add, and then follow up with a python -m pip install but that really bothers me. Poetry is suppose to be doing that for me and I worry that this will cause down the road issues with deployment. I ran into this issue with several packages including django, pymysql, and a few others. I'm running the latest My poetry env looks like..

Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

Virtualenv
Python: 3.10.6
Implementation: CPython
Path: /home/mryan/python/django/Surge/.venv
Executable: /home/mryan/python/django/Surge/.venv/bin/python
Valid: True

System
Platform: linux
OS: posix
Python: 3.10.6
Path: /home/mryan/.pyenv/versions/3.10.6
Executable: /home/mryan/.pyenv/versions/3.10.6/bin/python3.10

I would love to resolve this. Maybe I'm doing something wrong or there's just a limitation of poetry that I am not aware of. Any thoughts or advice?

[clintonroy]

Please give..all the details, your pyproject.toml, the command you run, and the error you receive.

[TerminatedProcess]

##############################################

Not in my virtual environment

	~/python/django/Surge $pip freeze
	asgiref==3.5.2
	certifi==2022.9.24
	cffi==1.15.1
	cryptography==38.0.1
	distlib==0.3.6
	filelock==3.8.0
	pipenv==2022.10.4
	platformdirs==2.5.2
	pycparser==2.21
	pydantic==1.10.2
	PyMySQL==1.0.2
	pyxtension==1.13.16
	setworkdir @ file:///home/mryan/python/setworkdir/dist/setworkdir-0.1.0-py3-none-any.whl
	sqlparse==0.4.3
	tblib==1.7.0
	tqdm==4.64.1
	typing_extensions==4.3.0
	virtualenv==20.16.5
	virtualenv-clone==0.5.7

##############################################

ENTERING virtual env with poetry

	~/python/django/Surge $poetry shell
	Spawning shell within /home/mryan/python/django/Surge/.venv
	. /home/mryan/python/django/Surge/.venv/bin/activate
	Virtual environment already activated: /home/mryan/python/django/Surge/.venv
	~/python/django/Surge $. /home/mryan/python/django/Surge/.venv/bin/activate
	(surge-py3.10) ~/python/django/Surge $pyenv local
	3.10.6
	(surge-py3.10) ~/python/django/Surge $pyenv global
	3.10.6
	(surge-py3.10) ~/python/django/Surge $which python
	/home/mryan/python/django/Surge/.venv/bin/python
	(surge-py3.10) ~/python/django/Surge $alias python
	alias python='pyenv exec python3'

##############################################

PYPROJECT.TOML

	(surge-py3.10) ~/python/django/Surge $cat pyproject.toml
	[tool.poetry]
	name = "surge"
	version = "0.1.0"
	description = ""
	authors = ["Mark Ryan <mryan.dev@outlook.com>"]
	readme = "README.md"

	[tool.poetry.dependencies]
	python = "^3.10"
	cryptography = "^38.0.1"


	[build-system]
	requires = ["poetry-core"]
	build-backend = "poetry.core.masonry.api"


```##############################################
# ADDING PACKAGE DJANGO

(surge-py3.10) ~/python/django/Surge $poetry add django
Using version ^4.1.2 for Django

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing asgiref (3.5.2)
  • Installing sqlparse (0.4.3)
  • Installing django (4.1.2)

# ADDING PACKAGE PYMYSQL

(surge-py3.10) ~/python/django/Surge $poetry add pymysql
Using version ^1.0.2 for PyMySQL

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing pymysql (1.0.2)

# ADDING PACKAGE django-model-utils

(surge-py3.10) ~/python/django/Surge $poetry add django-model-utils
Using version ^4.2.0 for django-model-utils

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing django-model-utils (4.2.0)

# ADDING PACKAGE django-rest-framework

(surge-py3.10) ~/python/django/Surge $poetry add django-rest-framework
Using version ^0.1.0 for django-rest-framework

Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Package operations: 3 installs, 0 updates, 0 removals

  • Installing pytz (2022.4)
  • Installing djangorestframework (3.14.0)
  • Installing django-rest-framework (0.1.0)

# POETRY SHOW and PIP FREEZE - LOOKS GOOD!

(surge-py3.10) ~/python/django/Surge $poetry show
asgiref               3.5.2  ASGI specs, helper code, and adapters
cffi                  1.15.1 Foreign Function Interface for Python calling C code.
cryptography          38.0.1 cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Django                4.1.2  A high-level Python web framework that encourages rapid development and clean, pragmatic design.
django-model-utils    4.2.0  Django model mixins and utilities
django-rest-framework 0.1.0  alias.
djangorestframework   3.14.0 Web APIs for Django, made easy.
pycparser             2.21   C parser in Python
PyMySQL               1.0.2  Pure Python MySQL Driver
pytz                  2022.4 World timezone definitions, modern and historical
sqlparse              0.4.3  A non-validating SQL parser.
(surge-py3.10) ~/python/django/Surge $pip freeze
asgiref==3.5.2
cffi==1.15.1
cryptography==38.0.1
Django==4.1.2
django-model-utils==4.2.0
django-rest-framework==0.1.0
djangorestframework==3.14.0
pycparser==2.21
PyMySQL==1.0.2
pytz==2022.4
sqlparse==0.4.3
(surge-py3.10) ~/python/django/Surge $

##############################################
# POETRY SHOW and PIP FREEZE - LOOKS GOOD!
# Checking DJANGO

(surge-py3.10) ~/python/django/Surge $python manage.py check --deploy
Traceback (most recent call last):
  File "/home/mryan/python/django/Surge/manage.py", line 10, in main
	from django.core.management import execute_from_command_line
ModuleNotFoundError: No module named 'django'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/mryan/python/django/Surge/manage.py", line 21, in <module>
	main()
  File "/home/mryan/python/django/Surge/manage.py", line 12, in main
	raise ImportError(
ImportError: Couldn't import Django. Are you sure it's installed and available on your PYTHONPATH environment variable? Did you forget to activate a virtual environment?
(surge-py3.10) ~/python/django/Surge $

# PIP INSTALL DJANGO

(surge-py3.10) ~/python/django/Surge $python -m pip install django
Collecting django
  Using cached Django-4.1.2-py3-none-any.whl (8.1 MB)
Requirement already satisfied: asgiref<4,>=3.5.2 in /home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from django) (3.5.2)
Requirement already satisfied: sqlparse>=0.2.2 in /home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from django) (0.4.3)
Installing collected packages: django
Successfully installed django-4.1.2

# DJANGO CHECK - DJANGO is now installed.

(surge-py3.10) ~/python/django/Surge $python manage.py check --deploy
Traceback (most recent call last):
  File "/home/mryan/python/django/Surge/manage.py", line 21, in <module>
	main()
  File "/home/mryan/python/django/Surge/manage.py", line 17, in main
	execute_from_command_line(sys.argv)
  File "/home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages/django/core/management/__init__.py", line 446, in execute_from_command_line
	utility.execute()
  File "/home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages/django/core/management/__init__.py", line 420, in execute
	django.setup()
  File "/home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages/django/__init__.py", line 24, in setup
	apps.populate(settings.INSTALLED_APPS)
  File "/home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages/django/apps/registry.py", line 116, in populate
	app_config.import_models()
  File "/home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages/django/apps/config.py", line 269, in import_models
	self.models_module = import_module(models_module_name)
  File "/home/mryan/.pyenv/versions/3.10.6/lib/python3.10/importlib/__init__.py", line 126, in import_module
	return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/home/mryan/python/django/Surge/data/models.py", line 3, in <module>
	from model_utils.fields import (
ModuleNotFoundError: No module named 'model_utils'
(surge-py3.10) ~/python/django/Surge $

# POETRY SHOW

(surge-py3.10) ~/python/django/Surge $poetry show
asgiref               3.5.2  ASGI specs, helper code, and adapters
cffi                  1.15.1 Foreign Function Interface for Python calling C code.
cryptography          38.0.1 cryptography is a package which provides cryptographic recipes and primitives to Python developers.
Django                4.1.2  A high-level Python web framework that encourages rapid development and clean, pragmatic design.
django-model-utils    4.2.0  Django model mixins and utilities
django-rest-framework 0.1.0  alias.
djangorestframework   3.14.0 Web APIs for Django, made easy.
pycparser             2.21   C parser in Python
PyMySQL               1.0.2  Pure Python MySQL Driver
pytz                  2022.4 World timezone definitions, modern and historical
sqlparse              0.4.3  A non-validating SQL parser.

##############################################
# PIP INSTALL DJANGO-MODEL-UTILS

(surge-py3.10) ~/python/django/Surge $python -m pip install django-model-utils
Collecting django-model-utils
  Using cached django_model_utils-4.2.0-py3-none-any.whl (33 kB)
Requirement already satisfied: Django>=2.0.1 in /home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from django-model-utils) (4.1.2)
Requirement already satisfied: asgiref<4,>=3.5.2 in /home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from Django>=2.0.1->django-model-utils) (3.5.2)
Requirement already satisfied: sqlparse>=0.2.2 in /home/mryan/.pyenv/versions/3.10.6/lib/python3.10/site-packages (from Django>=2.0.1->django-model-utils) (0.4.3)
Installing collected packages: django-model-utils
Successfully installed django-model-utils-4.2.0

##############################################
# DJANGO CHECK - NOW OK

(surge-py3.10) ~/python/django/Surge $python manage.py check --deploy
System check identified some issues:

WARNINGS:
?: (security.W004) You have not set a value for the SECURE_HSTS_SECONDS setting. If your entire site is served only over SSL, you may want to consider setting a value and enabling HTTP Strict Transport Security. Be sure to read the documentation first; enabling HSTS carelessly can cause serious, irreversible problems.
?: (security.W008) Your SECURE_SSL_REDIRECT setting is not set to True. Unless your site should be available over both SSL and non-SSL connections, you may want to either set this setting True or configure a load balancer or reverse-proxy server to redirect all connections to HTTPS.
?: (security.W009) Your SECRET_KEY has less than 50 characters, less than 5 unique characters, or it's prefixed with 'django-insecure-' indicating that it was generated automatically by Django. Please generate a long and random value, otherwise many of Django's security-critical features will be vulnerable to attack.
?: (security.W012) SESSION_COOKIE_SECURE is not set to True. Using a secure-only session cookie makes it more difficult for network traffic sniffers to hijack user sessions.
?: (security.W016) You have 'django.middleware.csrf.CsrfViewMiddleware' in your MIDDLEWARE, but you have not set CSRF_COOKIE_SECURE to True. Using a secure-only CSRF cookie makes it more difficult for network traffic sniffers to steal the CSRF token.
?: (security.W018) You should not have DEBUG set to True in deployment.

System check identified 6 issues (0 silenced).

[TerminatedProcess]

Hope it's readable!

[clintonroy]

So...

I don 't know about pyenv at all.

pip installing inside your poetry environment is definitely..not the right way to do things, as you've already stated.

Could you show the first few lines of manage.py, I'm not familiar with that. Is it just a standard python script, or is it doing funky things to sys.path?

From a completely fresh .venv, could you:
poetry install
poetry run python -c "import django ; print(django.file)"

[TerminatedProcess]

NOTE: made sure I wasn't in the virtual environment and deleted directory .venv

~/python/django/Surge $poetry install

Creating virtualenv surge in /home/mryan/python/django/Surge/.venv
Installing dependencies from lock file

Package operations: 7 installs, 0 updates, 0 removals

  • Installing pycparser (2.21)
  • Installing cffi (1.15.1)
  • Installing asgiref (3.5.2)
  • Installing cryptography (38.0.1)
  • Installing sqlparse (0.4.3)
  • Installing django (4.1.2)
  • Installing pymysql (1.0.2)

Installing the current project: surge (0.1.0)

~/python/django/Surge $poetry run python -c "import django ; print(django.file)"

Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'django' has no attribute 'file'

~/python/django/Surge $poetry shell

Spawning shell within /home/mryan/python/django/Surge/.venv
. /home/mryan/python/django/Surge/.venv/bin/activate
Virtual environment already activated: /home/mryan/python/django/Surge/.venv
~/python/django/Surge $. /home/mryan/python/django/Surge/.venv/bin/activate

Added this to show it is not specific to manage.py

(surge-py3.10) ~/python/django/Surge $python

Python 3.10.6 (main, Oct  5 2022, 12:42:52) [GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import django
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'django'
>>> exit()

(surge-py3.10) ~/python/django/Surge $cat manage.py

#!/usr/bin/env python
"""Django's command-line utility for administrative tasks."""
import os
import sys

def main():
    """Run administrative tasks."""
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'surge.settings')
    try:
        from django.core.management import execute_from_command_line
    except ImportError as exc:
        raise ImportError(
            "Couldn't import Django. Are you sure it's installed and "
            "available on your PYTHONPATH environment variable? Did you "
            "forget to activate a virtual environment?"
        ) from exc
    execute_from_command_line(sys.argv)


if __name__ == '__main__':
    main()

[clintonroy]

Sorry, meant to be Django.__file__

[clintonroy]

Gargh. Markup. Dunder file dunder

[clintonroy]

Again, I don't understand what pyenv is doing, so don't use any aliases referring to that

[clintonroy]

And poetry run manage.py I guess

[TerminatedProcess]

Out of time for the moment.. but I'll be back later and will re-create this test.

[MalikRumi]

Similar but not identical experience here. I am using:

• Poetry (version 1.8.2)
  

• pip 24.3.1
  

• Python 3.12
  

• PyCharm 2024.1.6
  

• bertopic-0.16.4
  

• Apple Silicon M1
  

• I have mlx but it is not installed in this repo.
  

• macOS Sequoia 15.1
  

Neither bertopic nor one dependency, llvmlite (0.34.0) would install with poetry. When I used pip instead, they installed as expected, but note that prior to this, poetry had successfully installed all the other dependencies. All packages are in the same site-packages subfolder.

poetry add bertopic
Using version ^0.16.4 for bertopic
Updating dependencies
Resolving dependencies... (5.8s)
Package operations: 43 installs, 0 updates, 0 removals

When I put bertopic's quickstart.py into the editor, PyCharm gave a warning that neither bertopic nor sklearn were in requirements.txt. But when you click to have that done, PyCharm knows to suggest pyproject.toml. It also gives me a little pop up warning that 'poetry.lock is out of date'. The reason I am posting is because neither pyproject nor poetry lock updated to show the rather long list of dependencies that poetry did install, even after calling poetry update. Is this going to be a problem in the near future? Should I manually put all of them into pyproject? Note, I no longer get the lock out of date pop up after running poetry update.

poetry.lock:

# This file is automatically @generated by Poetry 1.8.2 and should not be changed by hand.
package = []

[metadata]
lock-version = "2.0"
python-versions = ">=3.12,<3.13"
content-hash = "0ea8188f21dc2a6f62ce3b3fde9c8d0633e3e4c489a157e2dac223ce903e74c4"

pyproject.toml:

[tool.poetry]
name = "maarten"
version = "0.1.0" # 1.8.2, really
description = ""
authors = ["malikarumi <malik.a.rumi@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = ">=3.12,<3.13"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"