install pip under PACKAGE_DESTINATION instead of GLOBAL_DESTINATION #98
Conversation
|
Having a quick look at https://github.com/k-okada/break_rqt_graph/ I see a few things that you don't want to do in a python package :
In general, the package should follow https://www.pypa.io/en/latest/ + As a consequence, the |
|
I think my concern is a slightly different from yours, but hope we can share the motivation.
So question is that if your package depends on other pip package, which conflict with system installed python package. What should we do? |
|
As far as I understand the problem, this is what "distributions" are supposed to solve : some authority guarantees that all packages included in the distribution work well together... I investigated the problem you had (I think - I had the same thing on my machine already, I just didn't notice). Details there (I was not sure where to put it) : k-okada/break_rqt_graph#1 So looking at http://packages.ubuntu.com/search?keywords=pyparsing&searchon=sourcenames It seems the ubuntu distribution still ships an "old" version of pyparsing. To solve this kind of A --depends on--> B problems with A and B being compatible only for some versions, either A can be modified to support more versions of B, or B can be modified to provide some backward compatible API. In that situation, I usually :
So for your pyparsing / rqt_graph, I assume it is simple enough to modify rqt_graph to get it to work with whatever version of pyparsing. Good luck ! Don't hesitate to keep me posted. |
|
Thinking again about this, python packages and debian packages have quite a different usecase :
So mutating a python package into a debian package will take some work, in order to make sure potential conflicts with other existing packages are avoided. |
|
I found out pyparsing is used by many, many python packages, so you probably don't want to change that version in a specific distribution. Which means that to fix these problems, it is better to chose a minimum distribution to support, and change the version of software you want to use to work with the corresponding version of pyparsing. Also pytorch relies on pyyaml (no minimum version specified), and pyyaml is used as a base for ros distributions (It think empy uses it), so you probably need to check which minimal version of pyyaml is necessary for different version of pytorch, and decide on one pytorch version that match a specific ros distro... |
|
Thanks for comment,
I think challenge here is that ROS build firm will distribute debian/ubuntu system package, where as ROS development system allow us to use python development environment, mainly A rational behind this is that;
If I look into these situation, I think evolution of development system is moving towards encapsule their development environment, Current In future, I think we'll encapsule the environment for each node like https://gist.github.com/ruffsl/4a24c26a1aa2cc733c64, but I think this is different story. |
As far as I understand ROS development system allow to use pip only for devel space. When installing the use of
Using That is why catkin_pip install the dependencies or your python packages in the devel space. And the PYTHONPATH necessary modifications are handled by envhooks, which need to play nice (correct ordering) with underlays and overlays, and other pip installed packages (so many different ways to support even pip is buggy, and we need to work around these until they get fixed)... We currently have a way to distribute one package through the build farm. But we do not have a way to distribute a set of packages that can contain different version of packages with another set of packages, in the same distro. Which is why we have multiple distros. -- Apart from that, I agree with your observations.
As you noticed catkin_pip strategy is to follow the python strategy of WORKSPACE based file system, by setting up this python workspace in catkin devel space (with catkin_pip tools in build folder, so they - especially the correct version of setuptools - can also can be used when installing) When building install space we need to prepare a structure that can be put inside a deb package, and installed with other ros package, in the distro. => The WORKSPACE is not your devel environment anymore, it is the whole distro, which can create conflicts and dependencies issues. But this is already the case with usual debian package, and with python packages, and is solved by tasking the package maintainers to make sure their package fit into the distro.
I also agree with this, but only when space does not matter, so we do not want to change what we currently have, but add some different way to do it. What I am saying is that ROS currently does not provide the required tools/infrastructure to encapsulate your whole (python) development environment in a "thing" that is easily deployable. Debian packages and Python packages rely on files structure where the dependency of a package is also part of the distribution, and expect an install to fail when there is a conflict. So we cannot rely on python package system, or on debian package system, to address these dependencies conflicts automatically when the user install a package. An interesting example of this might be what rosbridge did (embed a different version of a python package into another deb package) : Also we (ROS community) do not have the resources to invent some new way of packaging python code and maintain it reliably, so we need to rely on existing software for this.There are multiple options to this (snaps, docker, dh_virtualenv, and probably many other package systems that I am not aware of ). So catkin_pip is not trying to invent a new way to distribute new kinds of packages. The point of catkin_pip is to fix the way python packages are handled in ROS currently, which is broken when following python packaging authority. So I dropped the idea of packaging an entire python environment in a deb package when I realized that even something like dh_virtualenv would require buildfarm changes. -- That being said, I think you are hitting multiple problems :
If we forget the last one, to solve 1. 2. and 3. you used catkin_pip, but there are other (maybe simpler?) possibilities If you care only about devel space (multiple people working on same set of repos) you can have in your package, a set of submodules satisfying its specific dependencies requirements (I did that in https://github.com/asmodehn/rostful/tree/ff8d1c156ada44094b72a3be3310b6a8f149e009/deps because changing all dependencies to be the right version for the distro you are targeting, to be released as ros packages, is very time consuming). This should not require any change to the python path (since your current package is already in the path) or to the cmake. If you care about install space (distributing binaries/installs by copying files around - encapsulating everything into a docker or so), there is an issue listed for submodules to work there : #51. It might be already solved (since I didnt get any bug report about this for rostful...) but I haven't looked at it in a long time. But if you have to put them in a ROS debian package, and distribute that package, then they need to not create any conflict into the ROS distro, or on the user system. Putting more python packages in that deb file is not supported by the package system and will create all kinds of tricky problems. I am open to investigate how we can incorporate an existing solution like dh_virtualenv into catkin_pip workflow, but it will require some effort. Let me know if you are interested in that, but I see it as "another way" of distributing python packages, not the "standard" one (we should aim to stay as close to https://www.debian.org/doc/packaging-manuals/python-policy/ as ROS permits). We could extend the cmake API for this... -- Sorry for the long message, but I hope it is clear and detailed enough. |
|
It seems we're agree on most (all?) of topics ;-)
rosdep allow to use pip, and that good news for Python based developer,
they'll create ROS packages using pip based Python libraries, but currently
we do not have good way to distribute/deploy such package, and that makes
trouble in many cases. (could not release deb packages, installing pip by
users may break their system, put pip library into their source tree
breaks other libraries, and so on, actually I remember I once trouble with
tornado)
so the question is the downside of
4. you need to distribute it
vs
tricky/effort to find a way to distribute pip under PACKAGE space.
and what I see here is just o install pip library under PACKAGE space is
not difficult, just to change `--prefix`, so remaining issue is how to
activate that. I'm not daily Python programmer so not have enough knowledge
on this, but...
- use package-level python executable (which activate virtualenv inside
when running)
- use special command to actiavete virtual env
- add special command within <apckage>/__init__.py
and we have to consider
- when running standalone node
- when running via roslaunch
- when imported from other package (difficult???)
BTW, `dh_virtualenv` seems very interesting. from
https://labs.spotify.com/2013/10/10/packaging-in-your-packaging-dh-virtualenv/
,
does this provide `/usr/share/python/my_ros_package/bin/python` and we'll
call this python whenever we want run python node wihtin `my_ros_package` ?
If so, it seems almost equal to my idea of this PR to run python within
each package space, off cause my code is super duty and need to fix up (do
not set PYTHONENV, but activate virutal env and so on...)
…--
◉ Kei Okada
2017-04-05 13:04 GMT+09:00 AlexV <notifications@github.com>:
I think challenge here is that ROS build firm will distribute
debian/ubuntu system package, where as ROS development system allow us to
use python development environment, mainly pip.
As far as I understand ROS development system allow to use pip only for
devel space. When installing the use of pip is not supported. Even if
your dependency in rosdep has a pip key, it will install in /usr/local/
which is advised against since it might break your system (it can silently
install a different package version than the one that an ubuntu tool rely
on...). It will also work for install space though, as long as you are sure
that no conflict will occur on any user machine.
So ROS developer build their package using pip environment using rosdep or
this catkin_pip tool, however currently we do not have way to distribute
the system through build firm.
Using pip environment and rosdep will work for devel space, and even for
install space, but might break someone else machine when doing rosdep
install because he was not expecting something to be installed on the
system and change his ubuntu package behavior.
Note there is a small different between trusty and xenial here, since pip
by default install in userspace (not system) from xenial (using a debian
patch). But even in user space the problem remains : *rosdep will change
the default python setup for the user*.
That is why catkin_pip install the dependencies or your python packages in
the devel space. And the PYTHONPATH necessary modifications are handled by
envhooks
<https://github.com/asmodehn/catkin_pip/blob/devel/cmake/env-hooks/42.site_packages.sh.develspace.in>,
which need to play nice (correct ordering) with underlays and overlays, and
other pip installed packages (so many different ways to support even pip is
buggy, and we need to work around these until they get fixed)...
We currently have a way to distribute one package through the build farm.
But we do not have a way to distribute a *set* of packages that can
contain different version of packages with another *set* of packages, in
the same distro. Which is why we have multiple distros.
--
Apart from that, I agree with your observations.
I think about it in multiple layers like :
- SYSTEM layer, based on unix files hierarchy, or GLOBAL path, i.e.
/usr/bin
- WORKSPACE/virtual environment layer (user's home, ros distro path,
python virtualenv, docker image)
- PACKAGE content (files from deb package, files from pip package,
files from npm package)
As you noticed catkin_pip strategy is to follow the python strategy of
WORKSPACE based file system, by setting up this python workspace in catkin
devel space (with catkin_pip tools in build folder, so they - especially
the correct version of setuptools - can also can be used when installing)
When building install space we need to prepare a structure that can be put
inside a deb package, and installed with other ros package, in the distro.
=> The WORKSPACE is not your devel environment anymore, it is the whole
distro, which can create conflicts and dependencies issues.
But this is already the case with usual debian package, and with python
packages, and is solved by tasking the package maintainers to make sure
their package fit into the distro.
If I look into these situation, I think evolution of development system is
moving towards encapsule their development environment
I also agree with this, but *only when space does not matter*, so we do
not want to change what we currently have, but add some different way to do
it. What I am saying is that ROS currently does not provide the required
tools/infrastructure to encapsulate your whole (python) development
environment in a "thing" that is easily deployable.
Debian packages and Python packages rely on files structure where the
dependency of a package is also part of the distribution, and expect an
install to fail when there is a conflict. So we cannot rely on python
package system, or on debian package system, to address these dependencies
conflicts automatically when the user install a package.
An interesting example of this might be *what rosbridge did* (embed a
different version of a python package into another deb package) :
RobotWebTools/rosbridge_suite#162
<RobotWebTools/rosbridge_suite#162>
which *breaks pip* : ros-infrastructure/rosdep#456
<ros-infrastructure/rosdep#456>, and probably a
number of other tools and import behaviour in other python packages, and
apparently nobody tested/reported that for years...
We kept having problem with these dependencies at Yujin until I found the
time to dig and find out what was silently installing python packages with
unexpected version in our rosdistro workspace...
Also we (ROS community) do not have the resources to invent some new way
of packaging python code and maintain it reliably, so we need to rely on
existing software for this.There are multiple options to this (snaps,
docker, dh_virtualenv, and probably many other package systems that I am
not aware of ).
So catkin_pip is not trying to invent a new way to distribute new kinds of
packages. The point of catkin_pip is to fix the way python packages are
handled in ROS currently, which is broken when following python packaging
authority <https://www.pypa.io/en/latest/>.
So I dropped the idea of packaging an entire python environment in a deb
package when I realized that even something like dh_virtualenv would
require buildfarm changes.
--
That being said, I think you are hitting multiple problems :
1. you need a specific version of a pip package not compatible with
the ROS distro version
2. you need to install it (not on the system)
3. you need to import it
4. you need to distribute it
If we forget the last one, to solve 1. 2. and 3. you used catkin_pip, but
there are other (maybe simpler?) possibilities
If you care only about devel space (multiple people working on same set of
repos) you can have in your package, a set of submodules satisfying its
specific dependencies requirements (I did that in
https://github.com/asmodehn/rostful/tree/ff8d1c156ada44094b72a3be3310b6
a8f149e009/deps because changing all dependencies to be the right version
for the distro you are targeting, to be released as ros packages, is very
time consuming). This should not require any change to the python path
(since your current package is already in the path) or to the cmake.
If you care about install space (distributing binaries/installs by copying
files around - encapsulating everything into a docker or so), there is an
issue listed for submodules to work there : #51
<#51>. It might be already
solved (since I didnt get any bug report about this for rostful...) but I
haven't looked at it in a long time.
But if you have to put them in a ROS debian package, and distribute that
package, then they need to not create any conflict into the ROS distro, or
on the user system. Putting more python packages in that deb file is not
supported by the package system and will create all kinds of tricky
problems.
I am open to investigate how we can incorporate an existing solution like
dh_virtualenv <https://github.com/spotify/dh-virtualenv> into catkin_pip
workflow, but it will require some effort. Let me know if you are
interested in that, but I see it as "another way" of distributing python
packages, not the "standard" one (we should aim to stay as close to
https://www.debian.org/doc/packaging-manuals/python-policy/ as ROS
permits). We could extend the cmake API for this...
--
Sorry for the long message, but I hope it is clear and detailed enough.
Feel free to let me know if you want me to illustrate some part with more
details/example.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#98 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAeG3J2qfxUcN1HVt7aobAmSoLMz9FmSks5rsxLmgaJpZM4McZZr>
.
|
|
Yes installing a pip package anywhere is easy. But setting up everything to make it usable properly is really tricky (as you know rosdep with pip, or a ros package with multiple python packages, can break your system). The import machinery of python is quite complex. There are other things you didn't list in the things we have to consider :
So trying to minimize our development effort here, as far as I can see there are two ways to solve your problems :
So I think you should investigate how catkin_pip :
|
The motivation of this PR is discussed in ros-infrastructure/bloom#412 (comment)
pip insstall pyparsingwill breakrqt_graphin this example, or you may update PR something like