Skip to content

Homebrew and Python

Jump to bottom Edit New page
Frizlab edited this page Mar 15, 2013 · 28 revisions

Python is a powerful and beautiful interpreted language.

Overview

Homebrew is designed to work - or should work ;-) - with any CPython, in particular with OS X system's Python, but also provides formulae to brew a more up-to-date Python 2.7.x. However, if you choose to use another Python than these two alternatives (system Python or brewed Python), the Homebrew team cannot support you with issues.

Brew Python

We recommend to brew install python because:

  • Comes with pip (and distribute)
  • Python (distutils) finds brewed software (includes, libs), knows about the compiler and flags even if the command line tools for Xcode are not installed.
  • No need to set the PYTHONPATH for Homebrew bindings.
  • No need to work-around the sudo-is-needed-for-easy_install issue, described there --> Gems, Eggs and Perl Modules.

Python 2.x or Python 3.x

Homebrew provides a formula for Python 2.7.x and one for Python 3.x. They don't conflict, so they can both be installed. The executable python will always point to the 2.x and python3 to the 3.x version. But which version should I use?

As the time of writing (Dec. 2012), almost all bindings are installed for Python 2.x only - even if the software supports Python 3.x. There is an issue to discuss this topic.

Distribute, Pip, etc.

The Python formula installs Pip and Distribute, the latter of which provides easy_install. Note that modules installed with easy_install cannot be uninstalled whereas pip uninstall is possible if it was installed with pip install.

Distribute can be updated via Pip, without having to re-brew Python:

pip install --upgrade distribute

Similarly, Pip can be used to upgrade itself via:

pip install --upgrade pip

Note, pip install --user is disabled for brewed Python. This is a bug in distutils.

The site-packages and the PYTHONPATH.

The site-packages is a directory to contain Python modules, especially bindings installed by other formulae. Homebrew creates such a directory at $(brew --prefix)/lib/pythonX.Y/site-packages for example /usr/local/lib/python2.7/site-packages for python2.7. The reasoning is that for (minor) upgrades or reinstalls of Python, your modules are not lost. And a rather strict Homebrew policy is not to write stuff outside of the brew --prefix, so we don't spam your system.

A brewed Python 2.7 also searches for modules in

  • /Library/Python/2.7/site-packages and in
  • python -c "import site; print(site.USER_SITE)" => /Users/<your_name>/Library/Python/2.7/lib/python/site-packages

Homebrew's site-packages dir is first created if any Homebrew formula with Python bindings is installed or if brew install python. A brewed Python already knows about this dir.
For other Pythons you'll have to add that dir to your PYTHONPATH environment variable. You may want to append a line like so to your hidden configuration file .bash_profile in your home dir:

This is not needed if you use a brewed Python!

touch ~/.bash_profile
echo export PYTHONPATH=\\"$(brew --prefix)/lib/python2.7/site-packages:\$PYTHONPATH\\" >> ~/.bash_profile
source ~/.bash_profile
echo $PYTHONPATH

Homebrew provided Python bindings

Some formulae provide python bindings. Sometimes a --with-python (or similar) option has to be passed to brew install in order to build the python bindings. Check with brew options <formula>.
If you have a brewed python, then the bindings are installed for that one. But basically homebrew just uses the first python (and python-config) in your PATH. Check that by which python.

Warning, Python may crash (see Common Issues) if you import <module> in a different python interpreter than the executable that was used during the brew install <formula_with_python_bindings>. Therefore, if you decide to switch between a brewed and system python, then (re-‍)install all formulae that provide python bindings (such as pyside, wxwidgets, pygtk, pygobject, opencv, vtk, boost to name just a few).

Other - non-homebrew - Python bindings

Our policy is that these should be installed via pip install <x>. To discover, you can use pip search, the new http://crate.io or http://pypi.python.org/pypi. Note, system Python does not provide pip but you can easy_install pip to fix that.

For a brewed Python, modules installed by pip or with python setup.py, will be put into the before-mentioned $(brew --prefix)/lib/pythonX.Y/site-packages directory, too.

Further, pip (or more precisely distutils) knows which compiler flags to set in order to build bindings for software installed in Homebrew (Find the includes, the libs, the compiler and so forth). In contrast to that, system's Python does not know about the the correct flags, so you may need to

CFLAGS=-I$(brew --prefix)/include LDFLAGS=-L$(brew --prefix)/lib pip install <package>.

Virtualenv

A brewed Python works nicely with Virtualenv. However, when you brew install formulae that provide Python bindings, you should NOT be in an active virtual environment. Activate the virtualenv after you have brewed or, alternatively, brew in a fresh Terminal window. The Python interpreter from that virtual environment would be used by Homebrew but the Python modules would still be put into Homebrew's site-packages and not into the virtual environment's site-package. Virtualenv has a switch to allow "global",i.e. Homebrew's, site-packages to be accessible from within the virtualenv.

How to create a formula with nice Python bindings

Don't use pip install <x> in a formula! If something can be installed with pip, it probably doesn't need to be in Homebrew. (See our criteria for acceptable formulae.)

Test the bindings with brewed and with system Python!

If the software provides a setup.py

The problem: Just calling system "python", "setup.py", "--prefix=#{prefix}" will not work.

We want to achieve that the Python bindings are installed into the Cellar for that specific formula but when brew link is run (automatically at the end of brew install), the Python modules should be linked into $(brew --prefix)/lib/pythonX.Y/site-packages and the scripts should go into $(brew --prefix)/share/python. Only then can we unlink/link/uninstall correctly.

  ...
  def install
    # In order to install into the Cellar, the dir must exist and be in the PYTHONPATH.
    temp_site_packages = lib/which_python/'site-packages'
    mkdir_p temp_site_packages
    ENV['PYTHONPATH'] = temp_site_packages
    
    args = [
      "--no-user-cfg",
      "--verbose",
      "install",
      "--force",
      "--install-scripts=#{bin}",
      "--install-lib=#{temp_site_packages}",
      "--install-data=#{share}",
      "--install-headers=#{include}",
    ]

    system "python", "-s", "setup.py", *args
  end

  def which_python
    # Update this once we have something like [this](https://github.com/mxcl/homebrew/issues/11204)
    "python" + `python -c 'import sys;print(sys.version[:3])'`.strip
  end

NOTE: just setting the --prefix and/or --root does not work because of a bug in distutils when we also define a distutils.cfg which already specifies the more precise --install-lib and --install-scripts.

Explanation:

  • --no-user-cfg ignore ~/.pydistutils.cfg that can set another prefix etc.
  • --verbose get better debug info's if brew install -v is called.
  • --force overwrite existing files.
  • --install-scripts for scripts.
  • --install-lib=#{temp_site_packages} for pure python modules and compiled shared libs.
  • --install-data for things that should end up in HOMEBREW_PREFIX/share
  • --install-headers for C header files (if any.)
  • -s to disable the user site-packages that may possibly conflict with our module.

If the formula used cmake

todo
It's rather messy. Have a look at the vtk formula.

If the formula uses configure/make

todo

Technical details for maintainers

Homebrew performs three actions to set up the site-packages.

  • First, the Cellar site-packages folder is removed, and a symlink to /usr/local/lib/python2.7/site-packages in the prefix (=HOMEBREW_PREFIX/Cellar/python/2.7.x) is created. This will allow site-packages to persist between Python updates, as Homebrew has special handling for some languages that use lib for user-installable libraries.
  • Second, a distutils.cfg file is written to set the install-scripts folder to /usr/local/share/python. Users can add /usr/local/share/python to the PATH to pick up installed scripts.
  • Third, a site-packes/sitecustomize.py is written in order to
    • allow other non-brewed python to parse .pth files in our site-packages
    • filter out dirs from the PYTHONPATH, starting with /System/...
    • remove the hard coded site-package location in the Cellar (where we installed Python to) so that pip uninstall works.

Standard python installers can be run as python setup.py install and the proper paths will be selected.

Add a custom footer
Add a custom sidebar

Clone this wiki locally