multiprocessing on OSX

The Problem

numpy is linked to one of several C-based linear algebra libraries that perform computations. On OSX, the default C-based library for linear algebra computations is called Accelerate. Unfortunately, Accelerate doesn’t support computation on either side of a fork, which means that if, for example, you try to use numpy to calculate a maximum a posteriori fit to a set of data using a Gaussian Process likelihood and then perform a parallelized Markov Chain Monte Carlo error analysis using multiprocessing, you will get an unexplained segfault (on my 2017 MacBook Pro, a segfault results in a “Python has quit unexpectedly” message). They could at least throw an informative error, in my opinion, but oh well.

This is a well-documented issue (you’ll find several conversations about it by Googling “multiprocessing accelerate segfault”) that Apple is opposed to fixing.

To work around this issue, we suggest:

1. Running your parallel Gaussian Process MCMC computations on a non-OSX machine, if you have access to one. This is only a problem on OSX, so a Windows, Linux, etc. machine will do fine.

  1. Running your Gaussian Process MCMC computations in serial.

3. Re-building your version of numpy using a different C-based linear algebra library. Instructions are below. It’s not as hard as it sounds!

Rebuilding numpy

conda-forge provides a version of numpy built on openBLAS, a C-based linear algebra library that works just as well as Accelerate, and is in fact the default on several other OS types.

First of all, we suggest doing all of this in a conda virtual environment, and subsequently running all of your radvel computations within that environment. Install anaconda or miniconda, then see instructions here. to learn more about conda virtual environments.

Step 0: optionally create and activate a conda virtual environment.

$ conda create --name radvel_virtualenv --python=3.6 # create a virtual environment for best results
$ source activate radvel_virtualenv # activate the newly minted virtual environment

Step 1: uninstall & reinstall numpy:

$ conda uninstall numpy # clear numpy installation from current environment
$ conda install --channel "conda-forge" numpy # reinstall conda-forge version of numpy

Step 2: reinstall packages that rely on numpy

$ conda install scipy matplotlib cython astropy pandas
$ pip install corner celerite emcee
$ pip install radvel --upgrade  #reinstall radvel

That should be it! As always, please submit an issue on the radvel GitHub page if you run into any problems.