Sequence to Install Darts in MacBook M1. Or The hard thing about hard dependencies

This post could called “The hard things about hard dependencies”, but since lot of people can ended up with the same issue I decided to have the original title.

…

Sequence to Install Darts in MacBook M1

For whom does not know, Darts it’s a framework for time series forecasting and it’s well-established in the Python ML community.

During some projects I got the following error, related to Facebook Prophet which is another time series framework:

After some search I found out that the solution was to use the conda installation, as we can see below:

The root cause is that the version of cmdstanpy fails during the build of the wheel when the installation comes from the PyPi, and updating the pip version using conda solves the issue.

However, after the solution, I need to confess that I got a bad taste in my mouth that is related to the state of the Python ecosystem and so on.

The hard thing about hard dependencies

At the time that I was tracing back the error, I found out that Darts has a hard dependency on prophet>=1.1.1 that has a hard dependency in cmdstanpy>=1.0.4.

The heart of the problem was that Facebook Prophet made a change in its backend that went from pystan==2.19.1.1 to cmdstanpy.

In other words, Facebook Prophet considered the change of a backend a minor version and Darts kept its compatibility with the up-to-date version. In the meantime since an entire family of last-generation laptops did not have the wheel. Nothing wrong here, but the thing that did bother me was the fact that it was a known issue, at least was known by the cmdstanpy community.

I should not be bothered, because at the end of the day Darts it’s a wrapper (so it’s expected those transient dependency issues), the Facebook Prophet team is entitled to consider that a change in its probabilistic backend is not something that should be in a major version, Apple has the right to deploy any architecture path as they wish, and the cmdstanpy community does not owe anything to anyone.

Fair play and life goes on.

I still strongly agree with @richhickey when he said in a terrific essay called Open Source Is Not About You, but the point of my initial frustration was the fact that those “hard second level dependencies” acts in somewhat like a compound over unknown-unknown complexities that makes any environment silently more brittle.

My feeling is that a not well-thought process to choose tools can make engineers feel more or less like being in a brittle Jenga tower of technological pieces tied together, more or less like the classical dependency meme.

Final thoughts

Understanding your hard requirements it’s an art that needs to be preserved and a well-reasoned tooling choice avoids anxiety and loss of productivity.

I hope that at some point in the future, someone will find this virtual venting useful either for the cmdstanpy problem or to reflect on 30 seconds about open source and hard dependencies.