I have never used Luigi. It's been around for a while and it is still active maintained with a great community around it.
We considered it in our selection process. We preferred Airflow because it included the scheduler. The author of that comparison would also go with Airflow.
Where I work, we've been using Snakemake[1] or data pipeliens. It's like GNU-make but with a Pythonic syntax. It determines which processing steps need to be executed by building a directed acyclic graph from the end to the beginning; it figures out which operations are needed to produce a given output, then looks at those and figures out their inputs, and so it. It can even submit parallel processing jobs to a batch-queueing cluster.
By far Azkaban was the most appropriate. We're investigating coupling it with Goblin and https://github.com/linkedin/WhereHows which has native integration.
Apache NiFi looks interesting as well, but we haven't looked at it yet.
Hey Jean-Nicholas, great post. You could try out https://www.blendo.co/ too. If you like it (I am sure you will) you may add it in the list too ;) (disclaimer: I am one of its co-founders)