Airflow works well for "static" jobs, but I miss something like airflow for dynamic jobs.
By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.
I work on such a system, to organize data processing (HPC) projects in the oil & gas industry, and I try to follow this space. I remember I got excited when I heard of Airflow for the first time, but quickly got frustrated with its "static flow" nature: many "flow" systems are like this, you first design the flow, then "deploy" it and let it run (usually many times).
What our tool does is allow users to organize the flow of their processing jobs on an infinite 2D layout, have some jobs run at the beginning of the flow while they organize another part to run later.
Unfortunately it's a big pile of messy code that depends too much on other "internal" systems so we can't open source it... I'd like to add "yet" because I try to gradually clean it up, simplify and make it more generic, but I'm not sure I'll see that day myself.
I've started working on a code generator for Airflow. Not primarily because I needed dynamic jobs, but more because I didn't want to keep writing the Airflow boilerplate.
I imagine I'll eventually need to add some sort of management system to move these dynamic jobs in and out of Airflow to keep them from bloating the database or cluttering the UI.
I'd be interested in hearing more about your project and if you plan to open source. Are you using something like Cookiecutter or Yeoman, or a different level of abstraction?
My effort is a collection of custom operators for operations that we use very commonly (run a query, python script, or mapreduce job) and a code generator that takes a simple text spec describing a set of operations and generates the Python DAG script. I'm not familiar with Cookiecutter or Yeoman but generating Python code myself hasn't been complicated. Open-sourcing it isn't an option right now with my current employer.
Thanks, this sounds really interesting. If this changes or you decide to write personal code with similar structure, I'd definitely like to discuss further. I'm also currently exploring DAG structure but with class and function abstractions with the goals of increasing code reuse and minimizing boilerplate around our custom operators.
Cookiecutter takes a Python code file as input with Jinja interspersed (the template). When you want to make a new instance from the template, it gives command-line prompts, and then evaluates the Jinja logic — custom variables, loops, etc to output Python code. It took me 15–30 minutes to get started and has already paid off.
This allows to build ETL jobs which react to all sorts of external application triggers (such as an upload made by a user in that case, but it could be an API notification / webhook etc).
While it's not the exact use case you've described (and it's hard to pinpoint without a concrete example), you could handle related tasks by writing a more generic static job that pulls a dynamic config, eg from a mongo db, and uses the branch operator.
How about a custom sensor class that never timesout and you define its poke method to listen for data and processes it? Granted, you'll have to make sure you allocate enough workers for that..
Airflow is just the workflow management layer on top of your data pipeline. The flexibility to generate custom graphs based on user-specific parameters should be handled within a pipeline task.
Based on your example, I would have a single dag that would 1. get user data and 2. generate a graph.
All the flexibility should be defined in whatever function, script or program you define to generate the graph.
I think they are referring to "event-driven" DAGs which would be both shaped and triggered dynamically. You can accomplish this now but it feels a bit hacky and is pretty clear that it goes against the Airflow paradigm of static, slowly-changing workflows
[Airflow author here] in general, and when thinking in terms of best practices, we like to think of a DAG's shape as slowly changing, in a similar way that a database's tables definition is slowly changing. In general, you don't want to change your table's structure dynamically. This constraint brings a certain clarity and maintainability, and most use cases can be expressed this way.
Now. Airflow allows you to do what you're describing as well and will explain how to. If you were my coworker I'd dig deeper and try to understand whether the design you want is the design that is best, but let's assume it is. So first we support "externally triggered DAGs", which means those workflows don't run on a schedule, they run when they are triggered, either by some sensor, or externally in some way. A use case for that would be some company processing genomes files, and everytime a new genome file shows up, we want to run a static DAG for it.
https://airflow.incubator.apache.org/scheduler.html#external...
Now if your DAG's shape changes dramatically at every run [a shapeshifting DAG!], I would argue that conceptually they are different DAGs, and would instruct to build "singleton" DAGs dynamically. Meaning you have python code that creates a dag object [with its own dag_id] for each "instance", with the schedule_interval='@once', meaning each DAG will run only once. You can shape each DAG individually, from that same script, and craft whatever dependency you might like for each one.
Though all of this is not only possible and easy-ish to do, it may not be the best approach. Try to think of your DAGs and tables as static [or slowly changing] if you can, and the data as the variable.
As an analogy, try to think of an oil pipeline that changes shape based on the quality of the oil it processes. Crazy?! It's easier to think of the pipeline as static and infrastructure, and to have components that can sort and direct the flow in [existing and static] pipes.
Starting from 1.8 you will be able to trigger dags through a rest API, that is fully supported.
Shaping DAGs dynamically poses a challenge to the scheduler on how to 'predict' what tasks need to run in the future. The scheduler needs to evaluate which tasks will need to run, without actually executing these tasks themselves. For Airflow in its current state that is a chicken and egg problem.
For the future, I can think of allowing dynamic dags being described through the Rest API, but that is definitely further out and has not really popped up yet on the horizon.
By dynamic, I mean something like "user sent us some new data to process, create a custom graph just for this data". I can create new airflow graph per each processing pipeline with new dag id every time, but airflow was not created for use case like this and it's not working well in such scenario.