Data Platform/Systems/Airflow/Instances
WMF's Airflow system is composed of several Airflow instances. Each instance is supposed to schedule and orchestrate jobs belonging to a particular grouping of teams or common purposes.
In general, each instance is associated with a Service user account, which equates to a POSIX user id and corresponding ownership on the Hadoop HDFS file system. It is now possible to override the service user for a particular DAG, so this mapping of instances to file ownership on HDFS is no longer a strict requirement.
The primary Airflow instance is called main which schedules jobs that generate and process analytics data sets. There's another instance called research, which orchestrates jobs that process research-related data sets. Usually, each Airflow instance is managed by a given WMF team, for example, the main instance is managed by the Data Engineering team, and most of its jobs have been developed by them. However, an Airflow instance can also be shared by several teams, and also one team can part-take in the development of jobs in multiple Airflow instances.
Multi-instance vs. single instance
During the development of WMF's Airflow system, we've had discussions about using a single instance approach versus a multi-instance approach. There are advantages and disadvantages in both cases. This thread contains most of the arguments we discussed, which include the following:
| Single instance | Multi-instance | |
|---|---|---|
| Pros | Single configuration, no custom stacks for teams, and thus, easy upgrades and maintenance. | No single point of failure, if a team deploys code that breaks Airflow services, the other instances continue working. Teams have more independence when deploying. |
| Cons | Airflow does not support Kerberos multitenancy (yet), so one single instance would require that all WMF jobs accessed Hadoop with the same Kerberos credentials, not allowing for access control or specific permissions. | When doing maintenance, the Data Engineering team will have to wrangle multiple airflow instances to stop jobs. |
We decided to kick off the project with a multi-instance approach, mainly because of the Kerberos issue, but we don't discard the possibility of switching to single instance in the future.
Airflow and Kubernetes
When we started, all of the Airflow instances were running on either bare-metal or VMs and the configuration of the instances was managed by puppet. Since then, we have completed a migration to Kubernetes so now all instances except the analytics instance now run on the dse-k8s-eqiad Kubernetes cluster. We only retain this instance for historical reference or back-fill purposes. The VMs are also retained temporarily in order to assist with the artifact deployment and will be decommissioned as soon as a replacement for this element has been deployed.
List of instances
analytics_product
Airflow instance owned by the Product Analytics engineering team. Contains all production jobs historically developed by the team.
analytics_test
Airflow test instance owned by the Data Engineering team. Contains some jobs analog to the ones in the main instance.
n.b. This instance is configured to use the analytics_test_hadoop cluster.
dumps
Airflow instance managing jobs producing artifacts downloadable from https://dumps.wikimedia.org/ (note: some WMDE-related dump jobs are running in the wmde instance)
fr-tech
Airflow instance owned by the Fundraising team. Contains all production jobs developed by the team.
main
Airflow instance currently owned by the Data Engineering team. We also named it this way in expectation that all instances would be folded into this one once Airflow becomes multi-tenant.
ml
Airflow instance owned by the ML team. Contains all production jobs developed by the team.
platform_eng
Airflow instance owned by the Platform Engineering team.
research
Airflow instance owned by the Research team.
search
Airflow instance owned by the Search team.
sre
Airflow instance owned by the ML team. Contains all production jobs developed by the team.
test-k8s
This is an instance that is owned by the Data Platform SRE team and is used to test new features and versions. It is currently also used to run dumps v1 DAGs.
wikidata
Airflow instance owned by the Wikidata Platform team. Contains all production jobs developed by the team.
wmde
Airflow instance owned by the WMDE engineering team. Contains all production jobs historically developed by the team.
Requesting elevated permissions for an instance
If you find yourself in need of removing/editing Airflow Variables records, looking at the instance configuration, etc, then you can now request elevated privileges by going to https://idm.wikimedia.org/permissions/ and requesting access to whatever airflow ops group makes sense. The request will be sent to managers for approval, after which, you'll need to logout from idp, log back in, reload airflow, and you should have the necessary permissions.
Custom test instance
More at Analytics/Systems/Airflow/Airflow_testing_instance_tutorial