Apache Airflow

wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.

• https://airflow.apache.org/

• Cloud Composer
• Installation:
  • PIP: pip3 install apache-airflow
  • Homebrew: not available
  • Configuration
  • Kubernetes:

helm repo add apache-airflow https://airflow.apache.org

helm upgrade --install airflow apache-airflow/airflow --namespace airflow --create-namespace

Commands

• airflow --help
• airflow db

Logs

• Started process

Related

• Directed acyclic graph (DAG)
• Airflow Scheduler
• Amazon Managed Workflows for Apache Airflow (MWAA)
• Apache Airflow providers: https://airflow.apache.org/docs/apache-airflow-providers/

/etc/metricbeat/metricbeat.reference.yml
- module: airflow
  host: "localhost"
  port: "8126"
  metricsets: [ 'statsd' ]

See also

• Astronomer and Astronomer Cloud
• Cloud composer service on Google Cloud Platform
• Apache Airflow, Installation, configuration airflow.cfg, monitoring, Amazon Managed Workflows for Apache Airflow (MWAA), airflow, Airflow Scheduler, providers, Workflows, Airflow executors, Airflow versions, Datadog integration
• DAG, Airflow