Difference between revisions of "Apache Airflow"

From wikieduonline
Jump to navigation Jump to search
 
(23 intermediate revisions by the same user not shown)
Line 3: Line 3:
  
 
* [[Cloud Composer]]
 
* [[Cloud Composer]]
* Installation:
+
* [[Apache Airflow installation|Installation]]
** PIP: <code>[[pip3 install apache-airflow]]</code>
 
** Homebrew: not available
 
** [[Airflow configuration|Configuration]]
 
** Kubernetes:
 
:::<code>[[helm repo add apache-airflow]] https://airflow.apache.org</code>
 
:::<code>[[helm upgrade --install]] airflow apache-airflow/airflow --namespace airflow --create-namespace</code>
 
  
 
== Commands ==
 
== Commands ==
 
* <code>[[airflow --help]]</code>
 
* <code>[[airflow --help]]</code>
 
* <code>[[airflow db]]</code>
 
* <code>[[airflow db]]</code>
 +
* <code>[[airflow webserver]]</code>
 +
* <code>[[airflow standalone]]</code>
 +
* <code>[[airflow version]]</code>
  
 
== Logs ==
 
== Logs ==
 
* [[Started process]]
 
* [[Started process]]
 +
* [[Fluentd]] recommended for logs
 +
* [[StatsD]] recommended for metrics to [[Prometheus]]
  
 
== Related ==
 
== Related ==
 
* [[Directed acyclic graph (DAG)]]
 
* [[Directed acyclic graph (DAG)]]
 +
* [[Drawing DAGs]]
 
* [[Airflow Scheduler]]
 
* [[Airflow Scheduler]]
* [[Amazon Managed Workflows for Apache Airflow (MWAA)]]
+
* [[Airflow Operators]]
 +
* [[Job scheduler]]
 +
* [[Apache NiFi]]
 +
* [[Amazon Managed Workflows for Apache Airflow (MWAA)]] (May 2021))
 
* [[Apache Airflow providers]]: https://airflow.apache.org/docs/apache-airflow-providers/
 
* [[Apache Airflow providers]]: https://airflow.apache.org/docs/apache-airflow-providers/
  
Line 29: Line 32:
 
   port: "8126"
 
   port: "8126"
 
   metricsets: [ '[[statsd]]' ]
 
   metricsets: [ '[[statsd]]' ]
 +
* [[Apache airflow metrics]]
 +
* [[Running Airflow in Docker]]
 +
 +
== Activities ==
 +
* Read https://www.astronomer.io/blog/7-common-errors-to-check-when-debugging-airflow-dag/
 +
* [[Datadog Airflow integration]]
  
 
== See also ==
 
== See also ==
Line 35: Line 44:
 
* {{airflow cmd}}
 
* {{airflow cmd}}
 
* {{Airflow}}
 
* {{Airflow}}
* {{DAG}}
 
  
[[Category:Big data]]
 
 
[[Category:Apache]]
 
[[Category:Apache]]
 
[[Category:Airflow]]
 
[[Category:Airflow]]

Latest revision as of 17:07, 18 January 2023

wikipedia:Apache Airflow Workflow management system is a platform to programmatically author, schedule, and monitor workflows.

Commands[edit]

Logs[edit]

Related[edit]

/etc/metricbeat/metricbeat.reference.yml
- module: airflow
  host: "localhost"
  port: "8126"
  metricsets: [ 'statsd' ]

Activities[edit]

See also[edit]

Advertising: