Datadog Airflow integration

TO_MERGE: Datadog Airflow

• Kubernetes: https://airflow.apache.org/docs/helm-chart/stable/production-guide.html#datadog
• Windows/Linux https://docs.datadoghq.com/integrations/airflow/
• Official documentation with dogstatsd_mapper_profile: configuration for the agent: https://app.datadoghq.eu/integrations/airflow
• https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html

statsd_datadog_enabled
statsd_datadog_tags

Airflow monitoring

[metrics]
statsd_on = True
statsd_host = localhost
statsd_port = 8125
statsd_prefix = airflow
#statsd_allow_list = scheduler,executor,dagrun
statsd_datadog_enabled = False

Logs

triggerer | [2022-12-05 14:02:35,051] {base.py:317} INFO - Statsd buffering is disabled
triggerer | [2022-12-05 14:02:35,052] {base.py:335} INFO - Statsd periodic buffer flush is disabled
triggerer | [2022-12-05 14:02:35,053] {base.py:317} INFO - Statsd buffering is disabled
triggerer | [2022-12-05 14:02:35,053] {base.py:335} INFO - Statsd periodic buffer flush is disabled

scheduler | [2022-12-05 14:02:35,223] {base.py:317} INFO - Statsd buffering is disabled
scheduler | [2022-12-05 14:02:35,234] {base.py:335} INFO - Statsd periodic buffer flush is disabled
scheduler | [2022-12-05 14:02:35,235] {base.py:317} INFO - Statsd buffering is disabled
scheduler | [2022-12-05 14:02:35,235] {base.py:335} INFO - Statsd periodic buffer flush is disabled

webserver | [2022-12-05 14:02:36,135] {base.py:317} INFO - Statsd buffering is disabled
webserver | [2022-12-05 14:02:36,135] {base.py:335} INFO - Statsd periodic buffer flush is disabled
webserver | [2022-12-05 14:02:36,136] {base.py:317} INFO - Statsd buffering is disabled
webserver | [2022-12-05 14:02:36,136] {base.py:335} INFO - Statsd periodic buffer flush is disabled

Errors

• ERROR airflow.stats Could not configure StatsClient: No module named 'datadog', using DummyStatsLogger instead
• Error submitting packet: [Errno 111] Connection refused, dropping the packet and closing the socket, "level": "WARNING

Related

• airflow.cfg
• airflow config list | grep statsd
• 8125 port
• Airflow base_url default value http://localhost:8080
• DogStatsD: pip install 'apache-airflow[statsd]'
• DD_DOGSTATSD_MAPPER_PROFILES

See also

• Datadog, Datadog agent, Spans, DBM, standard attribute list, facets, log explorer, Terraform datadog, Airflow, dog, agent, DogStatsD, Airflow integration