11,965 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
0
votes
1
answer
46
views
Why does Airflow BashOperator fetch multiple days of data when API URL has single-day date range?
I am learning Apache Airflow and created a simple DAG to fetch daily events from a REST API.When I call the API directly in the browser, it correctly returns one day of data
http://localhost:5000/...
2
votes
1
answer
48
views
Why are the "Last Run" and "Next Run" the same date?
When we run the simple dag in apache airflow the Last Run and Next Run show same date why is that
dag = DAG(
dag_id = '01_unscheduled',
schedule_interval='@daily',
start_date = dt....
0
votes
0
answers
18
views
Create Application Pools, Roles etc. on startup
We have an Airflow 3.1.5 instance which is deployed to k8s (AKS) using the official helm chart (1.1.8). The database is a PostgreSQL managed database instance.
This works well, however, I would like ...
3
votes
2
answers
180
views
Retrieve the error cause / exception from failed task instances
Let's say I have a DAG with some source1, source2, ... tasks and a sink task, with a simple [source1, source2, ...] >> sink dependency.
All of them are @task-annotated Python operators, and the ...
Best practices
0
votes
0
replies
36
views
Orchestrating multi-step workflow failure handlers
Let's say we have the following dummy data:
item_input = [
{"value": 1, "fail_step": 1},
{"value": 2, "fail_step": 1},
{"value": 3, "...
0
votes
0
answers
127
views
Local unit testing apache airflow
Cannot Run Apache Airflow Unit Tests: sqlalchemy.exc.OperationalError: unable to open database file
I am attempting to run a single unit test within the Apache Airflow repository (airflow-core/tests/...
1
vote
1
answer
86
views
XCom limit of 1024 in airflow
I am hitting the 1024 item limit in the size of a returned list in my task in airflow. I am currently on airflow v2.11. A simple bit of code that reproduces the issue I see is below.
import os, ...
1
vote
1
answer
98
views
Depedency Hell airflow+dbt
Hello I'm new to airflow, lately I'm struggling on a project with dbt+airflow+docker. My problem 1) I pip install dbt-core, dbt-duckdb adapter, 2) I try to install airflow with:
pip install "...
0
votes
1
answer
64
views
Add jinja template value with DateTime value
In an Airflow Dag file, I'm trying to handle following jinja template variables as DateTime values.
Specifically {{ data_interval_start }} and {{ data_interval_end }},
which (according to https://...
Jelly's user avatar
- 1,426
0
votes
0
answers
50
views
What does "state attribute is queued" mean for airflow 3?
I try to install the new airflow 3 with docker on a VM with this conf:
x-airflow-common: &airflow-common
image: ${AIRFLOW_IMAGE}
restart: unless-stopped
env_file: .env
environment:
# --...
Best practices
0
votes
0
replies
50
views
How to manage (Google Cloud Composer) Airflow roles with infrastructure as code?
Are there any best practices for handling roles in code?
Most of our setup is managed via Terraform. It would be great if there was a possibility.
There is a Terraform provider, however using the ...
0
votes
0
answers
65
views
KubernetesPodOperator - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
I’m running Apache Airflow inside a Docker container and trying to use the KubernetesPodOperator to run a simple "hello world" pod in an external Kubernetes cluster (not the same one where Airflow ...
Tooling
1
vote
0
replies
74
views
masking secrets airflow 2.4.3
I’m using Apache Airflow 2.4.3 and trying to securely store a Snowflake connection with a private key inside the connection’s extras JSON field.
I want to mask the sensitive private_key_content field ...
0
votes
0
answers
40
views
Airflow 3 - Run LocalExecutor with unlimited Parallelism
Airflow Docs say that the LocalExecutor supports an "unlimited Parallelism". Source
They do not specify exactly how to run LocalExecutor in this mode, but I assume it is through the core....
0
votes
0
answers
82
views
MWAA Airflow task_instance_mutation_hook not working
I have added a airflow_local_setting.py file to my mwaa s3 bucket at the root of dags folder. I have a dag_policy defined which is being picked up and works as expected.
In the same file I have added ...