While I have been using a simpler version in the past, it is only recently that I wrapped my head around how to simplify the creation–and really the maintenance–of upstream & downstream task dependencies in a DAG. The way I had been doing it involved writing these massive tuples with the task names over & […]
Tag: TDD
Testing SqlSensor & dealing with str.startswith
This was one of those “Of course multiple values can be passed/checked!”-type situations. I’ve written a couple new DAGs which use SqlSensor, and I want to use my existing test, test_database_operators_have_sql(), to make sure that I am passing a SQL statement to the SqlSensor task. Here’s how the test originally looked testing for just the […]
Using pytest.fixture to Test Airflow Context
I’ve been wanting to figure out a way that will allow me to test tasks that reference Airflow context (via provide_context=True), such as BranchPythonOperator orPythonOperator. What I’ve come up with is a fixture that returns a dictionary. (In my example below, I perhaps went a little overboard with configuring it to return all of the […]
Revisiting “Database Operators Have SQL” test
Back in my first post, https://learningtotest.com/2019/06/13/3-easy-airflow-tests/, the third test made sure that there was valid SQL attached to the task. I’ve come to realize that a downside to that particular snippet is that the output doesn’t tell you the name of the task that failed, but returns output like this: That’s not very helpful! We […]
Verifying Task Order Within The DAG
One of my favorite tests for a DAG is the one to confirm the connectivity between tasks. If there is a limitation to this particular implementation, it’s that it only works with rather linear DAGs. (So if you DAG contains a number of forks & merges, this probably isn’t going to work as shown.) Linear […]
3 Easy Airflow Tests
I haven’t exactly found a wealth of information on how to test with Airflow. Here are 3 tests though that I’ve used with every single DAG that I’ve written. The first one does nothing more than verify that there are no syntax errors within the DAG file, the second one confirms the existence of and […]