I’m continuing to enjoy the relaxed abilities that BigQuery brings compared to other common databases. However, yesterday I was bitten by how relaxed it can be. I had been refactoring an existing scheduled query & as part of it, I had been (manually) converting many of the reserved words in ALL CAPS. Well, I had […]
Tag: SQL
Today I Learned: BigQuery Scheduled Queries Edition
Today I learned to never, ever assume that any scheduled queries written & deployed by someone else won’t be contradicted by that same person in another scheduled query. If you find a scheduled query (don’t use them, just don’t…) that populates a table but produces a result different than what you expect (read: has significantly […]
Interviewing Data Engineer Candidates
At my current employer, as a part of the interviewing process we have a technical challenge & team interview (after an initial phone screen interview). One thing that I really appreciate about where I work now is that when it comes to considering data engineer candidates, we absolutely will bring in people that lack any […]
Revisiting “Database Operators Have SQL” test
Back in my first post, https://learningtotest.com/2019/06/13/3-easy-airflow-tests/, the third test made sure that there was valid SQL attached to the task. I’ve come to realize that a downside to that particular snippet is that the output doesn’t tell you the name of the task that failed, but returns output like this: That’s not very helpful! We […]
3 Easy Airflow Tests
I haven’t exactly found a wealth of information on how to test with Airflow. Here are 3 tests though that I’ve used with every single DAG that I’ve written. The first one does nothing more than verify that there are no syntax errors within the DAG file, the second one confirms the existence of and […]