The reason for the lack of Airflow and/or PostgreSQL posts is simple: I took a new job. I left my old data engineering (DE) job a few days before Christmas & started a new job at the beginning of January. While I greatly enjoyed many aspects of the old job, company, & people that I worked with, this new opportunity was simply entirely too good an offer to pass up & I left in a way that would perhaps allow me to return one day in the future if things didn’t work out for whatever reason. I’m still doing DE, but–at least for now–there’s no python, much less Airflow, which does bum me out a little. The new job runs all of their ETL on Pentaho Data Integration (aka PDI or kettle).
I’ve used Pentaho in the past at a previous job, and I’ll get into the things that I do & do not like about Pentaho’s ETL tool. But for now, I wanted to explain myself & let you know that the general details of what I might write about here will be on slightly different but still hopefully relevant DE topics.
One of those topics–and it was one of the major reasons for me accepting this new job opportunity–was to be part of an actual data team that works in an Agile software development environment. I’ve always previously been a data engineering “team of one”, so getting to split up the workload & in a way that should be leaner & more productive had great appeal to me.
Also, there’s a modest chance that I might just write a post or three about smaller (non-DE) python projects I’ve been working on at home in my personal time. The purpose of this blog has been & most likely always will be about me writing about interesting situations that I find & solve, so why limit myself to just professional problems & solutions?! And let’s face it, everyone has a lot more personal time due to COVID-19, social distancing, stay at home orders, & working from home (for those that are working from home).
Be safe.