Dec and Jan DataOps Recap with Meltano
We begin the new year refreshed and more compelled than ever to build Meltano into the DataOps OS tool we know it can be and the tool the community deserves.
Our outlook for the new year is one of excitement and energy. We have come back refreshed and more compelled than ever to build Meltano into the DataOps OS tool we know it can be and the tool the community deserves. We are more convinced than ever a layer of control for your data stack with open source and DevOps best practices sprinkled in is the optimal stack management solution.
We decided to combine our Dec and Jan newsletters because Meltano closed our virtual office for two weeks over the holidays to give everyone a much-needed break and dedicated time to spend with their loved ones. So here you go… rapid-fire 🔥 Check out the latest from the past two months.
Releases
Meltano (v1.89.0 to v1.92.0)
Check out the changelog for all of the details and take a look at the highlights below.
Newness :sparkles:
#3060 Set splitio variant as the default tap-fastly extractor.
#3076 Set transferwise as the default target-postgres loader.
#2967 Set meltanolabs as the default tap-google-analytics extractor.
#2838 Adds the meltano test command which defines first-class test and validation capabilities for Meltano plugins and projects. 🚀 Changes
#2967 Set the meltanolabs variant as the default tap-google-analytics extractor.
#3085, #3111 Upgrade target-bigquery default variant adswerve to 0.11.3.
#3113 Pin tag v1.4.27 of tap-salesforce default variant Meltano.
#3119 Bump Meltano variant of tap-salesforce to tag v1.5.0.
#3153 Published JSON Schemas for meltano.yml and discovery.yml. Thanks to @Zachary Wynegar for the initial MR 1804 and inspiration this!
#3001 Always print log advice on ELT failure.
#3176 Adds support for dbt v1.0. Transform packages for dbt now define their own dbt version constraints.
#3181 Add an empty data folder called output / for new Meltano projects
SDK Releases (v0.3.15 to v4.0)
Visit the changelog for more details
#287 Tap SDK: Add configurable timeout for HTTP requests -- Thanks, @josh Lloyd - (Widen)!
#293 Tap and Target SDK: Adds support for Python 3.10.
!223 Resolve lint errors when ArrayType is used to wrap other types -- Thanks, @David Wallace!
#284 Thanks, @Fred Reimer for Tap SDK: Fixed missing schema during development causing sync to fail.
#257 Inline Mapper SDK: Support for creation of inline mapper plugins.
#238 Tap and Target Cookiecutter: Support editable pip installs (pip install -e .) with PEP 660.
#241 Always sync one record per stream when invoking with --test or --test=all.
#281 Add --test=schema option to emit tap SCHEMA messages only- Thanks, @laurent!
#74 Add support for SQL Taps.
#263 Add support for SQL Targets.
#166 Added Licence tracking to SDK GitLab Project.
MeltanoHub Releases
hotglue coming in hot :fire: with all the taps
@Luis Atala for building tap-zendesk-sell and adding it to MeltanoHub!
Thanks Jonas Kalderstam tap-hubspot variant
Thanks @Daniel Luftspring for adding tap-hellobaton
To see what’s coming up next, take a look at our roadmap.
Community Wins
The Meltano community is a passionate bunch, and we’re so lucky to collaborate with them. Here are a few of our favorite moments from December and January.
In the news
Extract csv data and load it to PostgreSQL using Meltano ELT
The Meltano Open Source Community Weighs In on the Future of Data
Great overview of the Data Dog platform and where it is headed
All about truth seeking in the data profession with dbt and Taylor Murphy
As a data professional, it’s impossible to completely stand outside your own context, and your affiliations make it hard to do your own job (tell the truth).
Great writeup about a hard problem to articulate, but it's super important. “The lines are the hidden and often primary driver of organizational tech debt.” Most people don't see that, and it leads to (imo) 10's to 100's of people being hired to solve just the integration problem instead of focusing on growing the business. - Derek Visch
Chefs kiss for this post from Stephen Bailey about Data Platform Data Platform being the most meta
Massively good news for companies using Python. They will now have:
🔒 Private packages
🏢 Org accounts
✨ Namespaces This will help, simplify, and improve a lot of things for organizations, preventing/improving a lot of possible security issues, while also supporting PSF.
Hot Takes with Data Bae
Podcasts
The Data Stack Show, Eric and Kostas chat with Douwe Maan, the CEO of Meltano. During the episode, Douwe discusses data tooling, open-sourcing, and data houses.
Culture and company building
As many of you may have noticed, we did a web and documentation refresh. We are proud to share our redesign which has been updated to better help you find the info you need quickly. It is now easier than ever to get started with Meltano and DataOps.
We are hiring!
Want to work for a rapidly growing startup changing the DataOps landscape with great perks? Check out all of the awesome open roles. 👐
Upcoming
Register for this year’s first Tap-fest Feb 17-24, 2022. We have some sweet prizes for those of you that submit a tap this time around. ✨