Meltano v1.20.1 is keeping your Valentine's Day bug free!

It's been a bug-squashing week all over the globe with team Meltano, and we're even more excited as we keep working on better default dashboards and interfaces.

We squashed a number of bugs this week, and continue tweaking the interface to refine the UI and make Meltano simple and easy to use.

The team is also working hard to build a set of default dashboards for each new data connector, so it’s easy for anyone to have a great view of their business the moment they connect to a new data source.


Did You Know?

Meltano is completely open source - which means you can go look at the source code AND the work in progress through GitLab issues at any time!

Meltano on GitLab


New

  • #1650 create TOS page and add TOS link to website footer

Changes

  • #1681 Update transform during pipeline save to conditionally set skip vs. run to prevent wasted cycles for extractors that lack transformations

  • #1696 Update dashboards list to be alphabetically sorted

  • #1710 Hide tap-fastly in UI

Fixes

  • #1696 Fix duplicate chart renders when dashboard is loaded

  • #1696 Fix “Add to Dashboards” button when loading an existing report (additionally updated disabled button states)

  • #1711 Disable fields of all kinds when a plugin setting is protected or set in env or meltano.yml

  • #1712 Fix lock icon tooltip message on plugin settings that were set in env or meltano.yml

  • #1677 Properly represent values of boolean settings that were set using environment variables in UI

Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.

Try Meltano Now

v1.19.2 Introduces Embeddable Reports!

We’re thrilled to announce that the ability to embed reports in other webpages is now available in Meltano!

The entire team is excited about this new functionality, and this first version is the beginning of an exciting path forward for sharing and embedding Meltano reports anywhere.

Embedding a report anywhere an iframe can render is simple – just click on the “Embed” drop down menu on any Meltano report, and then “Copy Snippet” to add the code to your clipboard.

<iframe src='https://meltano.meltanodata.com/-/embed/zE8NFH9P3YwFuX2QNpA4WycsuwePpUEHj9CW8ubIgwE' style='margin: 0; padding: 0; border: 0; font-size: 100%; font: inherit; vertical-align: baseline; min-width: 500px; min-height: 400px;' />

The end result is a simple, beautiful graph that renders in any browser and automatically updates as your data updates.

The entire team is excited to hear what you think about embeddable graphs and reports, and as always, we’d love to hear about how we can improve Meltano.

Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.

Try Meltano Now!

Building Default Dashboards

The Meltano team continues to work on making a useful set of default dashboards available as soon as you setup any of the available data extractors.

Dashboards Ready Out of the Box

The Meltano team continues to work on making the Meltano experience positive as soon as an account is created. This means making sure that each data connector has a solid set of default dashboards that are insightful and actionable, and which don’t take a ton of time to setup.

This week the team updated the default dashboards for Google Analytics, and work continues on building a great set of dashboards for our new Facebook Ads extractor.


Try Meltano for Free

As part of our effort to work closer with users, we’ll setup a hosted Meltano instance for free, and help you get connected to the services most important to your business.

Try Meltano for Free


What shipped on January 27th and 29th

Contributions & Cadence

Thank you to all our contributors, both inside GitLab and across the broader open source community for Meltano. Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.

New

  • #1462 User will be able to reorder dashboard reports

  • #1482 Add future grants and revocations for schemas, tables, and views for roles in the meltano permissions command

  • #1376 Add last updated date to reports

  • #1409 Add data start date to Analysis page

  • #1241 Add dashboard plugin type to enable bundling curated reports and dashboards for data sources

  • #1241 Add --include-related flag to meltano add and meltano install to automatically install related plugins based on namespace

  • #1241 Add default dashboard and reports for Google Analytics

Changes

  • #1481 Add table and view revocations for roles in the meltano permissions command

  • #1459 Users can no longer install tap-carbon-intensity from the UI

  • #1625 Update docs on meltano.com to only include extractors and loaders provided in the hosted version of Meltano.

  • #1590 Add additional targets to dbt clean

Fixes

  • #1600 Fix tooltip for Data Source “Connect” buttons

  • #1605 Fix an infinite loop causing extraneous API calls to the configuration endpoint

  • #1561 Fix onFocusInput() to properly focus-and-auto-scroll to <input type='file'>s in the data source docs UI

  • #1561 Fix <input type='file'> styling to better accommodate flexible widths

  • #1618 Fix an issue where an expired session would not redirect to the Login page

  • #1630 Fix an integrations setup bug that prevented subsequent pipelines to be created unless a full page refresh occurred

Facebook Ads Data & Product Improvements

In the third week of January 2020, we've released the first version of a Facebook Ads Extractor and made numerous improvements to Meltano and its interface.

At the end of 2019, the Meltano team started working closely with startup founders to learn about their needs - and confirmed that it was still too hard for most users to get started. Getting started was difficult due to a lack of useful data extractors and an interface that allowed for more configuration than the average user needed.

Facebook Ads Data Extractor

In the last two weeks, Meltano has been updated to include a Facebook Ads data extractor, allowing Meltano users to start working with Facebook Ads data via the Facebook Marketing API. This integration is still being iterated on and improved, but is available for teams interested in dashboarding ads performance data from Facebook.

If you don’t already have a hosted Meltano account, click the button below to get started and we’ll work with you to setup an account and get Facebook Ads configured.

Sign Up for Facebook Ads Dashboards


Simplifying the Meltano UI

In addition to shipping new features and extractors, the Meltano team has been working on simplifying the user interface across the board - from removing views with no configurable options to renaming confusing links. These are small and spread far and wide, but they should be evident as you continue to use Meltano.


Try Meltano for Free

As part of our effort to work closer with users, we’ll setup a hosted Meltano instance for free, and help you get connected to the services most important to your business.

Try Meltano for Free


What shipped on January 20th and 23rd

Contributions & Cadence

Thank you to all our contributors, both inside GitLab and across the broader open source community for Meltano. Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.

New

Changes

  • #1541 Revert tap-csv's kind: file to text input for csv_files_definition as we don't fully support tap-csv via the UI with single (definition json) and multiple (csv files) file uploading

  • #1477 Add a read-only mode to Meltano to disable all modifications from the UI

  • #1510 Remove breadcrumbs (not currently useful)

  • #1589 Add dbt-specific files to a .gitignore

  • #1402 Onboarding redesign to minimize steps and friction ('Extractors' as 'Data Sources', pipelines are secondary to 'Data Source' integrations, and removed loader, transform, and pipeline name as editable in favor of preselected values in accordance with our hosted solution)

  • #1402 Local development now requires .env to connect a target-postgres loader (docs update to follow in #1586 )

Fixes

  • #1573 Fix docs shouldShowNavbar conditional and improve query string embed=true parsing

  • #1579 Make color contrast for CTA buttons accessible

Meltano December 2019 in Review

In October 2019, we released Meltano 1.0 and delivered an end-to-end experience for self-hosted users. Following the launch, adoption numbers and user feedback confirmed what we suspected, it was still too hard for most users to get started — and it became clear we were unlikely to hit our 1,000 MAUI goal for the year.

In November 2019, we clarified our target persona to focus on startup founders and released Meltano as a free hosted product with great success. Read the November 2019 recap here.

Create Your Free Meltano Account


December 2019 in Review

December was all about figuring out how to set up a large number of Meltano instance, get users onboarded, and gathering feedback during discovery calls. Since the launch of our hosted offering on November 26, 2019 we have created 169 hosted Meltano instances.

With the barrier of installation out of the way, we were able to observe users trying Meltano firsthand and came away with a clear set of priorities for improving their experience. The most common user stories coming up during our discovery and onboarding calls are:

  • Reduce friction in connecting data sources

  • Add key data sources requested by Meltano users

  • Provide configuration-free default dashboards

  • Offer the ability to join data across multiple sources into a single report

Right now, these challenges are the main barriers to deeper Meltano adoption and the team is hard at work to resolve them so we can discover the next points of friction. It’s turtles all the way down.

1,000 MAUI By EOY 2019

Our goal was to grow “MAUI” (monthly active UI users) by 10% week-over-week from where it was when I joined the team on Feb. 25, 2019 (15 users). The number was arbitrary, but we needed to start somewhere, and this metric is a great one for assessing product/market fit.

The fact that we missed the goal is disappointing, but that’s not actually the point of the metric. The point is that we tried a lot of things, and found it much more difficult to achieve than expected. The real value is in figuring out the answer to “why is this so hard?” through iteration.

And so we placed new bets on what might be the best way to move the needle, while at the same time respecting some constraints like a very strong aversion to offering a hosted SaaS solution. Ultimately, after trying many other approaches, we had to capitulate and try what we had been avoiding and offer Meltano as a service where we handle hosting and installation.

We saw a 30% increase in MAUI in a single week with this launch, but were initially constrained by infrastructure challenges since we were hand-cranking instance creation. It felt good to have real “startup problems” driven by demand, and now we have the instance creation process streamlined.

Create Free Meltano Account


Resetting the MAUI Goal for 2020

As we got further and further away from the MAUI pace goal it began to lose meaning, since it felt impossible to catch up. It’s important to have quantifiable goals to iterate toward, but at the same time they should be achievable if challenging. In 2019 this metric did exactly what it needed to do, raising a constant red flag about product/market fit. Despite driving lots of organic traffic and consistently having thousands of new visitors to our website each month, product adoption just wasn’t happening.

For 2020 we have reset the MAUI goal to baseline on our MAUI for the first week of January and then grow 10% week-over-week thereafter. Ideally we will reach 1,000 MAUI in mid-May or earlier, and this should be a true indicator of reaching some early product/market fit:

Retention: The #1 Driver of MAUI Growth

In addition to MAUI, there is a second KPI we will be tracking and reporting on in these monthly recaps going forward: retention. Ultimately, we need to be offering a product that people want to come back and use again and again. MAUI doesn’t mean much if its all new user adoption and no return users, that’s what we call a “leaky bucket”.

Focusing on retention in our product decisions will enable us to make improvements that we can regularly communicate to our users, and having a less leaky bucket will help us justify larger investment in top-of-funnel marketing efforts.


What Shipped in December 2019

Contributions & Cadence

Thank you to all our contributors, both inside GitLab and across the broader open source community for Meltano. Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.

MRs Merged by Month

Consolidated CHANGELOG

New

  • marketing#103 Add Google Site Verification token to site

  • #1346 Add new tutorial for using FileZilla with a Meltano project

  • #1292 Add guide for setting up Meltano projects on meltanodata.com

  • #1343 Add current Meltano version to main navigation

  • #1361 Add kind: hidden to discovery.yml so certain connector settings can validate with a default value but remain hidden from the user for improved UX

  • #1351 Add “Create Meltano Account” promo for meltano.meltanodata.com

  • #1055 Add “Disable” button to Tracking Acknowledgment toast so user’s can opt-out from the UI

  • #1408 Add “Last Run” context to each pipeline

  • #1408 Add “Started At”, “Ended At”, and “Elapsed” to Job Log modal

  • #1390 Display of extractors and loaders can now be configured through the hidden property in discovery.yml

  • #1374 Add role revocation for users and roles in the meltano permissions command

  • #1377 Document cleanup steps after MeltanoData testing

  • #1438 Add documentation for DNS spoofing error

  • #1436 Add video walkthrough on how to setup Google Analytics so that the Meltano Extractor can be able to access the Google APIs and the Google Analytics data.

  • #1437 Users can now share their dashboards with an automatically generated email

  • #1269 Add kind: file so single file uploads can be used with extractors (tap-google-analytics‘s key_file_location is the first user)

  • #1494 Add LIKE options to Analyze Filter UI so users better understand what filtering patterns are available

  • #1461 Display toasted notification for report adding to dashboard

  • #1419 Add ability to edit and delete dashboards

  • #1411 Add download log button to Job Log Modal

Changes

  • #1341 Various discovery.yml and connector configuration UI updates to improve UX.

  • #1341 Updated documentation to communicate the various optional settings of a connector

  • #1358 Update MeltanoData guide with maintenance and debugging instructions

  • #1337 Add CTA to installations for free hosted dashboards

  • #1365 Add process for deleting meltanodata instances

  • #1340 Update connector settings UI to communicate the required status of each setting

  • #1357 Update LogModal Analyze CTAs so Analyze can preselect the correct loader for a given analysis

  • #1373 Update MeltanoData deletion process with 1Password

  • #1373 Update Analyze dropdown as scrollable to better display model CTAs (scrollable dropdown vs. scrolling entire page)

  • #1371 Provide more specific instructions for Google Analytics configuration

  • #1381 Update the default directory for client_secrets.json for the Google Analytics Extractor to be located under the extract/ directory and not the project’s root.

  • #1345 Update the documentation for the Salesforce Extractor to contain additional information on Security Tokens

  • #1383 Add CTA for hosted solution signup to navigation

  • #1389 Temporary Profiles feature removal (conditionally removed if 2+ profiles not already created so existing users can continue using multiple profiles if created)

  • #1373 Update MeltanoData deletion process with 1Password

  • #1398 Update default Transform from “Skip” to “Run”

  • #1406 Update Analyze Query section CSS for improved UX (visually improved organization and scanability)

  • #1417 Update SCSS variable usage in components for SSOT styling

  • #1408 Updated date and time displays to be human-friendly (moment.js)

  • #1268 Remove Transform step from UI (Create Schedule still allows choosing “Skip” or “Only” but will intelligently default to “Skip” or “Run”)

  • #1435 Change “Model” to “Analyze” so the Pipeline CTA is actionable and less abstract

  • #1432 Changed “Close” to “Back” in Log Modal to help mitigate “Am I ending the pipeline?” concerns

  • #1350 Switch to all lower case for Snowflake permission comparisons in the meltano permissions command

  • #1449 Hide the Marketo Extractor form Meltano UI

  • #1397 Optimize workflow for MeltanoData setup

  • #1423 Update sidebar and docs to include Ansible

  • #1403 Remove “Orchestrate”, “Model”, and “Notebook” from the main navigation until each respective UI is more useful (the /orchestrate and /model routes still exist)

  • #1476 Add database and warehouse revocations for roles in the meltano permissions command

  • #1473 Update Release issue template to recent guidelines

  • #1422 Update start date field to have a recommendation

  • #1431 Add “pipeline will still run if modal is closed” message in the Job Log Modal

  • #1399 Log Modal now has a prompt to explain potential factors in required time for pipelines to complete

  • #1464 Remove the “only” option for transforms in Create Pipeline form

  • #1466 Filters now have clear language and indiciation that they use AND for chaining

  • #1399 Log Modal now has a prompt to explain potential factors in required time for pipelines to complete

  • #1433 Remove /orchestrate route and thus the Airflow iframe as this is overkill for our current target users

  • #1311 Remove unused meltano/meltano/runner docker image

  • #1502 Update configuration file uploads to occur on save vs. file picker completion

Fixes

  • #1334 Fix automatic population of airflow.cfg after installation

  • #1344 Fix an ELT automatic discovery error when running Meltano on Python3.6

  • #1355 Upgrade version of discovery.yml so that not upgraded Meltano instances with a pre v1.9.0 Meltano version do not break.

  • #1364 Fix instructions to SSH into MeltanoData.com instance

  • #1373 Fix formatting on custom containers in MeltanoData guide

  • #1379 Fix an issue with Airflow scheduling too many jobs.

  • #1386 Fix connector modal clipping issue where small browser heights prevented accessing the “Save” area

  • #1401 Fix double instance of self hosted CTA on desktop sites

  • #1439 Fix relative elapsed time since last run time display in the Pipelines UI

  • #1441 Fix auto advance to “Create Pipeline” when coming from “Load” step (previously “Transform” step, but this has been removed from the UI)

  • #1440 Allow installed plugins to appear in UI even if hidden in configuration

  • #1447 Various fixes around loading and reloading reports to mitigate false positive sqlErrorMessage conditions

  • #1443 Fix tooltip clipping in modals

  • #1500 Fix meltano install not running the migrations.

  • #1434 Fix Analyze CTAs to only enable if at least one related pipeline has succeeded

  • #1447 Various fixes around loading and reloading reports to mitigate false positive sqlErrorMessage conditions

  • #1509 Allow plugin profile config to be set through meltano.yml

  • #1518 Fix bug that caused all text fields to show up as required in configuration modals

  • #1446 Fix bug that could result in a broken report when the report URL was manually modified

  • #1411 Fix bug when reading too large a job log file


To see a full list of all improvements to Meltano at any time, make sure to check out our CHANGELOG. Our release cadence is every Monday around 9am Pacific Time. Learn more about our release process.

Loading more posts…