Building Meltano in Public
October 2019 in Review: +90% MoM MAUI, 7 Releases, 124 MRs Merged
Thank you for subscribing to hear about Meltano. As we progress in maturity, functionality, and community we are publishing more MRs, updates, and videos than ever before. Starting this month, we will recap our progress in growing Meltano’s user base, releasing product improvements, and increasing the cadence of contributions.
7 Releases in October
In October we released Meltano v1.0, which is thematically focused on delivering an efficient and predictable end-to-end user experience through the self-hosted Meltano UI. Since then, we have continued to iterate and encourage Meltano users to upgrade to the current version to take advantage of the latest improvements.
90% Month-over-Month Increase in Meltano Usage
Our goal is to grow Meltano MAUI (monthly active UI users) by 10% week-over-week, which is similar to the growth metric we are challenged to achieve while in Y Combinator. At the target pace (in red) we will have 1,000 MAUI by the end of 2019. While we experienced significant month-over-month growth of MAUI relative to September, we are still behind pace.
As you can see, we’re on track when it comes to command line users but we are lagging behind when it comes to converting them to the self-hosted Meltano user interface and achieving our 10% WoW target.
It is possible that we’re experiencing some degree of measurement error due to analytics opt-outs and some mistakes we made in setting up Google Analytics (which is why you see a big dip in August) that have now been fixed, but that does not change our goal.
To help improve MAUI and get these goals back on track, we added a marketer to the team in mid-October and look forward to seeing the benefits of his work as he continues to ramp up our efforts.
Contributions & Cadence
Thank you to all our contributors, both inside GitLab and across the broader open source community for Meltano. Learn more about how you can get involved by reading our Contributor Guide and join the discussion in the public Meltano Slack community.
MRs Merged by Month
Top Contributors by Opened MR Count
Ready to get started? It’s easier than ever with our DigitalOcean 1-Click App. Get $50 of DigitalOcean credit when you create a new account and install Meltano.
Consolidated CHANGELOG
New
#1208 Add description to
Plugindefinition and updateddiscovery.ymland UI to consume it#1195 Add temporary message in configuration communicating their global nature until “Profiles” are implemented
#1245 Add detailed information on the documentation about events tracked by Meltano when Anonymous Usage Data tracking is enabled.
#991 Add e2e tests for simple sqlite-carbon workflow
#1103 Add Intercom to Meltano.com to interact with our users in real-time
#1130 Add Tutorial for extracting data from Google Analytics and loading the extracted data to Postgres
#1168 Speedrun video added to home page and new release issue template
#1182 Add
nullable date inputs so optional dates aren’t incorrectly required in validation#1169 Meltano now generates the dbt documentation automatically
#1123 Add first-class “Submit Issue” CTA to help expedite resolution when a running job fails. Also updated the “Log” CTA in the Pipelines UI to reflect a failed state.
#1121 Add ability to configure listen address of Meltano and Airflow
#1022 Add “Autorun Query” toggle and persist the user’s choice across sessions
#1060 Auto advance to Job Log from Pipeline Schedule creation
#1111 Auto advance to Loader installation step when an extractor lacks entity selection
#1106 Add description metadata to the GitLab extractor’s Ultimate License configuration setting
#1057 Auto advance to Entity Selection when an extractor lacks configuration settings
#51 Update Google Analytics to track
appVersion, customprojectId, and to properly use the defaultclientId. The CLI also now usesclient_idto differentiate between a CLI client id (not versioned) and the project id (versioned).#1012 Add intelligent autofocus for improved UX in both Extractor and Loader configuration
#758 Update ‘meltano permissions’ to add –full-refresh command to revoke all privileges prior to granting
#1113 Update ‘meltano permissions’ to have the ability to find all schemas matching a partial name such as
snowplow_*#1114 Update ‘meltano permissions’ to include the OPERATE privilege for Snowflake warehouse
#1020 Update Command Line Tools documentation to reflect a standard format with opportunities for improvement in the future
#524 There is a new Plugins section on the site to contain all ecosystem related libraries (i.e., extractors, loaders, etc.)
#51 Add Google Analytics tracking acknowledgment in the UI
#926 Add step-by-step intructions for using the DigitalOcean one-click installer
#1076 Enable Log button in pipelines UI after route change or hard refresh if a matching log exists
#1067 Add Model landing page and update Analyze main navigation to a dropdown displaying the various analysis CTAs associated with each model
#1080 Add live chat support on Meltano.com website using Intercom.io
#950 Removed the Analyze connection configuration: Meltano will now infer connections out of each loader configuration.
#1002 Analyze UI now displays the Topic’s (analysis model’s) description text if applicable
#1032 Add ‘Model’ and ‘Notebook’ to main navigation to communicate that Meltano plans to empower users with modeling and notebooking functionality
#949 Add “Log” button and dedicated sub-UI for tracking an ELT run’s status more granularly
#932 Meltano can now be upgraded from the UI directly.
Changes
#1244 Add instructions on how to deactivate a virtual environment
#1126 Minor UI updates to improve clarity around Schedule step and Manual vs Orchestrated runs
#1210 Improved SQLite loader configuration context (name and description)
#1185 Remove majority of unimplemented placeholder UI buttons
#1166 Clarify in documentation that plugin configuration is stored in the
.meltanodirectory, which is in.gitignore.#1200 Link to new Getting Help documentation section instead of issue tracker where appropriate
#1227 Update Notebook
MainNavlink to jump to our Jupyter Notebook docs!1061 Update the Getting Started Guide and the Meltano.com documentation with the new UI and information about job logging and how to find the most recent run log of a pipeline.
#1213 Add VuePress use and benefits to documentation
#922 Document the importance of transformations and how to get started
#1167 Iterate on docs to improve readability and content updates
#1013 Toast initialization and analytics initialization cleanup
Compress meltano-logo.png
#1058 Updated the selected state of grouped buttons to fill vs. stroke. Updated the docs to reflect the reasoning to ensure consistency in Meltano’s UI visual language
#1068 Replace dogfooding term in docs to speedrun
#1101 Add new tour video to home page
#1101 Update design to improve readability and contrast
#1115 Update ‘meltano permissions’ to not require an identically named role for a given user
#1087 Fix
meltano selectnot seeding the database when run as the first command.#1090 Update the namespace for all plugins. Also the default schema used will go back to including the
tap_prefix to avoid conflicts with existing schemas (e.g. a localgitlaborsalesforceschema). This also fixestap-csvandtap-google-analyticsnot properly working after the latest Meltano release.#1047 Fix a bug where some configuration values were not redacted
#1069 Meltano will now use the schedule’s name to run incremental jobs
#926 Move manual DigitalOcean Droplet configuration instructions to advanced tutorials
Collapse Installation docs into a single section
#1045 Make it clear that ‘meltano add’ is not hanging while installing plugins
#1000 Update Getting Started guide with updated screenshots and content
#854 Charts now use pretty labels rather than the ID
#1011 Removed “Catch-up Date” in favor of default “Start Date” of extractor
#578 Remove support for
tap-zuora.#1002 Update
discovery.ymlwith explicitkind: passwordmetadata (we infer and set input types ofpasswordas a safeguard, but the explicit setting is preferred)#1049 Change default
target-sqlitedatabase name towarehouseto not conflict with system database#949 Update the way Meltano handles logs for ELT runs: Every elt run is logged in
.meltano/run/logs/{job_id}/elt_{timestamp}.log. That allows Meltano to keep logs for multiple, or even concurrent, elt runs with the samejob_id.#949 Update “Create Pipeline” redirect logic based on the previous route being ‘transforms’ (this is a UX win setting up the user with the sub-UI for the next logical step vs. requiring a manual “Create” click)
#1051 Automatically set SQLALCHEMY_DATABASE_URI config to system database URI
Fixes
#1173 Fix
sortBydrag-and-drop bug in Analyze by properly usingtryAutoRunvs.runQuery#1079
meltano eltwill now run in isolation under.meltano/run/elt#1204 move project creation steps out of the local installation section of the docs and into the Getting Started Guide
#782 Update timeframe label and fix timeframe attributes to properly display in the Result Table
#1172 Fix analytics issue related to app version
#1050 Fix a bug where the Job log would be created before the
transformare run.#1122
meltano eltwill now properly run when usingtarget-snowflake.#1159 Minor UI fixes (proper
MainNavModel icon active color during Analyze route match & “Run” auto query related cleanup) and...NameFromRouterefactor renaming cleanup#1120 Fix a concurrency bug causing
meltano selectto crash.#1086 Fix a concurrency issue when the
meltano.ymlfile was updated.#1112 Fix the “Run” button to improve UX by properly reflecting the running state for auto-running queries
#1023 Fix last vuex mutation warning with editable
localConfigurationclone approachPatch technicality due to PyPi limitation (v1 already existed from a publish mistake seven+ months ago) with needed changelog New/Changes/Fixes section headers
#1071 Fix
rehydratePollersso the UI reflects running jobs after a hard refresh or route change (this surfaced from the recent !963 change)#1075 Fix an issue where
meltano eltwould fail when a previous job was found#1004 Fix error when deselecting last attribute in Analyze
#1048 Fix various actions that should have been mutations and did minor code convention cleanup
#1063 Fix the “Explore” button link in Dashboards to properly account for the
namespace#1085 Fix Analyze model dropdown to properly reflect installed
models#1089 Properly re-initialize the Analyze page after a new analysis is selected during an existing analysis (this issue surfaced due to the recent Analyze dropdown CTAs addition which enables an analysis change during an existing one)
#1092 Fix async condition so the design store’s
defaultStateis properly applied before loading a new design viainitializeDesign
Breaks
None
To see a full list of all improvements to Meltano at anytime, make sure to check out our CHANGELOG.Our release cadence is every Monday around 9am Pacific Time. Learn more about our release process.