ETL and IIO Dashboard redesign release
Software patch update for Jetty
Error log summary explanation feature now enabled for all accounts
Temporary staging table updates to avoid race condition
Connections/Packages/schedules etc lists are now full-width on the screen filling all the space
Salesforce
Package validation on source
Destination - removed case sensitivity on Auto-Fill button. Will now auto-fill fields as long as the incoming fields alias is the same and not case sensitive.
Bulk API 2.0 for source - to support more streamlined and optimized workflow. In this version, we:
Introduce “Max records” - specify number of records per pages to prevent timeout
Disable “PK Chunking” option
MongoDB
Merge operation _id updates
Snowflake
Destination - load empty data as null option
HubSpot
Destination’s new system variable: _HUBSPOT_API_REQUEST_MAX_RETRIES
Paginations
Beauhurst
CitySpark
Google Analytics (GA4)
Added documentation on the Connection creation modal
Pipeline Engine Update
Released version auto update
Billing Mailer - new opt-in feature, which allows customers to get notified when they surpass certain percentages of their monthly volume limit.
Bulk selection for tables
You can type comma separate table names in the input instead of clicking checkboxes for each item to select given tables.
MySQL
Removed RELOAD from required permissions
Upgraded library for large data volume timeouts
Add backoff retry support on retryable errors
Added new logging metrics for MySQL and S3 write duration
Improved Job Error Debugging With Error Log Summaries
Receive AI-generated summary of job error log to pinpoint reason for failure
Job load balancing option on scheduler
Ability to reuse cluster with the least number of jobs running
Google Service Account Json Key validation fixes
Allowing universe_domain in JSON key
Auto-map workspace when importing package JSON
Automatically assigning the package to a workspace based on the name
DB Source Component
Updated where the values on Table mode isn’t being removed when switching to Query mode which causes jobs to fail
Create Major Version
Updated where it removes the last few changes when creating a major version
Create Connection Forms
Added a modal to refer to our documentation for setup instructions
Heroku Addon V3
Workspace auto-mapping on package import JSON
Mitigations from pentest in re-test
SOC 2 tests and evidence collection
Password limit on Rest API component connection (up to 128 characters in length)
Microsoft Excel
Facebook Marketing API update to v17
Netsuite JDBC upgrades to 8.10.136.0
Paginations
Survio
Validation server certificate update
Mitigations from pentest in re-test
SOC 2 tests and evidence collection
Github connector validation fixes
Parsing the latest position when a GTID set contains multiple positions.
Adds tinyint and smallint as datatypes for number like primary keys.
Version fetching for MariaDB > 11
GA360
Snowflake connector
Fixed permission validation bug when database, warehouse or schema name are the same
Shopify connector
Auto retry on SSLv3 connection error
MySQL connector
GTID support fixes and configurations net_read_timeout , net_write_timeout and wait_timeout are no longer mandatory for test connection
Show only warning if non-mandatory configurations are not set to recommended values for test connection
Updated password policy
Enforced new password policy on password update / signup (at least 1 lowercase, 1 uppercase, 1 number, and min. 12 characters)
SSO
Updated cases for handling camel-cased emails
Updated Workspaces view with pagination support
AWS marketplace signup and billing updates
RSS feed updates
API key retrieval without password update
Package variables saving updates
HubSpot Destination
Custom objects and properties supports
Google Spanner Data Boost
Netsuite
SOAP validation updates
Salesforce
Multi-reference column support
Paginations
LeadFeeder
TST Vandale
Add PIPELINE_ERRORED event tracking
New metric gauge which calculates the binlog lag based on the latest record received
Deployed SSH Tunnel feature
Updated session expiration to 12 hours
Add new pipeline events to logging feature
OAuth token leakage fix
Add the collection name to the error object to identify the collection to remove from sync temporarily
Release of hooks feature for notifications
Add a validation run before the force-start pipeline action on staff view
Make SSH tunnel as default and move it to first option
Add security response headers
MariaDB
10.6 support
Validation support for BINLOG MONITOR and REPLICA MONITOR grants for 10.5.8 and above
Convert array and bbject to string
Salesforce
String PK chunking support
GA4
Global variables on account level
We now support global variables creation at account level. It can be modified from Account Settings -> Global Variables
Package version on jobs list
We now show the particular version of the package that ran on the jobs list (and clicking on it views the particular version of the package read-only)
Global variables validation function
We are now validating the value or expression of global variables when saving. Please note that creating a global variable using values from other global variables as reference is currently not supported.
Released schema-importer OOM fixes
EKS upgrades
API
Security updates for bug bounty reports
BigQuery
Complex datatype support - load MAP datatype from source to BQ (as BQ’s JSON datatype)
GA4
Error logging with Property ID
Introduced backoff retry mechanism with maximum attempts system variable (_GA4_API_REQUEST_MAX_ATTEMPTS default: 3). This is to address issue when customer exceed their quota limits.
Rate limit workaround
Google Ads
Stability improvement. Customers might need to increase _ADWORDS_API_REQUEST_READ_TIMEOUT in order to adhere with these changes (especially customers with high number of properties/records).
Salesforce
Error output - empty file fixes
Column with multiple references - visible through source component’s schema and destination’s column mapping.
Bing Ads
Authentication updates
Add fields (breakdown properties)
Intermittent connection errors will now be auto retried without failing (ECONNRESET, Connection timed out, Timeout acquiring a connection)
Fixed issue on invite when email has capital letters
Create new pipeline event RATE_LIMIT_ERROR for connectors
Migrations, models and constants for SSH tunnel
Salesforce CDC
Fetch source schema in batches, set query batch size to 200, update column fetching, implement renew credentials on backend
Salesforce REST & BULK
Add KnowledgeArticle to tables not supported by BULK API
Show username when first creating the connector, renew and save credentials on the backend
Snowflake
Updated setup on dashboard - separated small scripts to ease customer source setup
Test connection now shows the encountered error (previously only generic message)
Google Sheets
OAuth request only needed scopes by each connector (instead of requesting all previous granted scopes)
Github
Rate-limit issues fix
Dynamic connection on Execute SQL Task component on Workflow. This feature is behind our existing feature flag “Dynamic connection”
Import JSON API - updated variables when creating new packages with dataflow JSON
Sandbox clusters idle time update
Documentation search improvements
Google Cloud Storage
V2 connection
Facebook Ads
Update to v16
GA4
Property name parsing updates
Google Ads
Missing fields fixes on Product Partition Report: campaign.shopping_setting.merchant_id, ad_group_criterion.listing_group.case_value.product_item_id.value
File Storage destination
Dynamic destination directory creation
File Storage Source
Disabled re-ordering of fields which caused data integrity issues
Salesforce Source
Disabled re-ordering of fields on Query mode which could cause data integrity issues
Paginations
Inkdesk
Increase timeout when fetching dynamic schema connector catalog
Sentry tracing updates
Added ToggleCase and ReplaceInField transforms in Staff API
Salesforce -
Secure Tunnel Creation - prevent the page from reloading when downloading tunnel template
Confirmation modal for Archive and Deactivate
Create subscription proration behavior setting support
HTTP Connector
Change JobId from random number to UUID
Salesforce Marketing Cloud
Incremental sync updates
Allow choosing between REST and Bulk
Salesforce
Move tables LoginAsEvent, ApiEvents and ListViewEvent to forced Full Sync list.
Move table AccountUserTerritory2View to unsupported table list.
SQL Server
Custom certificate updates
Github
Action parallel build run optimization
Snowflake
Role permission updates
Github Connector
Implement new tables: release_asset, branches, commit_files, commit_parents, commit_pull_request, commit_users_emails, deployments, deployment_statuses, issue_assignees, issue_labels, repository_teams, repository_topics, repositories, workflows, workflow_runs, workflow_pull_requests, collaborator_details
Add option to handle nested bookmarks (both on incremental sync and resync)
Redshift
Fallback to string on Geometry types
New last successful job timestamp added:
_WORKFLOW_PACKAGE_LAST_SUCCESSFUL_JOB_SUBMISSION_TIMESTAMP
Salesforce
Hard Delete operation - allowing records to bypass the recycle bin and immediately become available for deletion
Error output file updates
HubSpot
Support for Reverse ETL destination added
Support for Rest API source. Supports using HubSpot connection for authentication in Rest API source component.
Bing Ads
SDK upgrade to 13.0.15.1
Added new fields
GA4
Added new dimensions
Support empty rows
Google Ads
Update to AdsV13
New fields on Campaign report
Snowflake
New regions added:
North America
Snowflake GCP - US East (N. Virginia)
Snowflake Azure - South Central US (Texas)
Europe
Snowflake AWS - EU West (Paris)
Snowflake Azure - UK South (London)
Asia Pacific
Snowflake AWS - Asia Pacific (Osaka)
Snowflake Azure - Central India (Pune)
Middle East
Snowflake Azure - UAE North (Dubai)
South America
Snowflake AWS - South America (Sao Paulo)
Paginations
Klaviyo (a.klaviyo.com) (March)
Released pipeline event optimization
Google sign-in updates
BigQuery support on Data Observability
Secure tunnel updates
Existing tunnels shows all tunnels (inactive ones included) but active ones are on top of the list
Test tunnel connection button
Link to documentation on secure tunnels
Download bash script from tunnel creation
Select date range for pipeline usage dynamically in the pipeline details page based on the pipeline creation date
Update observability graph
Salesforce
Added type casting support for avro-based destinations
Support more retry error messages
Github
Added rate-limit error message handling
BigQuery destination
Parse years to forcefully have 4 digits.
Google Ads
Update API version to v13
MariaDB
Fix GTID issue
NetSuite
Update collections on yaml definition
Redshift
Additional types
httpTap
nodeType support
Salesforce Marketing optimizations
Support catalog S3 fetch and update (For dynamic connectors, reuse of catalog files)
Processing Engine
Fix NullPointerException during staging table deletion
GA4
Update for accounts fetching
Paginations
Google Drive
Source Tokens Autofill when creating from “Sources” tab.
Logging feature (Phase 1)
Show pipeline events on pipeline details page (up to 1000 events)
Highlight errored events
Hide full errors on errored pipeline stopped events
Salesforce
Change Salesforce types to align with other snowflake converters
Adobe Analytics
Add source attributes
SQLServer
Create mappings for data types.
Salesforce CDC
Redshift schema and avro converter update
Redshift add support for scale and precision
Salesforce update for column type changed error
Add support for scale and precision for floating value types
Google Ads
Updated catalog replication method
Facebook Ads
Update catalog.json for Full Sync tables
Update bufferTime to 7 days
Upgrade to v14.0 (singer-io repository)
Add sync_frequency recommendations
MySQL
UTF16 support
Google Search
Add bufferTime / custom PK feature support
Youtube Analytics
Add bufferTime / custom PK feature support
Updated yaml to fetch video details
TikTok Ads
Fix data mismatch issue
Add bufferTime / custom PK feature support
Shopify
Upgrade to v2022-07
Add rate-limit handling
Fix referenced schemas
LinkedIn Ads
Add sync_frequency recommendations
Frontend framework migration
Heroku add-on SSO updates
SAML SSO update for an issue during the callback phase of authentication
System upgrade (Ruby version)
Tiktok Ads
Launch as Reverse ETL destination
BigQuery
Staging table deletion on job failure
Salesforce
Added destination parallelism (both Bulk and SOAP API).
Adding a new field: Thread count to apply parallelism during loading data to the destination and reduce the job runtime.
This is very helpful for customers running a job with high number of records in small batch on production cluster with high number of nodes.
Fix discrepancy on error file output
LinkedIn creation/reconnect update
Paginations
Younium
Auth2AuthProvider - Automatic renew credentials function
Data Observability
Updated alert sync frequency update
Updated alert validation issues
Added date range filter
Updated date format from MON-YY to MON-YYYY (Pipeline and Observability)
Clamp precision and scale to 38 and 37 in Redshift
BigQuery
Add support for null insert on array columns
Shopify
Metafields collection data mismatch
Sendgrid
Updated YAML definitions