New Features

📊 Modeling with Microsoft Fabric​

We have added support for Microsoft Fabric for Modeling and associated Transform Tasks. When creating a new Model, you now have the option to choose Fabric or Databricks. When creating a model, simply choose your preferred environment through the new Application Type selector. This selection determines where your Model notebook should live as well as which compute will be used to run any Transform Tasks associated with that Model.
Learn More →

🔜 Keep an eye out for future releases where we will be adding Fabric support for Acquire and Publish Tasks as well.

↔️ Guided Data Migrations​

The Data Migration experience has been redesigned into a guided, step-by-step workflow that makes it easier to move through migrations with confidence. Each step provides clearer instructions and contextual guidance, helping reduce guesswork and manual effort throughout the process.

You can now also view Connector Migrations directly in your migration history, giving you better visibility into past activity and making it easier to track and validate changes over time. Additionally, Fivetran Acquire Tasks no longer require Advanced Options to be enabled.
Learn More →

🎯 Task-Level Model Entity Selection​

Transform Tasks now allow you to enable or disable specific entities from the associated model. Instead of running an entire model every time, you can tailor each task to only include the entities you need. This makes it easier to optimize schedules and compute, reduce unnecessary processing, and better align runs to business needs.
Learn More →

📂 FileDrop Connector​

The Dropbox connector has been renamed and reimagined as FileDrop, a more flexible, event-based ingestion experience. FileDrop automatically ingests new files as they are added to your storage location based on the rules you configure. This enables a more reactive, automated ingestion pattern without manual intervention.

Because FileDrop is event-based, there is no need to created associated Acquire Tasks. Additionally, there is the ability to add an Extend Task to run prior to ingestion to do pre-processing to your files, reach out to your managed services team for assistance in setting up pre-processing Extend Tasks.
Learn More →

🔌 Multiple Integration Runtime Support for Publish​

Publish Connectors now support multiple Integration Runtimes for hybrid environments. This enhancement makes it easier to support hybrid and multi-environment architectures by allowing a single connection to operate across different runtimes without additional configuration overhead.
Learn More →

✨ Quality of Life Improvements​

Connect​

• The Metadata Catalog now supports integer data types for watermark column selection.
• Load Strategy is now always shown as an option in the Metadata Catalog.
• The field details pop out in the Metadata Catalog now includes a search box.
• The Azure Data Lake Storage connector has added an option for service principal authentication.
• If you have Advanced Option enabled, Key Vault ID is now display the connector details page.

Tasks and Workflows​

• Description maximum length has been increased to 1000 characters.
• Scheduling Enhancements
  • Empower advanced scheduling system extended to Acquire Tasks with a Fivetran connector.
  • Additional schedule validations to surface invalid inputs before saving.
  • Schedule time zone selectors now include a search box.

Model​

• Automated Entity Sync defaults new columns to type “Attribute”.
• An alert banner has been added to Automated Entity Sync to surface errors.
• New model entities are now set to Active by default.
• Datatype has been added as a field to the Entity Column tab to give context when setting default missing values.

Staging to Gold Merge​

• Support for merging data into the Gold layer directly from staging notebooks.

🔃 Power BI Table Level Refresh You can now control exactly which tables in a Power BI dataset are refreshed during a Publish Task. Instead of refreshing the entire dataset every time. Navigate to the configuration tab of a Publish Task to select only the tables that have changed and adjust refresh frequency as needed.

Power BI is also now supported as a Connector type, allowing connection details to be defined once and reused across tasks. This reduces execution time, lowers compute costs, and simplifies configuration.

Learn More → ⏰ Advanced Scheduling Scheduling with Empower now gives you significantly more control over when Workflows and Tasks run. In addition to standard interval-based schedules, you can now define more precise execution patterns that better reflect real business operations.

With Advanced Scheduling, you can now also:

Choose to run on a contunious basis where the next run will immediately kick off after the previous run completed. Define specific run times and limit execution to a configurable time window, such as limiting runs to after business hours. Add exclusion rules to skip specific dates or recurring periods, such as holidays or month-end blackout windows. Control concurrency behavior to prevent overlapping runs when a previous execution is still in progress. These enhancements allow teams to reduce unnecessary runs, avoid peak-hour processing, and eliminate manual scheduling workarounds, resulting in more predictable, efficient, and business-aligned automation.

Learn More → 🔌 Multiple IR Hybrid Environment Support Empower now supports mapping multiple Integration Runtimes to a single logical source type across different environments, including on‑prem, cloud, hybrid, and multiple self‑hosted IRs. This removes previous deployment limitations for hybrid architectures and enables more flexible, enterprise-ready connectivity across customer environments.

Learn More → ✨ Quality of Life Improvements ⚙️ Cluster Configuration for Publish Publish Tasks that are "Publish to Target" now allow you to customize the Spark cluster configuration used during execution, including cluster type and node sizing. This gives teams better control over performance, scalability, and cost. Learn More →

⚠️ Data Migration Error Visibility When a data migration fails, you can now view the full error message directly in the UI. This improves troubleshooting and reduces guesswork during environment promotion.

🧩 Extend Tasks – External ADF Support Extend Tasks can now execute pipelines in external Azure Data Factory instances, rather than being limited to the Empower-managed ADF. This enables tighter integration with existing customer ADF investments and broader orchestration scenarios. Learn More →

Itemized Changelog Enhancements & Features Advanced Scheduling Enhancements: Introduced a new scheduling foundation with support for advanced intervals, validation, policy evaluation, guard integration, and expanded schedule options across Tasks and Workflows. Power BI Table Level Refresh: Added support for refreshing individual tables during Publish Tasks and introduced Power BI as a first-class Connector. Multiple Integration Runtime Hybrid Support: Improved support for multiple IRs and SHIRs across hybrid environments, including corrected naming conventions and routing. Publish Cluster Configuration: Added the ability to configure Spark cluster settings for Publish Tasks, including integration with ADF. Extend Tasks – External ADF Support: Added a configuration panel to run pipelines in external Azure Data Factory instances and improved Extend Task configuration and metric routing. Data Migration Enhancements: Added a View Error modal for failed migrations and improved safeguards to prevent concurrent migrations. Dashboard Performance Improvements: Optimized dashboard object charts to better support large datasets. Ingestion Performance Optimizations: Improved Spark ingestion efficiency with configurable schema post-processing, optional DataFrame caching, scoped deduplication, and asynchronous telemetry logging. API Telemetry Improvements: Added asynchronous telemetry logging to improve API performance and observability. Connector Enhancements: Added Power BI Refresh Connection support. Updated SAP SAC connector. Added rich text fields to source types. Hid Batch Extraction options for Custom Sources. Task & Workflow Usability Improvements: Added filters to additional task columns. Improved task page loading behavior. Fixed long task name truncation in Overview pages. Modeling & Metadata Updates: Moved search bar on Entity and Entity Column pages. Updated Transform description column to align with model metadata. Added Power BI metadata to the Metadata Catalog. Data API Improvements: Optimized the /object-ingest-delta endpoint and added org metadata to ASB payloads. Notifications Reliability Improvements: Improved handling of invalid payloads from notification subscribers. Bugfixes Fixed Extend Task failures for Fabric Data Pipelines and Fabric Notebooks when Self-hosted Integration Runtime was offline. Resolved ADF and SHIR naming convention issues related to Multiple IR support. Fixed regression where Fivetran ADLS workflows appeared as failed despite successful runs. Addressed extraction failures for SAP Concur connectors. Fixed issue where ADF Extend Tasks failed when executed within Workflows. Corrected model list sorting to ensure most recently modified models appear first. Resolved errors when saving or testing SHIR and ADF connectors. Fixed Power BI connector metadata refresh failures resulting in 422 errors. Addressed scheduler errors including inconsistent schedule summaries, 400 errors when saving schedules, and mismatches between overview and configuration views. Fixed Transform Run Blade status bar display issues.

Data Migration Asset Expansion​

Error Message Banner​

Notifications Nudge​

Itemized Changelog​

Enhancements & Features​

• Extend Tasks Expansion: Added support for Azure Data Factory Pipelines, Databricks Workflows, Fabric Notebooks, Fabric Pipelines, Logic Apps, and Azure Functions.
• Extend Tasks UI Updates: Removed Advanced Options from all Extend types except Databricks Notebook; updated form fields for ADF and Logic Apps; added secret management and flow model field updates; improved metric card routing.
• Config Migration Enhancements: Added Workflow Migration to UI; enabled migration for Fivetran Tasks, Extend Tasks, and Publish Tasks; updated download endpoint; introduced Advanced Options toggle for Workflow/Fivetran migration; UI refinements.
• Workflow Enhancements: Redesigned Workflow Configuration page; added task descriptions to Workflow returns; displayed descriptions across Acquire, Publish, and Workflow modules; minor UI updates.
• Task Enhancements: Updated Run Preview Modal to Blade layout; added Start and End Date fields to Schedule Card.
• Modeling Enhancements: Added Overview Page Cards; introduced Error Message Banner for real-time validation in editable grid views; updated Entity Sync button location; expanded Global Search to include Entity and Entity Column pages.
• Notifications Improvements: Introduced Notifications Nudge to remind users to enable email notifications; quality-of-life updates for notification settings.
• Dashboard Updates: Improved Failure Banner display; corrected banner logic for Object Acquisition Failures.
• Administration Updates: Updated Administration routes for better navigation.
• Spark & Ingestion Optimizations: Added asynchronous telemetry logging; avoided repeated toDF() calls on DeltaTable ingestion path; implemented conditional silver view recreation to skip unchanged views.
• Connector Enhancements: Refactored ADLS connector to ingest data from files; optimized ADF pipelines; enabled Dynamics CE Publish as a Publish Task instead of Extend Task.

Bugfixes​

• Fixed metadata extraction status display for connections.
• Resolved issue where notification service couldn’t start if FDS was unavailable.
• Fixed Extend Task failures for Fabric Notebook when run through Workflow.
• Addressed logout issue when environment was changed.
• Corrected dashboard banner showing no unaddressed errors despite existing failures.
• Fixed extraction failures for IBM DB2 and Azure MariaDB connectors.
• Resolved ingestion errors caused by special characters in ADF connectors.
• Fixed connectivity and routing issues across modules.
• Corrected UI inconsistencies and minor layout bugs.

Itemized Changelog

Enhancements & Features​

• Multiple IR Assignment: Support for assigning multiple Integration Runtimes.
• Databricks Workflow Custom Flow: Enhanced custom flow capabilities for Databricks Workflows.
• Quick Help Page Updates: Improved page-specific guidance through Quick Help.
• Grid Views Column Resizing: Introduced functionality for resizing columns in grid views.
• Workflows Enhancements:
  • Added a Workflow Overview page.
  • Introduced a Workflow Schedule page.
  • Updated the Main Workflow and Workflow Runs pages.
  • Added Run Filters to Workflow Run Tables.
• Model Enhancements:
  • Added Sync Entity Metadata feature for easier column management.
  • Integrated a button for adding Sync Metadata.
  • Enabled Archive Filter and added a Disable Model Confirmation Modal.
  • Added a Filter for Last Modified Column.
• Tasks Enhancements:
  • Added tooltips to last runs cell.
  • Updated routing for Start Time Overview Link.
  • Added lastFlowRunId to GET flow returns.
  • Updated UI to catch environment errors on tab pages.
  • Enhanced scheduling and Get All Last Runs Array.
• Connect Enhancements:
  • Added Archive Tag for Connectors.
• Settings Update:
  • Updated "Settings" to "Administration" for better clarity.
• Documentation:
  • Updated IFrame Docs Linking for improved integration.
• Dashboard Enhancements:
  • Added filters to missing AG Grid columns.
• CoreApi Notification Service Enhancements: Improved notification mechanics.

Bugfixes​

• Fixed toast message visibility issues with the Need Help page open.
• Resolved modal overlay issues with the Need Help iFrame.
• Fixed workflow name reporting in failure notifications.
• Addressed Core API 400 error when posting to /fivetran/sources/.
• Corrected layout bugs involving white space in Advanced Options and Entity Dependency Graph crashes.
• Addressed notifications visual issues in email templates.
• Fixed ingestion failures due to obfuscation in Netsuite.
• Resolved expiration scheduling bugs.
• Fixed UI discrepancies and saving issues across connectors.
• Standardized time responses for analytics endpoints.
• Addressed extraction failures due to special characters in ADF connectors.
• Corrected authentication failures in Capax QA environment.
• Resolved data migration issue allowing multiple task types when only one should be migrated at a time.
• Improved workflow run details blade and column sizing in Workflow Run Preview Modal.
• Corrected various connectivity-related discrepancies between Neo and Meso environments.

Itemized Changelog

Enhancements & New Features​

• Expanded CLMA connector capabilities.
• Improved column filtering in Metadata Catalog.
• Added grid filters to model page tables for enhanced data management.
• Introduced autocomplete for Entity ID field in Entity Column settings.
• Optimized Model Builder processing across Spark, ADF, and StateDB.
• Updated the default schedule form view in Tasks for more intuitive setup.
• Enhanced loading behavior for Extend and Transform tasks.
• Implemented new grid styles throughout the UI for consistency.
• Updated filter behavior for runs pages in Tasks.
• Enhanced email link functionality in Notifications.
• Added new filter options to duration columns across the UI.
• Introduced an archive filter in Tasks for better management.
• Updated search functionality in Tasks for improved user experience.
• Enhanced the UI to match changes in notification endpoints.
• Improved redirection query parameters in Notifications.

Bugfixes​

• Corrected failure notifications to accurately report workflow names.
• Resolved pipeline failures when Logic App workflows complete with a Succeeded state.
• Enabled enterprise admins to update enterprise logos in FDS.
• Fixed expiry date settings in the Schedule tab for Tasks.
• Addressed metadata failures due to bad credentials in SAP SAC, Snowflake, and Salesforce connectors.
• Resolved connection string issues in Salesforce and ServiceNow with the latest credentials.
• Fixed routing errors and run menu issues in PowerBI and Extend tasks.
• Ensured the dashboard accurately redirects from "Object Acquisitions Running Now" modal.
• Corrected metadata catalog errors in Fivetran due to excluded tables.
• Fixed Empower UI issues during custom flow updates.
• Addressed capitalization handling in ModelBuilder Spark stage tables.
• Corrected toast notifications when models are created.
• Adjusted icon sizes in the Associated Task section for Connect.
• Resolved connection string errors in SAP HANA and fixed metadata extraction failures.
• Corrected LastRunStatus always returning null.
• Ensured correct page navigation and routing in dashboard sections.

Enhancements and Features​

• Enable Notifications for 1.40 release
• Enhance Logging for Extend tasks
• Implement MAR Service for utilization tracking
• Deploy Data Factory Pipelines for Fivetran acquisition
• Create new version of OnPremise Oracle Connector
• Remove databricks_account_id, databricks provider from external-organization in 1.40
• Enhance UC Terraform application to support existing metastore
• Add support for enable_databricks_private_endpoint for enabling databricks private endpoint
• Add support for enable_storage_account_sftp for Datalake storage account
• Add new task/model side panel
• Unfeature flag entity dependency table toggle
• Update breadcrumb to scroll to location on grid
• Unfeature flag new task views
• Create edit form for tasks and model pages
• Create Overview page and tabs for all Task types (minus Extend tasks)
• Add configuration tab for all task types
• Add Scheduling tab for each task type
• Add runs tabs for task type
• Add column filters to each of the task main page grids
• Add Task Display in Connector pages
• Add scheduling for Fivetran tasks
• Update Overview and Config pages for Fivetran tasks
• Integrate API and unfeature flag notifications
• Integrate API affiliated task endpoint to failure modal
• Clean up loading of Publish and Acquire pages
• Update size of model icon in sidenav
• Update overview page action behavior
• Unfeature flag object failure
• Add Fivetran docs route
• Include Fivetran Tasks in Dashboard Analytics
• Update UI endpoints to match change in notification endpoint
• Data Migration : Validate data migration between the environment with different versions
• CoreAPI Integration to notification service for email alerts
• Create Acquire Task in API automatically when Fivetran Connector is made
• Update GET/POST endpoint recipients for notification service
• Enhance Logging for flows_flowrun
• New Fivetran task type creation experience for acquisition tasks
• Put required proxy endpoints into the fixture
• Allow filtering tasks by multiple task types

Bugfixes​

• Fixed 400 error in custom flows for Oracle and MariaDB
• Properly configured UTF-8 values in code
• Resolved Quickbase CData Connector failures
• Fixed JSON parse errors in ADF custom flow history
• Fixed JSON parse errors in ADF data acquisition history
• Fixed PostgresSQL connection template
• Fixed DB2 batch extraction failures
• Improved logging for ADF custom flows when Databricks jobs don't start
• Added error messages for failed Logic App custom flows
• Attached Empower Policy to interactive cluster
• Fixed Databricks extraction retries in Acquire
• Resolved extraction failure for Workday RAAS
• Fixed infinite "in progress" state in Acquire when task status is Success
• Set proper value in url column for failed flow runs
• Corrected DAQ failed object count in notification service
• Enabled separate notification toggles for the same user
• Fixed inconsistent subscription behavior across workflow levels
• Displayed correct table when toggling entity dependency table
• Standardized time format in notification emails
• Ensured back button is visible when toggling entity dependency table
• Refreshed affiliated tasks when switching between models
• Updated "New Record" button when switching to Entity Column tab
• Resolved image rendering issues for source types
• Aligned grids in Preview Run and Schedule modals
• Fixed incorrect back navigation from run history
• Corrected schema names in Fivetran filter dropdown
• Made columns visible without scroll in run history
• Updated archived chip display in Acquire grid
• Renamed "View Affiliated Flows" to "View Affiliated Tasks"
• Displayed tooltip for disabled Reset Error State button
• Added "Cleared for Retry" to Error State filter
• Disabled Run Error State button when no objects selected
• Displayed toast when editing metadata catalog options
• Prevented toggle jumping in bulk edit banner
• Highlighted watermark column when using dedupe merge strategies
• Bolded text in metadata catalog bulk edit banner
• Fixed repeated sentence in confirmation modal
• Removed white banner when opening new task
• Enabled Archive/Restore on Tasks Overview
• Redirected after deleting task from Overview
• Added tooltip for Metadata Sync button
• Displayed breadcrumbs in Publish/PowerBI Overview
• Corrected tooltips for archived/deleted tasks or connections
• Enabled Save for non-Spark workers in Acquire without advanced options
• Fixed styling of chips in Add Team Members modal
• Fixed comment styling in Data Migration
• Disabled cell/row editing in archived Publish task
• Highlighted correct row in Extend history when URL has runId
• Added left border in Preview grid of Schedule page
• Fixed breadcrumb routing from Acquire schedule to correct task type
• Enabled Schedule button for "Repeat on NONE"
• Fixed validation for start/end dates in Schedule tab
• Prevented past date selection for Start Date
• Removed extra columns in Dynamic CE Publish config
• Fixed new schedule saving issue in Schedule tab
• Standardized time zone display in Flow Scheduling
• Removed already-added users from Add New Email search
• Enabled Run and Schedule buttons in new views
• Allowed deleting duplicate email entries
• Displayed underscore in disabled Fivetran schedule column
• Disabled Schedule button without required fields
• Made Transform config grid read-only
• Prevented object selection in archived tasks
• Sorted Est Rows correctly
• Fixed incorrect labels in model edit form
• Opened correct docs from Model Overview
• Showed correct tooltip for Include toggle on archived tasks
• Reset filters in Acquire config
• Displayed filter data in Publish config
• Aligned Fivetran Acquire run history
• Displayed correct icons on Publish main page
• Enabled Fivetran scheduling from Overview page
• Fixed 404 when viewing field details
• Made last row visible in Acquire config tab
• Resolved small UI issues on Task Overview
• Made Schedule toggle clickable
• Fixed routing for workflow link in Dashboard modal
• Prevented creating/updating tasks with archived parent connections
• Moved Archived chip to correct location in Runs tab
• Corrected time format in Runs tab
• Displayed correct task name in Create Task panel
• Updated Last Modified when changing name/schedule/archive
• Fixed layout in PowerBI Overview
• Removed Fivetran connections from new/edit forms
• Fixed polling in Run tab
• Removed debug log of typed values in New Task form
• Stopped infinite skeleton loader in Connect
• Pre-filled connection field when navigating from "Create New Task"
• Loaded task grid properly from "Create New Task"
• Fixed issues with task scheduling form
• Re-enabled Re-Authorization for Fivetran connections
• Enabled/disabled scheduling in Fivetran Schedule tab
• Reflected Schedule toggle changes for Fivetran tasks
• Fixed 404 on Transform History page navigation
• Enabled Publish Configuration button
• Enabled model enable/disable toggle
• Added search suggestions in Publish config new/edit forms
• Fixed 502 error during merge promotion
• Fixed 400 error on Fivetran source creation
• Prevented 500 error when fetching recipients by level
• Automatically archived task on Fivetran connector deletion

Enhancements and New Features​

• Added Textura, SAP Concur (with retry support), Excel on Dropbox, and SQL Server NonSpark CDC connectors
• Made the new Dashboard the default homepage with real-time task and ingestion monitoring
• Displayed empty states and improved skipped ingest handling in the Dashboard
• Updated Cluster Name field to an autocomplete dropdown for easier configuration
• Added help tooltip for first-time users to assist with onboarding
• Replaced “Flow” with “Task” throughout the UI
• Enabled bulk editing and improved endpoints in the Metadata Catalog
• Upgraded Spark clusters to version 15.4 LTS for improved performance
• Added new merge strategies with deduplication options in Spark pipelines
• Refactored file type conversion utilities for SharePoint, OneDrive, and SFTP
• Improved handling of special characters in data obfuscation columns for connectors
• Recorded skipped extractions in metadata to maintain audit completeness
• Integrated Fivetran connect cards into the UI (private preview only)
• Implemented grid views for Acquire, Transform, Publish, Custom, and Model pages (private preview only)

Bugfixes​

• Fixed invalid Client ID and Secret ID retrieval for PowerBI connections using Key Vault
• Resolved UI overlap issues on the Data Migration settings page
• Fixed object count not updating with calendar date range selection in the Dashboard
• Addressed private_link_id assignment errors in Fivetran destination setup
• Restored breadcrumbs visibility for PowerBI Refresh flows in the Publish view
• Resolved single-column layout issue when returning to Dashboard from Settings
• Fixed Fivetran UI not updating after archive/restore/delete actions
• Removed improper display of metadata and re-authorize buttons on archived Fivetran connections
• Fixed “Get Latest Metadata” button appearing for archived Fivetran connections
• Corrected timezone display in Fivetran object configuration modal
• Fixed incorrect schedule toggle display when paused in Fivetran config modal
• Dashboard object filter now correctly shows number of selected objects
• Fixed inconsistency in Object Acquisitions bar chart headers and running state display
• Standardized timestamp format for migration creation times to UTC
• Resolved polling mismatch in Last Month time range selection on Dashboard
• Fixed Dashboard errors that caused object acquisition data not to display
• Addressed issue where publishing entities remained stuck in running state
• Fixed nav label inconsistencies when changing “Flows” to “Tasks”
• Resolved issues from incomplete replacement of “Flows” with “Tasks” across the UI
• Fixed pipeline crash caused by undefined init_params in Spark script
• Resolved gateway timeout due to secrets listing
• Fixed breadcrumb misplacement on Connect page
• Removed redundant buttons (Cancel, Continue, Request) from Connect page
• Corrected scrolling behavior of Cluster Name dropdown
• Fixed excessive API calls (~100) when opening Transform page
• Restored visibility of Cluster Name dropdown on transformation edit form
• Corrected wrong entity name sent to Fivetran metadata catalog
• Resolved metadata field detail styling issues
• Hid cluster config options for non-Spark sources in Acquire edit task
• Fixed missing row highlight when clicking cell in Publish/Custom config view
• Resolved search issues in Metadata Catalog for all-caps object names
• Restricted Databricks cluster API to return only “All-purpose compute” clusters
• Fixed “Need Help” button in Model page opening incorrect documentation
• Restored deleted connection chip visibility in Acquire connections table
• Fixed visual mismatch in metadata catalog no-results state
• Restored Model Name column sorting in Model page
• Renamed “Created” column to “Last Modified” in Model page
• Fixed 404 errors when viewing Transform and Publish configurations
• Resolved dropdown visibility issues when using overflow menus in Flows
• Fixed Connect page loading issue when grid view is enabled
• Corrected task ID/run ID order in object ingest failure tracking
• Resolved 400 error when fetching affiliated flows by object ID
• Fixed false discard changes modal when no changes were pending in Metadata Catalog
• Restored missing “Not Selected” option in Watermark Type bulk edit
• Corrected merge strategy options visibility based on Advanced Options toggle
• Fixed failure to load Metadata Catalog for some connections
• Resolved issue saving Change Tracking merge strategy
• Fixed mass-disable action unintentionally disabling unrelated Acquisition tasks
• Addressed failure in Netsuite workflow pipeline test
• Corrected isIncremental flag value for FULL_MERGE strategy in API

Enhancements & New Features​

• Displayed Empower documentation directly within the application interface during onboarding.
• Added sorting functionality for the Active column in Workflows.
• Implemented an empty state view for the side panel in Workflows.
• Removed pagination for large object lists in the Metadata Catalog.
• Updated Watermark field options in the Metadata Catalog.
• Refined requirements for merge strategies in the Metadata Catalog.
• Integrated UTF-8 options into Spark job pool configurations.
• Deployed default Blob Tiering policies for Datalake environments.
• Developed the DB2 Connector for IBM DB2 database integration.
• Enabled persistence of configuration values when switching batch extraction options in Cluster Config.
• Unfeatured flag-based group management from the roles page in Access Control.
• Created template for converting Gold Layer Object to Liquid Clustering in Spark.
• Added a view trace button to the error modal in Flows.
• Updated descriptions in Cluster Config for clarity.
• Hid Cluster Config options for non-spark data sources.
• Implemented various UI updates on the Connect page.
• Updated PowerBI flow styles within Workflows.
• Enabled jobLink data in flow details endpoint returns.
• Provided a list of Databricks cluster names through Core API for Cluster Config.
• Surface trace links for ingestion, modeling, and publishing processes in Troubleshooting tools.
• Included ObjectPublish.job_link in flow history detail responses.

Bugfixes​

• Fixed Last Runs alignment issue in Workflows when Running state is included.
• Corrected status update failure for objects on error due to JSON message issues.
• Resolved tooltips inconsistency during Workflow run.
• Fixed display of '-optional' values in Cluster Config form for Acquire flow edits.
• Corrected console errors for stack size exceeding when toggling advanced options.
• Addressed incorrect default cluster type display in Cluster Config for Acquire flows.
• Resolved breadcrumb navigation issues upon switching to settings in Metadata Catalog.
• Removed unwanted horizontal scrollbars in side navigation for small screens.
• Fixed metadata extraction errors with SAP Hana and PostgresQL.
• Resolved visual issue with breadcrumbs application-wide.
• Updated field label from "Connection Name" to "Display Name" on Connect screen.
• Fixed alignment of "Source Notebook" information on Workflows Configuration page.
• Corrected pointer cursor behavior for "?" in object failures modal.
• Ensured correct documentation redirection via Need Help button.
• Addressed incorrect default documentation display from Need Help button.
• Removed unexpected text from "New Publish Flow" form.
• Fixed "Node Size" field display in edit mode for old flows.
• Resolved '-Optional' label issues when switching from non-spark to spark data sources.
• Fixed incorrect API calls in Metadata Catalog connections.
• Corrected cluster config mismatch for custom flows from Regression workflows.
• Resolved navigation issue in error modal with "View Trace" button.
• Fixed UI issues with trace buttons for null jobLinks.
• Corrected error styling for Watermark columns in Metadata Catalog.
• Enabled inactive connections to be selectable.
• Resolved publishing errors due to schema column issues.
• Ensured archived connections appear in the connections list.
• Fixed missing "JobLink" population after flow runs.

Enhancements & New Features​

• Created a Spark connector for SAP Concur integration into Empower Spark.
• Created a Spark connector for Redshift integration into Empower Spark.
• Added connection schema validation for Data Source.
• Implemented created by filter in Workflows.
• Added and displayed filtering for runs modal in Dashboard.
• Enabled sorting for Created By column in Workflows.
• Added sort by Schedule column in Workflows.
• Executed Custom Data Factory Pipeline as part of a Custom Flow.
• Executed Databricks Workflow from Data Factory.
• Created OneDrive Connector.
• Extended connectors (Sharepoint, AWS S3, SFTP) to support XLS, XLSX files.
• Updated flows to support cluster configuration.
• Fixed batch extraction issue with non-spark sources in Data Factory.

Bugfixes​

• Fixed issue where user cannot set custom "Repeat On" flow schedule in Empower UI.
• Resolved problem where workflow breadcrumbs are lost after opening settings.
• Corrected issue where user can't select filter option by clicking on the checkbox in Empower UI.
• Fixed Expires parameter having a value on schedule when it shouldn't.
• Corrected dialog box loading issue the first time when trying to schedule flows in Empower UI.
• Resolved publishing failure due to duplicates error at the target environment.
• Fixed issue where the "Refresh" button at the Run Logs blade doesn't update Duration in Flows/Workflows.
• Ensured Access Control search works at any level and doesn't throw errors in console.

Itemized Changelog​

Enhancements & New Features​

• Reengineered Watermark functionality for Data API.
• Reengineered publish process for SQL Server and Azure SQL DB in Publish Connector.
• Made Interactive Clusters work for Spark and custom connector data sources in Data Factory.
• Changed Databricks authentication type for improved security.
• Created workflows runs overview tab for better monitoring.
• Renamed menu items for empower navigation.
• Added documentation menu item to the sidebar.
• Implemented workflow name filtering support.
• Created a Cornerstone connector.
• Extended SFTP connector to support different file formats.
• Added SQL Server & Azure SQL DB CDC support.
• Created JDBC Connector for Azure MariaDB.
• Developed S3 Bucket Connector.
• Created a PowerBI refresh flow type.
• Updated publishing page to display PowerBI flows.
• Added instructions information modal on PowerBI flows.
• Added tooltips for workflows overview tab.
• Added support for migrating flows of type 'CUSTOM' in Config Migration.
• Updated Metadata Extraction Process to update Last Connected datetime for Data API.
• Rewrote code for Azure SQL DB JDBC Connector.
• Enhanced PowerBI workflow runs to save 'workspaceId' and 'datasetId'.
• Created a script to fix legacy MergeStrategy setup in StateDB.
• Refactored workflows pages to utilize shared state.
• Updated create new flow on the publishing page in UI for PowerBI Flows.
• Ensured PowerBI Flows appear in workflows 'Publishing' tab.
• Updated flow run history for PowerBI flows to surface a details pop-out blade.
• Removed 'View' button from System Versions page in empower UI.
• Clicking workflow name from Workflows overview now routes to /history instead of /config.
• Updated menu to remove blue collapse arrow on Settings sidebar.

Bugfixes​

• Fixed issue where Interactive cluster incorrectly runs when passing -1.
• Resolved candidate key overwrite issue.
• Corrected error message display when user clicks multiple times on Complete Migration button during Data Migration.
• Fixed inability to schedule runs for a future time on the same day as the start date in Workflows.
• Addressed Max Watermark Logic issue in Extract Stored Procedures causing problems with Synapse Link Loading.
• Resolved issue where POST /promotions/sources/ can't create data source on the target environment in Config Promo.
• Fixed SQL Server error preventing user from creating Flows migration preview in Config Promo.
• Corrected Pipeline behavior to properly fail records in case of AZURE_QUOTA_EXCEEDED_EXCEPTION (CLIENT_ERROR) for PL_PUBLISH_DATA.
• Addressed queueing issue for single objects for ingestion in MariaDB and MYSQL.
• Ensured Custom Flows work by resolving pipeline failure.
• Fixed issue where DAQ Flows ingestion filter dropdown does not show "Queued" state filter option.
• Made three dots menu visible for the last records on the grid in Workflows.
• Ensured workflow name updates correctly in the workflows configuration page.
• Aligned Flow Run Logs Grid and traffic light component in DAQ.
• Resolved filter icon issue in Chrome where it is shifted and not fully visible when sorting Data Publishing Flow Run Logs.
• Fixed issue where Runs tab does not show data on opening the tab for first time after login in Workflows.
• Ensured data loads on page refresh for Workflow Runs.
• Fixed Save & Test failure for sources of type MariaDB_New and Postgres_New.
• Ensured workflow history loads when switching from Configuration page to Runs in Workflows.
• Corrected workflow status change when running workflow from Configuration page.
• Adjusted delete workflow confirmation modal to display as expected.
• Updated restore workflow confirmation modal message to be as expected.
• Corrected delete confirmation message on an Archived workflow.
• Fixed flows order at the Run/Schedule modal when opened from Overview page in Workflows.
• Ensured Workflow Name is displayed for all flows.
• Updated "Last Modified by" date and time after updating name and description in Workflows.
• Fixed issue where switching between Configurations to Runs tabs, updated flows are not getting saved in Workflows.
• Resolved access issue where users are getting logged off.
• Displayed running state in "Info" column in Workflows.
• Ensured Workflow column is sorted alphabetically without case sensitivity.
• Fixed deselection issue of filter values unrelated to the first column selection in Workflows.
• Corrected navigation from specific Run back to Overview page in Workflow Runs.
• Adjusted confirmation modal text to correctly show Archive, Restore, and Delete Data Source in Connect.
• Updated naming inconsistencies where Acquire flow shows as Data source while creating new Acquisition flow.
• Corrected toast messages to show as Data Source for Archive, Restore, and Delete in Connect.
• Fixed modal text showing Deleting an Acquire flow as Data Flow in Acquire.
• Corrected flow status text showing as Data Publishing for new Publish flows.
• Ensured "Last Modified by" date and time are updated after toggling Schedule Active state from configuration page in Workflows.
• Updated Icons for Acquire and Transform on the Workflow - Add flow screen.
• Fixed editing transformation name display issue in Transform.
• Ensured flow run ID is copied correctly to the clipboard from the Flow Run Details modal in Workflows.
• Corrected various name updates across the UI.
• Fixed Delete, Archive, Restore functionality for Acquire and Transform flows.
• Resolved the wrong Flow Group names being returned from the API in Data Migration.
• Updated menu to remove blue collapse arrow.
• Ensured PowerBI Flows appear correctly in workflows 'Publishing' tab.
• Updated pop-out blade for detailed flow run history for PowerBI flows does not surface.
• Removed 'View' button from System Versions page.
• Corrected routing behavior when clicking workflow name from Workflows overview to /history instead of /config.
• Fixed display issue with Workflow Name column title at the "Runs" tab.
• Resolved error in the console when switching tabs in System Settings.
• Fixed "All Other Configurations" toggle name at the New Migration screen in Data Migration.
• Corrected element placement issue in the sidebar under Settings.
• Fixed remaining naming inconsistencies across the UI.
• Ensured workflow name is displayed when restoring workflow in Workflows.
• Corrected behavior to prevent search on Active/Info/Schedule/Last Runs in Workflows.
• Ensured ability to remove the workflow in configuration in Workflows.
• Updated toast message to show correct flow name when creating new Refresh PowerBI flow.
• Ensured all filter dropdowns show appropriate options when the grid is empty in Workflows- Overview.
• Fixed issue where the add flow modal doesn't open right away on click.
• Ensured last item in Configurations is fully visible in Workflows.
• Corrected "Source Deleted" chip text to "Connection Deleted".
• Ensured Publish flow appears correctly as Publish in Workflow history page.
• Updated toast message to correctly indicate Workflow is saved.
• Fixed duration update for running workflow state.
• Ensured flow status is updated on running without refresh in Publish.
• Fixed spacing issue between last two items at skeleton loader in Sidebar.
• Corrected Cancel button functionality on Workflow schedule page.
• Ensured "Archived + Scheduled" Workflows are not displayed if "Not Scheduled" filter option is selected.
• Fixed issue where Archive workflows should not show previous schedule in Schedule column in Workflows.

Itemized Changelog​

Enhancements & New Features​

• Developed SharePoint Data Source Connector.
• Updated workflows endpoint to include lastRunsList computed field.
• Created new history endpoint for getting aggregate workflow details in Workflows.
• Implemented FTP Spark connector in Empower Spark.
• Updated flows endpoint to include lastRunsList computed field.
• Metadata extraction pulls candidate keys.

Bugfixes​

• Ensured "Last Connected" and "Last Modified Date" fields are updated after updating Connection Details in Data Source.
• Resolved issue of duplicate records in publishing logs.
• Fixed conversion of key values from "snake_case" to "camelCase" in Custom Flows.
  Removed RTK property from FTP connector.
• Updated ObjectName column in Sharepoint connector.
• Addressed issue where Dynamics Synapse Link Trickle Feed does not work.
• Fixed limitation with max number of fields in ingestion process.

Enhancements and New Features​

• Updated flow history page to include Target and Schedule for Publish Flows.
• Created a pop-out blade for detailed flow history in Publish Flows.
• Introduced stoplight progress bars for Object Publish logs.
• Updated pop-out blade to contain flow run information for Publish Flows.
• Developed grid view for Object Publish logs on Flow Runs.
• Implemented Object Publish fail modal.
• Added tooltip hover states on Publish Logs.
• Tied custom flows to advanced options toggle.
• Displayed External Id instead of Id for flow runs.
• Connected Refresh Button on Publish Logs.
• Connected details blade to API in Publish Flows.
• Connected stoplight progress bar to API.
• Connected table and tooltips to API.
• Enabled merge strategies in the Metadata Catalog UI.
• Updated merge strategy setting to utilize isIncremental.
• Updated load strategy visibility to advanced options only.
• Updated run modal to display merge strategy in DAQ Flows.
• Implemented error trace for high-level flow history failures in DAQ Flows.
• Provided error modal for DAQ flows if error is provided.
• Developed data flow-related workflows modal.
• Implemented schema search/sort filters on DAQ flow details blade.
• Implemented name search/sort filters on DAQ flow details blade.
• Implemented extraction sort filters on DAQ flow details blade.
• Implemented ingestion search/sort filters on DAQ flow details blade.
• Assumed load strategy from Merge Strategy against old API without IsIncremental.
• Hid merge strategies by advanced options in Metadata Catalog.
• Created new Databricks self-service connector and UI capability.

Bug Fixes​

• Resolved issue where running or scheduling a Data Publishing flow shows disabled entities in the run plan modal.
• Fixed issue where "Objects" text is displayed near entity count at the pop-out blade for the Flow Log History in AE Flow.
• Corrected behavior where Flows unhighlight during polling.
• Fixed display issue for the longest possible Source Notebook path in Custom Flow.
• Corrected default schedule issue when clearing schedule in Workflow/Flow.
• Removed unnecessary text in the Schedule modal in Workflow/Flows.
• Fixed issue where start time is not populated for entities returned from GET: /details endpoint in Analytics Engineering.
• Resolved issue where breadcrumbs disappear during navigation in UI.
• Fixed required fields validation issue when selecting "Full Dedupe Merge" for Merge Strategy column in Metadata Catalog.
• Corrected issue where Load Strategy value is not saving after selection in Metadata Catalog.
• Fixed issue where Clear Filters button is displayed enabled when nothing is selected at the filter dropdown in DAQ.
• Resolved issue where selected filtering remains when switching between flow runs in DAQ.
• Corrected issue where not all filter options are displayed at the Filter dropdown in DAQ.
• Fixed issue where search component is present at the filter dropdown for the Extraction column in DAQ Flows.
• Corrected issue where sorting is not working for the Extraction column at the DAQ Flow Logs pop-out blade.
• Fixed issue where "Merge Strategy" column is not displayed by default at the Run/Schedule modal in DAQ Flows.
• Fixed Databricks METADATA extraction issue.
• Resolved issue where Start/End Time is cleared out after closing Extraction/Ingestion filter dropdown at the DAQ Flow Run Logs pop-out blade.
• Fixed issue where Publishing Error modal is opened when clicking on Succeeded Records at Flow Run Logs pop-out in Data Publishing.
• Corrected required Fields validation issues when selecting Merge Strategy column in Metadata Catalog.

Enhancements & New Features​

• Implemented ingestion check to avoid ingestion collisions.
• Improved Data Publishing pipeline logging.
• Enhanced workflows in Data Factory Pipelines.
• Introduced deduplication logic for Synapse Link sources.
• Developed a Parquet file Data Source Connector.
• Added BeautifulSoup Library for HTML content removal in strings.
• Enabled incremental publishing in Dynamics package.
• Created a Spark connector for Microsoft Entra integration.
• Supported interactive clusters configuration.
• Enhanced custom flow creation capability.
• Updated stored procedures for model management.
• Removed all mount points from code in Spark applications.
• Automated registration for new organizations and backfilled existing ones.
• Refactored UI for Self-Service Onboarding review and deployment submission.
• Added Role-Based Access Control (RBAC) descriptions to all endpoints.
• Implemented a feature for auto-enabling or disabling functionalities via waffle switch.
• Removed feature flagging for flow promotion.
• Updated the Entity Column Model ID for Advanced Options in Modeling Entity.
• Updated endpoint for detailed flow history.
• Enhanced API to support new migration types in the future.
• Implemented a pop-out blade for detailed flow history.
• Introduced stoplight progress bars for Object Model logs.
• Updated tooltips for modeling status indicators.
• Implemented refresh button for AE Flows flow run details.

Bugfixes​

• Resolved errors across multiple widgets due to premise service issues.
• Fixed errors in various load groups and data processing failures.
• Corrected behavior for notebook error handling and user configuration issues.
• Addressed connector linked service changes and metadata failures.
• Ensured proper failure mechanisms for custom flow pipelines and source type errors.
• Fixed UI issues in Account Settings and Self-Service Onboarding pages.
• Corrected behavior for environment setup limitations and search functionalities.
• Resolved display issues in Flows UI when switching between running and completed flows.
• Fixed an issue where updating a flow with an empty "description" field was not possible.
• Addressed incorrect completion date for immediate Data Flow runs.
• Fixed Object Model fail modal text copies and display issues in AE Flows.
• Resolved error trace formatting and incorrect value display in Data Publishing.

Enhancements & New Features​

• Cluster Configuration: Added support for cluster configurations in Data API, changes in ingestion execution per source, and use of cluster configuration in ingestion and modeling.
• Batch Extract Updates: Spark connectors can now extract data in batch, resulting in more efficient cluster utilization. This will be rolled out slowly over several releases.
• Metadata Error Messaging: Enhanced metadata error messaging for failed metadata extraction.
• Metadata Catalog Delta Ingestion: This setting is now automatically enabled when an object is enabled.
• Config Migration - Data Flows: Data flows (Data Acquisition, Analytics Engineering, and Data Publishing) are feature flagged to be promotable. Talk to your delivery team with getting access today.
• Data Acquisition Flow Logs: Data Acquisition flows now contain interactable flow log history, including flow run logging on an object-by-object basis.
• Custom Flows: Currently in Beta, custom flows are a new flow type available under a feature flag in the 1.31 release. A custom flows will enable the execution of a single notebook, specified in its configuration. Talk to your delivery team with how to enable the custom flow flag in your environment today.

Bugfixes​

• Fixed issues causing ADF pipelines to fail, including problems with data acquisition, PL_MODEL_DATA, META_SPARK_PL, and more.
• Resolved issue where service returned a 500 error on getting user photos.
• Corrected failures in data ingestion and extraction processes. Addressed issues with wrong associations and failed batch ingestion.
• Fixed a problem with internal PBI reports.
• Addressed various UI bugs related to column titles in uppercase and incorrect display of end times and statuses across different parts of the application.
• Corrected display issues on the Data Migration and Settings pages, including active migrate buttons and information text placement.
• Fixed issues with the status field not updating after refresh and error modals not opening as expected.

Enhancements & New Features​

• Model building now allows watermarking when doing an incremental model with direct SQL queries.
• Model building now supports missing value -> ghost record automation for more robust entity buildout when columns are missing values.
• SQL Spark connector build for Spark extraction to support non-ADF streaming. NOTE: this is currently only available by request. Talk to your delivery team to learn more.
• Improved change tracking for MSSQL by modifying candidate key fetch logic.
• The Metadata Catalog now contains an Options column to allow for object-specific options.
• Dynamics F&O and CE are now available as publishing targets in the UI.
• Various UI updates including support for editing missing value columns and null values in source notebooks.
• Improved minimap in AE Entity Dependency for better navigation.
• Added measures to prevent 404 errors when switching environments.

Bugfixes​

• Fixed a 500 error when downloading changelog due to permission issues.
• Resolved an issue where the connection was closed unexpectedly during Spark publish operations.
• Fixed syntax errors in Modeling and Parsing operations, enhancing stability.
• Addressed issues leading to data duplication and errors during object ingestion without Flow ID.
• Corrected empty Flow tables in StateDB and modeling failures due to missing values.
• Improved error messaging by filling out the Job Link field when a notebook fails.
• Fixed various UI bugs affecting the AE Entity Dependency graph, records jumping in modals, buggy mini maps, and inconsistent search field behavior among others.
• Corrected timezone application to accurately reflect UTC settings.

Enhancements & New Features​

• A new connector has been created for integrating Origami into Empower Spark, enhancing data ingestion capabilities.
• A new connector for NetSuite has been developed, improving data extraction and loading processes.
• Enhanced DDU to support the insertion of inactive rows, improving data management flexibility.
• Implemented incremental load with filter for SAP SAC.
• Metadata Extraction Default Behavior Update: New objects now default to off for new and existing data source connections.
• A new template notebook has been introduced for Empower for Dynamics, simplifying the development process.
• New table modes supported in ingestion: RAW DATA TABLE and HISTORY TABLE. Note: not UI-accessible yet.
• Enhanced the API to include previous promotion blob path in PromotionTable.
• New models for advanced filtering on target environment and migration status in the Data Migration have been created.
• Enhanced filtering on Log - Ingest and Log - Extract pages to support querying with Database ID List, allow users to find logs faster.
• Updated APIs to support an array of Database IDs.
• Removed the google-libphonenumber node module from empower-ui.
• New Entity Dependency creation, entity connection UI, and enablement of connection and node deletion in the entity dependency view have been implemented.
• Created a mini map and controls for better navigation and visibility within the Analytics Engineering Graph View.
• Public Preview - Direct SQL Statements are now public-preview supported during model buildout using the entity's "notebook" column to specify the path of the SQL file.

Bugfixes​

• Connector Bugfixes: Various issues related to ingest and extract failures across platform connectors like Workday RaaS, SAP SAC, Origami, and Netsuite have been resolved.
• WEB PUT FAILURE Error Handling: Improved error handling for WEB PUT FAILURE steps not catching errors from ADB in SPARK_PL.
• UI Enhancements and Fixes: Multiple UI issues related to AE Entity Dependency, Metadata Catalog, and advanced options in the ingest logs have been addressed, improving user interaction and visual presentation.

Enhancements & New Features​

• Primavera P6 - Created a spark connector implemented for self-service.
• SAP HANA - Created a spark connector implemented for self-service.
• Attendance On Demand - Added new endpoints.
• Updated Empower Spark Dependencies.
• Added Class D source SAP SAC documentation.
• Updated Dynamics Synapse Link to read CSV files more intelligently.
• Config Migration - Enhanced promotable entity configuration.
• Config Migration - introduced failure handling for merge if any exports fail, and included support for bulk updates.
• Metadata Catalog - Incorporated bulk edit enablement functionality,
• Metadata Catalog - Moved the Enabled column for better accessibility.
• Updated empower ui dependencies.

Bugfixes​

• Fixed issue with StateDB report not displaying data.
• Resolved UC_FILE_SCHEME_FOR_TABLE_CREATION_NOT_SUPPORTED error.
• Power BI Refresh - Fixed 'value' does not exist error.
• Fixed multiple issues with Guild Quality and P6 Connector failures.
• Config Migration - Addressed object_list.csv export failure and broken functionality due to umlaut from test ServiceNow instance.
• Better handling of unsupported sourcetypes in the UI.
• Addressed issues with enterprise logo changes not reflecting without page reload, and handling of non-square images and fallback images for logos.
• Fixed issues with field detail list staggering, enable pane overlapping AG Grid rows, bottom pane remaining after search/filter, and selection of not-allowed file types for logos.

Enhancements & New Features​

• Adjustments made to PowerBi pipelines for enhanced functionality.
• Introduced incremental load support for Workday.
• Cleaned up options field within state object list for better management and performance.
• Connectors have been updated to accommodate different field types.
• Added new metrics to the ingestion pipeline for better insight, planned to be added to the UI in an upcoming release.
• Updated deletion process to automatically permanently archive associated flows.
• A new endpoint has been created to get related information for a given source, improving data source management.
• Various updates made to Publish Flows and Entities, including migration updates and endpoint enhancements for better flow management.
• Multiple updates to improve the user experience across Config Promo, Data Sources, and Flows including better archival support and deletion UX flow updates.
• A new page has been created for enterprises to upload their logo, enhancing self-service brand customization.
• Improved the RBAC assignment page and made several updates to RBAC UI for a smoother user experience.

Bugfixes​

• Fixed an issue where authorization failures occurred during config migration.
• Fixed an issue where the system reported a blank state for the database, ensuring accurate data reflection.
• Fixed an issue with missing Databricks notebooks, preventing model building failures.
• Fixed an error related to the non-existence of specified filesystems.
• Corrected an endpoint failure in the Data API.
• Fixed various attribute and value errors that occurred during data extraction and publishing processes, ensuring data integrity.
• Addressed multiple user interface bugs, including error message color inaccuracies, wrong warning icons, and tooltip issues, enhancing overall user experience.
• Fixed a display issue in the Metadata Catalog, ensuring proper listing of field details.

:::warning Other Notable Enhancements and Fixes

• Fixed a Google Analytics bug that prevented successful ingestions.
• Addresses a variety of 4XX and 5XX level errors granting much greater platform stability.
• Built a template data modeling staging notebook and user guide, found here. :::

Enhancements & New Features​

• Public Preview release of Data Acquisition, Analytics Engineering, and Publishing Flows and respective UI modules.
• New About tab in the System Settings page, containing all Empower Service versions.
• Updated config promo source DB with the latest data.
• Included INTERNAL_VERSIONS_URL in the CoreAPI Container.
• Built a template stage notebook and user guide.
• Supported Business Effective Dates for Type 2 Dimensions.
• Modified Publishing Flow Pipeline to improve efficiency.
• Widened Empower DDU Logging Columns for better readability.
• Updated extract and ingest methods to support optional filtering.
• Modified Empower Spark's Ingestion engine to support flow execution.
• Support for friendly system reporting display names.
• Bulk update include toggle support for Data Acquisition flow configurations.
• Advanced Options specific improvements:
  • DB2Step renamed to Orchestration.
  • Flow Config renamed to Execution/Scheduling.
  • Entity renamed to Model - Entity.
  • Entity Column renamed to Model - Entity Column.
  • Entity Dependency renamed to Model - Entity Dependency.
  • Publish Entity renamed to Publish - Entity.
  • Object Model renamed to Log - Model.
  • Object Ingest renamed to Log - Ingest.
  • Object Extract renamed to Log- Extract.
  • Publish Execution Log renamed to Log - Publish.
  • Filtering on Success/Failure on all Log - tables.
  • Filtering by Data Source on Log - Ingest and Log - Extract tables.
  • Unused column cleanup in Orchestration table.

Bugfixes​

• Fixed issues causing failed responses from CoreAPI and System Versions endpoints.
• Addressed permissions and missing notebooks issues that were causing failures in Extract & Load.
• Resolved issues with metadata acquisition and ingestion process errors.
• Fixed extra columns issue in Google Analytics causing ingestion failure.
• Addressed various bugs leading to 500 errors, incorrect redirects, and UI discrepancies in Data API and UI.
• Corrected issues with column resizing, breadcrumbs, tooltips, and button naming in Advanced Options & Flows.

:::warning Notable Fixes

• Description field added to objects extracted from Dynamics FO via Synapse Link sources.
• Updated default logos for data sources and enterprises .
• Power BI System Reporting now properly displays in all environments.
• Data Migration comment box now moved to the bottom of the page for a better experience. :::

Enhancements & New Features​

• Updated Databricks Instance Pools to version 14.3 for Empower Spark.
• Modified policy to accommodate interactive computing with Dockerize Spark.
• Removed keycloak/empower-enterprise from Helm chart in version 1.25 to streamline Terraforms & Kubernetes deployments.
• Updated Textera to format timestamp columns to Spark type for better data handling.
• Enhanced Empower Spark with cleanups and improvements.
• Removed legacy environment variables from Data API as part of technical debt cleanup.
• Added Append-Only Mode to DDU for improved data management.
• Refined Connector Refactor by cleaning up the SourceBuilder base class and updating the network client.
• Supported a string datatype for the key column in DDU, enhancing flexibility.
• Removed the PyDantic dependency as part of technical debt reduction.
• Improved unit testing for Model Builder and added a retry policy to the Empower Client for enhanced reliability.
• Added a "description" field to objects extracted from Dynamics FO via Synapse Link.
• Updated default logos for sources and enterprises.
• Enabled updating enterprise logo via a new endpoint, enhancing brand customization.
• Included changes to payloads after QA feedback for better data handling.
• Filtered Source Type return based on active status.
• Added group endpoints to RBAC for enhanced security.
• Implemented server-side support for pagination and updated various API endpoints to support new features like Delta Ingestion and table name fields.
• Removed old KC client from Empower API service for cleaner architecture.
• Created a new endpoint for publishing entities by flow, improving data management.
• Implemented searchable dropdowns and updated tables to support PaginatedHttpService for better user experience in Advanced Options.
• Added support for filtering to various Entity pages and implemented UX support for pagination.
• Enhanced Metadata Catalog with support for readonly TableName
• Added polling to Flows pages for real-time log/status updates.
• Moved the 'Comment' section of config promo for better user interface design.

Bugfixes​

• Fixed Power BI reports not displaying in System Reporting.
• Resolved issues with surrogate keys not being created when capitalized in Model Builder.
• Addressed Object Ingest failures and metadata extraction overwrites in CoreData.
• Fixed Dynamics publishing failure and various extract issues with Workday Raas, MariaDB, and AOD.
• Corrected Azure Devops connector errors and other data extraction and publishing issues.
• Fixed bulk load failure, broken user sign-in links, and image errors.
• Resolved UI issues including misaligned icons, visual glitches, and dropdown behavior across Empower UI.
• Addressed sorting issues in Object Ingest, Extract, and Model tables in Core API.
• Fixed filter dropdown reset and added globalId support for Data Flows.
• Corrected various UI issues in Advanced Options, Data Sources, and Flow Scheduling.