Microsoft just released a new tutorial on implementing Type 2 SCD with a Dataflow: 👉🔗 https://learn.microsoft.com/en-us/fabric/data-factory/slowly-changing-dimension-type-two

Happy Monday everyone! For those of you exploring metadata-driven architectures (which I think is quite a lot of you!)... here are some ideas for you: As a quick recap: the metadata-driven data pipeline is a technique commonly used in data engineering. Rather than explicitly declaring the source and the destination for a Copy Data Activity (for example), we instead design our pipelines so that the Source and Destination can be passed in dynamically. This means we can store details of the Source/Destination connections in another location, which is read at Execution time. This adds a lot of benefits: scalability, maintainability, and many more. However, the point of this post was to start a discussion about how/ and where you can store such metadata. The two most common ways you see metadata stored (in a Microsoft environment) are 1. In structured tables (like the Data Warehouse) 2. In a JSON File (perhaps in your Lakehouse Files area). However, I'd like to throw in a third option for discussion: storing your metadata in a Notebook (and passing it into your pipeline using mspsarkutils.notebook.exit(). Pros of this appoach: - make your configuration trackable by version control (which is not possible with the previous two methods) Cons: - maybe more difficult to read, if you have quite a few Key/Value pairs Thoughts? Where are you storing your metadata at the moment?

Big announcement yesterday: Use Trusted workspace access and Managed Private endpoints in Fabric with any F capacity and enjoy the benefits of secure and optimized data access and connectivity  As the name suggested, you can now use Trusted Workspace Access and Private Endpoints on any F-SKU capacity. Previously it was reserved for F64 and above, so it's great that they are now making these features available to everyone. From the article: - Trusted workspace access allows seamless and secure access to firewall enabled Azure storage accounts. It is designed to help you securely and easily access data stored in Storage accounts from Fabric workspaces, without compromising on performance or functionality. This feature extends the power and flexibility of OneLake shortcuts to work with data in protected storage accounts in place without compromising on security. You can also use this capability with Data pipelines and the COPY INTO feature of Fabric warehouses to ingest data securely and easily into Fabric workspaces. To get started with this feature and to learn about limitations, see Trusted workspace access in Microsoft Fabric – Microsoft Fabric | Microsoft Learn. This feature can be used in any F capacity.  - Managed private endpoints provide secure connectivity from Fabric to data sources that are behind a firewall or not accessible from the public internet. Managed Private Endpoints enable Fabric Data Engineering items to access data sources securely without exposing them to the public network or requiring complex network configurations. Managed private endpoints are supported for various data sources, such as Azure Storage, Azure SQL Databases, and many others – the most recent addition being Azure Event Hub and Azure IOT Hub. To learn more about Managed Private Endpoints and supported data sources see Overview of managed private endpoints for Microsoft Fabric – Microsoft Fabric | Microsoft Learn. This feature can be used in any F capacity as well as Trial capacity. 

Microsoft publish a Known Issues Report which is a good thing to be aware of 👍

Read more here: https://blog.fabric.microsoft.com/en-US/blog/announcing-github-integration-for-source-control-preview/