I've found the below blog states that, Microsoft Fabric increased the VARCHAR(MAX) will not truncated at 8kb and it'll support like mb now and mentioned as a private preview.Did anybody has that feature and just wanted to know is this really working if it really works and not truncated the big string values I can wait for this to implement a feature in my current project. Thanks, Article link: https://blog.fabric.microsoft.com/en-us/blog/working-with-large-data-types-in-fabric-warehouse/

I'm working on a Microsoft Fabric Data Pipeline where my use case is to get the latest files from a specific folder to process them incrementally. I have implemented the following logic: 1. Get Metadata Activity (activity name: Get Meta data_PF) – This activity retrieves the list of files inside the parent folder along with their properties: - Child Items - Exists - Item Name - Item Type - Last Modified 2. ForEach Activity – I use a ForEach loop to iterate over the items returned by the Get Meta data_PF activity and apply the "Filter by Last Modified" option. The filtering uses: - Start Time (UTC): @variables('start_date') - End Time (UTC): @variables('end_date') This pipeline runs daily and pulls files incrementally based on the date range. The files are fetched from an FTP server and copied to the Lakehouse for further processing. Currently, I process two files per day (e.g., 2024-10-21_Find.csv and 2024-10-21_Auto.csv), but I’m concerned about scalability. If the folder eventually contains thousands or even 100k files, iterating through all of them to find the required ones could be a performance bottleneck. My Current Idea: Since each file follows a date-based naming convention (e.g., 2024-10-21_Find.csv and 2024-10-21_Auto.csv), my idea is to: 1. Fetch the last successful run date from an audit table (e.g., 2024-10-19). 2. Find the difference between the last run date and the current date (e.g., 2024-10-22 - 2024-10-19 = 3 days). 3. Generate an array containing the expected filenames for these days: ["2024-10-20","2024-10-20","2024-10-21","2024-10-21"] Then transform it into: ["2024-10-20_Find.csv","2024-10-20_Auto.csv","2024-10-21_Find.csv","2024-10-21_Auto.csv"] 4. Use this array in the Copy Activity to fetch only the relevant files without iterating through all files in the folder. Senior’s Suggestion: A senior colleague suggested that instead of using ForEach to iterate over all files, I could: 1. Use Lookup activity to fetch the file list and order it by the "Last Modified" timestamp in descending order.

Hi Everyone, It seems that there is no straight forwqard way of passing an array variable from a ForEach activity to use as input to a Notebook activity. I would love to hear how this is being handled in the community. Any docs or examples would be fantastic Thank you

How can I implement a Tumbling Window Trigger in Microsoft Fabric, similar to Azure Data Factory, where the functionality is used to trigger a pipeline based on the successful completion of another pipeline? What are the steps to configure such a dependent trigger in Microsoft Fabric, and how does it compare to Azure Data Factory's Tumbling Window Trigger? Thanks in advance for any help or insights! @Will Needham Would you please share you're thoughts on this?

Hello, I’m really new to Fabric and I’m trying to configure a Data warehouse pipeline to get csv files from a `root` folder in an FTP Client. The challenge is that it doesn’t take all the folder but only the first file. I’ve already checked this with ChatGPT but it explains that the menu should have the option to select a folder, but it doesn’t appear. Any hints on how this could be fixed? Thanks in advance. Daniel