Activity
Mon
Wed
Fri
Sun
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Sep
Oct
Nov
What is this?
Less
More

Memberships

Learn Microsoft Fabric

Public • 5.3k • Free

1 contribution to Learn Microsoft Fabric
Seeking Best Practices for Migrating to Microsoft Fabric
Hi all, I’m still new to this community and Microsoft Fabric, and I want to thank you all for letting me be a part of it. 🙏 I am currently considering migrating from traditional Microsoft Power BI solution to the Microsoft Fabric platform. I have a few questions that I hope you can answer. I've already watched a ton of videos on the topic—they inspire a lot of great thoughts but don’t quite give me a "solution" or a possible best practice to follow. We are an organization with many different systems and companies, which means we have many different data ingestions. Our data sources are primarily on-prem SQL servers, Microsoft 365 CRM, API calls to the HR system, and a few Excel sheets on Sharepoint. I’m particularly thinking about best practices for the architecture in Fabric. What should we use for data ingestion, transformation, and storage? Dataflows, notebooks, or pipelines—or maybe a combination? And what about the medallion architecture—is that for data ingestion to files in a data lakehouse or directly as deltatables in a lakehouse or warehouse? And how do you then proceed from there to the other steps in the architecture? I hope you can get back to me on my considerations so I can create a structured roadmap from the start, avoiding the need to redo the data architecture later on. Thanks in advance.
0
2
New comment Sep 5
0 likes • Sep 5
Hi Samuele, First of all thanks for your sharing your thoughts. Overall I have done the below considerations: As mentioned, we have on-prem SQL data as our primary data source. From what I understand, we have narrowed it down to data pipelines and Dataflow Gen 2. Ideally, I would like to copy the data from the source without any transformations. (100 % raw data) Here, I’m thinking about using data pipelines, but I’ve encountered an issue when trying to copy the data directly to Delta tables because there are spaces in the column names, which prevents me from doing so. Would it then be better to copy the data to JSON files in the Lakehouse files section and perform data validation through Spark notebooks, which would then convert the data into Delta tables in the Lakehouse? The final data transformation is intended to be done in SQL, so I’m considering creating a warehouse as the gold layer. I just think I need some expert considerations and maybe a example flow to see the Fabric in practise.
1-1 of 1
Tobias Jensen
1
5points to level up
@tobias-jensen-6336
Finance Business Analyst with passion about data, data analysis, data engineering and studying a master in data science as well.

Active 60d ago
Joined Aug 1, 2024
powered by