Since I have not much experience with API requests I don't know how to proceed further. What I am trying to do is to use PySpark Notebooks for getting first of all the response and then convert it correctly into a dataframe. This needs to work well on scale, because I need to ingest approximately 3 million rows. I can perform simple requests with the request python library, but don't know how I can translate that to creating a solution for big data. Therefore I need to perform paging/looping. The API I am using does only support paging through offset and limit parameters. So I need to loop till all items are retrieved while I still need to ensure that it's not causing overhead and running parallel. However, the output is nested which doesn't make it easier for me. I have issues with loosing data while converting it into a dataframe due to objects hold other objects or a new array with objects and somewhere in between the schema doesn't properly convert. Below I do have an example of all the levels in my json output. "company": { "companyId": "932xxx5stest", "companyCode": "TEST", "_links": [ { "rel": "self", "href": "https://api.test.com/v1/companies/932xxx5stest" } ] } The Copy Data Activity in a pipeline doesn't really work for me because my api doesn't provide really well the count of total items. So I can't extract that by using a pipeline which means i have to do things manually. That's why I prefer the notebook. Any ideas/usefull resources or (your) best practices are welcome! Thanks in advance. If you need more information, please ask and i'll provide some more context.

Hey everyone, happy Monday! Data Pipelines are a really useful (and powerful) tool, and used by many different personas in Fabric (from Data Engineers, Analytics Engineers, Data Scientists and sometimes Power BI Developers), but they can be a little difficult to learn for the first time. I'm designing some course content and tutorials specifically for people learning Data Pipelines, and I want to hear your experiences. I would LOVE to hear your perspective on: 1. How has your experience been so far learning Data Pipelines? 2. What have you learnt so far? How have you approached your learning so far? 3. What did you struggle to understand so far? 4. What are you hoping to learn in the future (related to Data Pipelines)? THANKS SO MUCH for your engagement - it really helps to fine-tune future courses 🙌🙂

Yesterday I passed the DP 600 exam with a score of 810. Thank you Will for creating this community and all the very detailed video's. They provided me a good foundation to pass the exam. Have a nice weekend!! Cheers

Read more here: https://blog.fabric.microsoft.com/en-US/blog/announcing-github-integration-for-source-control-preview/

Whenever I receive a question to choose the most cost effective capacity for a certain scenario, I find it difficult to answer because the documentation doesn't give for a beginner good guidelines and I dont understand what Microsoft expects. Most of the time you get a scenario of a fictive organization which has x users needed for Power BI and Fabric and some additional requirements. How would you answer certain questions? Which considerations do you make? And could that be visualized in a decision tree?