Part 3- Data Ingestion (1) | End to End Azure Data Engineering Project

Ironically, if i have never use MS SQL server before or any type of SQL database. i wouldn't really know much about how you can fix this. He should really cover on how to install and add adventure database since it can be challenging if it is your first time touching on these subject.

Hi I got below error for parquet. It was successful for csv. Please give solution.
ErrorCode=JreNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Java Runtime Environment cannot be found on the Self-hosted Integration Runtime machine. It is required for parsing or writing to Parquet/ORC files. Make sure Java Runtime Environment has been installed on the Self-hosted Integration Runtime machine.,Source=Microsoft.DataTransfer.Common,''Type=System.DllNotFoundException,Message=Unable to load DLL 'jvm.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E),Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'

Excellent! Thank you.

I M GETTING THIS ERROR DURING CREATION OF LINK SERVICE DO ANYBODY HAVE IDEA OF THIS
"Input Payload is invalid, validation result - '["Connector or activity name: SqlServer_LINK_S, connector or activity type: SqlServer, error: Calling partner RP EvaluatePolicyAsync returned an invalid status code 'Conflict', ReasonPhrase Conflict"]'
"

Hi Mr.K. Thanks for the amazing content. I'm facing below error while using parquet format in sink for address table. Not sure how to proceed further

ErrorCode=JreNotFound,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Java Runtime Environment cannot be found on the Self-hosted Integration Runtime machine. It is required for parsing or writing to Parquet/ORC files. Make sure Java Runtime Environment has been installed on the Self-hosted Integration Runtime machine.,Source=Microsoft.DataTransfer.Common,''Type=System.DllNotFoundException,Message=Unable to load DLL 'jvm.dll': The specified module could not be found. (Exception from HRESULT: 0x8007007E),Source=Microsoft.DataTransfer.Richfile.HiveOrcBridge,'

Where we can get data base script to use that in our laptops? I'm doing this project along side you.

SqlErrorNumber=229,Class=14,State=5
:i got this error when loading LT.address under pipline-source ,resolve this by executing query in local sql server

USE [master]
EXEC master..sp_addsrvrolemember @loginame = N'sai', @rolename = N'sysadmin'
GO
:in login name (sai) give your usename ,
it is for updating user role to system admin

Hi I am getting an error when testing connection with my storage account name. How do I resolve this issue?

ADLS Gen2 operation failed for: Operation returned an invalid status code 'Conflict'. Account: 'mrddatalakegen2'. ErrorCode: 'EndpointUnsupportedAccountFeatures'. Message: 'This endpoint does not support BlobStorageEvents or SoftDelete. Please disable these account features if you would like to use this endpoint.'. RequestId: '06c1dffb-e01f-0077-0f7e-fd67e3000000'. TimeStamp: 'Fri, 13 Oct 2023 02:36:48 GMT'..
Operation returned an invalid status code 'Conflict'

not able to access access policies please help

Hi Mate, I am getting the below error while installing the SHIR, Could you please help
Error:- Detail Error: {{Error [1]. }}Service '[2]' ([3]) failed to start. Verify that you have sufficient privileges to start system services. 1920 Integration Runtime Service DIAHostService Suggestion: Please use the manual setup to install the Integration Runtime (Self-hosted).

Hi All,

While replicating the ADF and ADLS data ingestion I got an error and do some minor changes in the code..

Error:-

ErrorCode=AdlsGen2OperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=ADLS Gen2 operation failed for: Operation returned an invalid status code 'BadRequest'. Account: 'storagedatabricksproject'. FileSystem: 'raw'. Path: 'SalesLT/ProductCategory
/ProductCategory.parquet'. Message: 'Bad Request'. TimeStamp: 'Sat, 09 Sep 2023 16:16:23 GMT'..,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=The file name 'SalesLT/ProductCategory
/ProductCategory.parquet' does not conform to the naming rules by the data store 'Azure Data Lake Storage Gen2'.,Source=,''Type=Microsoft.Azure.Storage.Data.Models.ErrorSchemaException,Message=Operation returned an invalid status code 'BadRequest',Source=Microsoft.DataTransfer.ClientLibrary,'

I removed the outer bracket {} from the below code

@{concat(dataset().tablename,'.parquet')}
@{concat(dataset().schemaname,'/',dataset().tablename)}

Correct code ->

@concat(dataset().tablename,'.parquet')
@concat(dataset().schemaname,'/',dataset().tablename)

Excellent work... please do more videos on real time projects... waiting for another Azure E2E pipeline

I am planing to go for Data Engineer jobs, In this sample project I dont see any coding. Is it requred to be in this field ? can you please clarify this thanks in advance.

I am using mac , is there any way to use Self hosted Integration Runtime ?as it shows .exe file if i download

I use Mac OS, this works there ?

Hello, how do I make the connection as I am using windows authentication. I have been stuck here. Thanks

I have get many clairification about azure data engineering in this channel. Thax sir 😀

Useful Channel 😍

wow loved the explanation , to the point .Thanks for sharing the knowledge .

Hi, I'm unable to connect to database. I wrote my login, password and so on. Only difference was that I didn't login with key vault. What might be also the issue?

HI SIR CAN U BRING MORE VIDEOS ON PROJECT

Best ADF demo, thanks for sharing.

Hello bro, Thank u for replying to all my issues, I got one more error, "Check the linked service configuration is correct, and make sure the SQL Database firewall allows the integration runtime to access.
Login failed for user '<MyUserID>.,"
Please help me on this, Thanks in Advance

Hello bro, Thank u for the amazing content. When i try to create access policy. I have this information displayed on the screen,"Access policies not available.
The access configuration for this key vault is set to role-based access control. To add or manage your access policies, go to the Access control (IAM) page."
Please help me on this bro. Thanks in advance

Hi Sir, Excellent Video. Clearly explained. Please upload all other videos. Waiting. Thanks