How to build an ETL pipeline with Python | Data pipeline | Export from SQL Server to PostgreSQL

Thanks very much for your invaluable contribution towards eternal learning process.

Hello Haq,

I keep having these 2 errors while executing the .py file in VS:

1."Error Locating Server/Instance Specified [xFFFFFFFF]."
Already stackoverflowed it and changed to Automatic the SQL Browser startup mode. The Service is also running now.

2.Also there's this one: "Server is not found or not accessible. Check if instance name is correct and if SQL Server is configured to allow remote connections"

For this 2nd one I did two changes: First, I checked the name of the server with query "SELECT @@SERVERNAME;" and it checks out when I found it in Server Properties in SSMS. Your server name looks quite different from mine ( laptop name). I slightly think my issue comes from my server name somehow.
Secondly, I allowed remote connections under the menu Server Properties in SSMS. I think I'm quite close. Any suggestions?

Hope I presented the question in a neat and understandable way.
In any case, congratulations and many thanks for these series of tutorials, I'm quite close to make it!

Cheers,

NB: I've completed with same results as you the 2 previous videos on SQL Server and Postgre installation, thanks again!

txs man!!!

Thank you so much

Why not use the SQL functions in pandas?

Thanks for sharing 😀

Hi,bro. Video so cool
I have trouble, can you help me. i installed msodbcsql. Tks you so much
Extract error:('IM002', '[IM002] [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified (0) (SQLDriverConnect)')
Data Extract Error: 'src_connect' is not defined

Fantastic videos, short and to the point! Great work, thank you for sharing!

Thanks

Thank you for this please continue to make these informative videos, you are a very good illustrator

This is cool! But isn't this more of an Extract and Load process than Extract, Transform and Load process? Most of the time while creating the Data Marts transform scripts is a heck and long. Also in the same scenario the Load procedure is mainly for insert query of records than tables. Nevertheless good video!

Sometimes int will auto convert to float in pandas and get error

Any suggestions or available code which can copy the compatible data type from MySQL to postgres..?? Please respond

Hey, great video. I was wondering if this would be a good guide for creating a basic pipeline project for job interviews’s like Data Analysis/Engineering? Thank you.

Exactly what I was looking for.Very well done! Thank you

Data sources are sap hana and sql server. My target table is same sql server. How can I perform upsert. I do have primary key on target table but it might be possible other columns might update in future . Please help

Thanks! Subscribe and like buttons done.

Where is the transform part

houly shit

I'm impressed!

I'm still don't see benefit from it why need to make it like this if we can extract from source dirctly

Such an easy to follow video, fantastic tutorial!

What is the ETL tool you used data injection and extraction?

Is this also working if ODBC Driver will be used?

Hi bro, very informative session.. am working on the ETL QA framework creation using pyspark. for that I have to create a directory structure in pycharm.. is there any reference video you have created? Please share

Hello, I am getting the following error.

Data extract error: ('28000', "[28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'etl'. (18456) (SQLDriverConnect); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'etl'. (18456)")
Error while extracting data: local variable 'src_conn' referenced before assignment

driver is installed and I have done the other troubleshooting steps found in the previous comments.

Thanks a lot !

great great video

THIS IS AMAZING THANKS!!!
HELP!!!!

1)I did the same thing and it got importeed into postgres, But i need the column data types to be in specif cformat for example Datetime, VARCHAR, DOUBLE ,INT not these text,bigint etc. Please suggest what should i do!!

2) Data is same but the data types are different, i tried changing the data type in the data frame and created custom columns with CREATE sql query but it doesnt matter as pd.to_sql always replaces that table adn creates a new one.

So what should i do?
Thank you

If my destination is Dimentions and fact table then I need to write separate method for table or is there any other way

Hi sir,
Can you make a video how to migrate data from SQL server to snowflake

I encountered this error while running the file: raise KeyError(key) from None at code 'uid' and 'pwd'. Could you help me with this? Also a very nice video! Thank you

hai friend.. I have problem. in the sql server name is ENGINEERDATA\ENGDATA and when I'm declared variable ='ENGINEERDATA\ENGDATA' in code when I run the scripts always appear pycopg2.operationalError couldn't translate host name "engineerdata\ENGDATA" to address :unknow host.
please help me

This is just EL.

So cool!! where does transform happen?

Helpful. Thank you for this video

hello sir I got some error and I don't get any clue through it after trying other alternative.

Data extract error: ('28000', "[28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'etl'. (18456) (SQLDriverConnect); [28000] [Microsoft][SQL Server Native Client 11.0][SQL Server]Login failed for user 'etl'. (18456)")error
Error while extracting data: local variable 'src_conn' referenced before assignment


I really appreciate your help!!!!

Perfect explanation. 👍👍

Excellent presentation. Keep it up.

Explicação muito boa. Vc tem algum tutorial extraindo dados de um ERP?

Top tier staff this!!!

I was looking for this. Thankyou!!. One question about scheduler. If script is run every day at certain time, everytime script ll run the query "select * form table" query? This means whole dataset in database is loaded everytime when script runs?

great stuff thanks

amazing Video, I followed all but I had this error and could not find solution
Data extract error: ('08001', '[08001] [Microsoft][ODBC Driver 17 for SQL Server]Neither DSN nor SERVER keyword supplied (0) (SQLDriverConnect); [08001] [Microsoft][ODBC Driver 17 for SQL Server]Invalid connection string attribute (0)')
Error while extracting data: local variable 'src_conn' referenced before assignment