Beautifulsoup vs Selenium vs Scrapy - Which Tool for Web Scraping?

Hello John, Thanks for the video!

I have two questions:
What the best tool for scrapper a website with login or autentication?

And, when the website use a api with autentication, what can i use?

I am new to scraping. If you have a dynamic website that requires you input dates or numbers and click on buttons, what else besides selenium works? Does Beutiful Soup work? Very interested.

Greatly informative, thank you.

thanks bro

I liked a very didactic explanation

I want to scrape data from infinite scrol website which library should I choose

Great video thank you. Thoughts on AutoHotkey?

Thanks for a great rundown of the options available for web scraping in Pyhton. There were a few that I was not familiar with.

what would you recommend for creating scraping tool ?

Very useful for beginners

most of the time what I do is use selenium to get me where I want then extract what I want by making soup of the page using beautifulsoup extracting specific tag info then afterwards using pandas to save the list data in data frame and exporting it out as a csv or excel .

do you have any videos on scrape masking?

what tool i can use to bypass blocking websaits if it understands that I m using automation tools?

You mentioned that selenium sends information about itself to websites being scraped, so that websites could detect that selenium is being used. I'm curious if you know more about this and any workarounds?

I have been using requests along with bs4 & I did heard about scrapy and I agree it isn't good for beginners so I was a beginner that time it really was daunting. But now I think it's time to scrapy time

Wish I had watched this before choosing Selenium for a scraping project. Really feel you hit the nail on the head. Great video!

Thanks for the whole content!
I got a question and it would be very helpful for me, if you can support:
I have to scrape a dynamic website. If I scroll down, more objects are loading to this page (always 50 new). When I look in "developers" of my browser, I find the Data I need in the folder "XHRs" and with every scroll for new 50 objects, there is a new file called "730" with the new 50 objects in json-format. I need all the 730-files.
do you know how to scrape them?

requests + bs4 VS scrapy?

Can you make a video about recent scrapy-playwright bug about implementing scrapy-playwright setting implementation and some books or resources to learn scrapy.