Dynamic Javascript Scraping - Web scraping with Beautiful Soup 4 p.4

Hey guys.. please read the problem here: I use usually Selenium to scrape data, but now I'm facing a website that identifies Selenium and blocks its JavaScript functionality so as to not reveal the data I need. Like for the first 10 times, it gives the proper data, then it just blacklists any approach with Selenium and gives no proper response.
Hence, I used the BS4 module and the approach discussed here (with PyQt5 though), and the Website worked only ONCE! And then again it just gives the 'source' HTML without any dynamic data. How is it possible??? Do websites recognize PyQt calls etc.???
What do I do?? Please help guys (especially sentdex if you are still getting this!)

Hi Sentdex, thanks for great explanation, but I am not able to import PyQt4.I tried in both OS: Windows as well as Linux. It seems now PyQt5 is also available. I am able to install these but I am not able to import the same.

Request you to please create a lecture video to install and import PyQt4

Thanks a lot. I have encountered this error (I am using pycharm)
ModuleNotFoundError: No module named 'PyQt5.QtWebEngineWidgets'
Any ideas?

Works fine with PyQt5, Thanks for tutor

Hi there, I want to get all URLs from a domain, but I don't know how to do.... can you suggest me something? Thanks for reading!!!

I have seen your all videos. Sir I wanted to identify the computer subject sites only. I wanted to build such tool which can recognise only computer subject sites.

pip install PyQt5 for windows

its WOW... i spent whole lot of time scraping dynamic content but couldn't. This was a very good idea.

How do you get this code to run within a loop ?
I've managed to make it work for a single url, but once I put the bottom portion in a for loop, I only get it to run 2 times and then the python Shell restarts. Anyone ?

I was banging my head with all those headless browser methods to run javascript. This is so much simpler. Thanks man! appreciated!

This is great, however in my situation, the website is adding content every 1 minute, how do I loop and reload just the page without reloading the PyQt client every loop?

How about Selenium

Any one try this method but using a for loop to do the same work on multiple pages? It works on the first loop and then gets stuck on the "self.loadFinished.connect(self.on_page_load)" part. I have tried everything to get it to work but no luck.

Seriously what's the point of even using QApplication ! Can I remove that >_>

Wait WTF we can do multiprocessing in python damn I didn't knew thanks for telling bro !!!

I installed PyQt5 for Python 3.7 (x64) but getting this error in VS Code: that "No module named 'PyQt5.QtWebKit'". Could you please suggest any solution?

You are GOD.. thank you

I've installed pyqt5 and these statements aren't working for me:


from PyQt5.QtWebKitWidgets import QWebPage
from PyQt5.QtWebKitWidgets import QWebView
from PyQt5.QtWidgets import QApplication
from PyQt5.QtCore import QUrl


is there any solution to it?

Hey sentdex.. please help me.
In my case the html is generated dynamically through ajax call. With this code, I am not able to scrape the required data. Is there any way through which I can wait till the ajax call is made? i have tried qWait but it did not worked.

i think it's me, didn't understand a word. Selenium is much better than this..

Is it possible to scrape the pinned location from an Embedded google map which loads all its data using the jaavscript?
The problem is that the location and data that i want to fetch is only load up for that location when we click on the particular location

Thanks in advance

Some of the words are actually hard to listen. It seems like your sound intensity gradually decreases before completing a sentence. Nonetheless your videos are really helpful.

Could you please explain the reason for using PyQT and not anything else here? Also state the alternatives. Thank You.

You can't imagine how your tutorials are useful to me. I'm really thankful and hope to you continue making videos like this

How can I get the Source Code showed in this video ? It could be faster than retype all :) Thanks

How do I scrape content of pseudo elements like ::before and ::after?

Why not use selenium

I've done something similar yesterday with PyQt5. I've combined html, javascript and python into one app (and some css goodies)

I was just searching for a problem with this and BAM, u have an entire series on webscraping. I think its the 5th time this happens. Just sayin realy appreciate ur channel.

Selenium seems like a better option for scraping dynamic webpages

It's better to use selenium webdriver (headless) instead of using PyQt to run JavaScript...

Hi, I have been working a lot lately on web scraping tasks and I was using selenium as it required interaction with the web page. My question is there a generic or more common way to extract any web page content instead of navigating and identifying tags which has required information. If not, why?
Also looking for how to control sending multiple requests to a server at a time while trying to fetch the data so that it would not stop taking my requests.

Great Tutorial Chum! Many thanks.

I tried the PyQt5 equivalent to this, but I am not getting the expected results. The dynamic content still cannot be extracted. Any suggestions?

Using pyqt5 im getting Unresolved reference 'Client', i have the same code as the tutorial

Could you please make an update video of this? PyQt has had a few updates or there is other modules to use. I'm trying to do it using selenium because I feel like it is the best for what I want but I just can't pass the "verify your identity" bs since webdriver doesn't take headers, and I haven't found a different way to do it. Thank you!!!

in 2021 I am unable to install PyQt4 on the latest version of Python 3.9. I use PyCharm under Windows 10 and just can't figure out how to get it to install. Any ideas would be greatly appreciated.

How to resolve content security error ,
I'm scrapping LinkedIn page

Spyder is not launching after installing PyQt

but, this is'nt working with pyqt5 and I'm unable to install pyqt4. What's the solution???

Why can't you just parse the script tag instead of the p tag?

If it is possible I would like give this video thousands of likes