Slow Web Scraper? Try this with ASYNC and Requests-html

John thank you so much for your video, everything is explained in the most understandable way.
I tried using your teachings in scraping some html sites with excellent results.
However, I have big problems adding to the function code await response.arender() for scraping sites with data in javascript.
After running the script, I get no results because the script enters an infinite loop.
this is the complete function I use:

async def name (session,url):
response = await session.get(url)
await response.html.arender()
return response


When I use the code non-asynchronously and therefore not in a function, the code works fine:

session = AsyncHTMLSession()
response = await session.get(url)
await response.html.arender()

Can you please help me in fixing this problem?
Thanks a lot,
Luca


P.S.(Congratulations for your coding channel, absolutely the best!)

John Watson Rooney, can you explain, please, the method to render JavaScript using AsyncHTMLSession and asyncio?

Can this be incorporated with Scrapy?

another great video as always

DUDE, WHY ARE YOU SO SERIOUS?

Where code

Thank you John for this tutorial.
Could you suggest on how to approach getting the list of urls from the beginning?
Should I use another HTMLSession() to define a get_urls() function and return the list of urls? Or how should I use async/await with this case?
Again, thank you!

Thanks a lot for your video. This was exactly what I was looking for.

Thank you for the nice tutorial.

What if you have for example a dynamic list of URLs? The list could be updated by the scrape if it found a pagination on the target page. How would you create or manage tasks for something like that?

i've used this async, but just 3 sec more faster than manual looping use requests

so, it depends on the website server too?

Thank you for all your videos! Recently found your channel and wish I had seen it ages ago. These videos have been incredibly helpful for me. I would like to use this technique with a post request to sign in to a site, but I can't figure out where to place this to preserve the sign in throughtout the session. Any suggestions on where or how to place this?

Can you make another video about async and request-html more detail. Really need it hahaha

When webpage is dynamicly generated by javascript and we use render it stays so slow isn't? Or how to use render correctly that it will work faster/

Hey John, great video once again. How can i use this async way using proxies and different sessions?

Why did you not use session.render command after session.get(url) ? I am working on a project and my program runs fine without using threads or asyncio, but when I use threads or asyncio the render command doesnt works. Cna you help me understand how can I overcome this?

Thanks a lot for the video :)

How could we use arender() to render javascript html asynchronously??

This code does not close sessions at the end and leaves a lot of open chromiums (not all, but with 980 links, for example, it went over a hundred) I hoped that here I will find a solution to this problem, but here the same thing. And by the way, this option using asyncio is not suitable if you need to render something(i mean if you use arender), because then the result does not produce at all, all the code simply ends at the moment leaving open chromiums. But your other videos are good anyway.

Awesome and thanks for the amazing videos. Does this also works with Beautifulsoup?

This was very smooth for experienced programmer. Time flew and the information stuck

Hi Sir

it is showing this error on my side : RuntimeWarning: Enable tracemalloc to get the object allocation traceback

Kindly help me

Thanks!

I am stuck in a project ,
If you could help me out then please contact me

is there a way to scrape data from cloudflare protection website

i watch you past video its really help me the grequests module is this better than grequests

Just wanted to say this is the best coding channel I have ever stumbled upon. Teaching through actual examples, explaining only the necessary parts, trusting that the viewer has an actual brain for themselves. Perfect.

Thanks so much, can you please upload the code on git

Thank you. Excellent quality as always

Excellent as usual, Thanks a lot

Would you suggest this method over grequests?

Could you show an example of how to scrape a page that has lazyloaders? For example: Aliexpress. Thank you for all the free contents.

Very useful to speed up things.👍💖

Thanks John , always learn a thing or two from your videos.
Is there a high chance of an IP ban due to such a high volume of requests in such a small timeframe without mitigating with proxie rotation / headers etc?

Hey you’re one of the best, do u have discord? I want to contact you about a business opportunity. Thanks

John, thank you for your time on making these vids, appreciate your time and do know your time is worth it...

How can I extract all links in a Javascript website using request-html?? Can I automate the web scraping in Javascript driven html website??

Thanks a lot