Industrial-scale Web Scraping with AI & Proxy Networks

Thanks for making this video. I am actually working on a project where the users can add amazon products and look for price changes and also get notified with price changes. My objective was to learn web scraping.

really cool

OK that was way cooler than I thought

I feel like a gangsta...finding ways around data collection for my business.

i guess jungle scout or other product hunting tools work in same way

Do you have a code where I can scrape ecom websites like shopify stores only and not other websites that is not shopify? I have a software where I can test and check the websites if they are alive or not. So I can use that if they are still on the market. Hope you can help

Thanks for alerting me to Puppeteer so I can block it. Content theft is despicable.

You forgot a very important fact: tools like Puppeteer or Playwright can work perfectly on a local development environment but make it work on production, on an actual server, can be an absolute nightmare. 😠

Awesome.

Thank you for teaching me puppeteer and bright data, beats all content on internet

have this
issue - SyntaxError: Cannot use import statement outside a module

If I keep rotating proxy while scraping, is it 100% guaranteed my ip will not get blocked?

That is a cool website to use. I'll try it one day

this would be a great use for the new "using" keyword, am I right?

Virgin rate limited API user vs Chad web scraper

Microbots AI chrome extension helps with building prompt with HTML code included. Chech it out it you want to write automation code faster.

I'm here because I need to hire someone who can provide this service for me. Great video!

good ol console.log

Just create a drop shipping master class and I'll buy it anyway from fireship lmao 🤣

Thank you sir

Great video, i have a question for you, how do you know that this is the industry standard for modern web scraping?
Like how can you find out this information.

Hi everyone, just wondering, can a company measure or track how much scraping is going on? Like Elon Musk just said there's a lot going on Twitter, how is he measuring that?

Do all search engines do it like this? I don't think that a website for searching furniture from my country bothered to talk to each one of the sellers and make an arrangement with them. Or did they?

the way you started the video reminded me of the XML pipedream, to have the data in xml and then apply xslt styles this would give easy access to the data, then it failed and Microsoft got their little black heart broken and Java took over XML and turned it multiple torture devices called "schemas" and also SOAP became a thing to torture even more developers. Until yet another fad appear on site, JSON, Javascript the good parts.

I do all this with proxychains and python.

🤔is Twitter accusing others of doing what they are doing to deflect attention and play de victim?

Can you also make a video on how we can use web scrapping to extract all betting related data from different football related betting sites

It took Twitter limiting my tweets for me to learn that this was a thing. Maybe Elon was right. If I get off Twitter I’ll actually leave something😂

ㅤ

Would it be possible to scrape base file types from a website to access their asset?
For example; there's a T-shirt image that I want to save, but I can only save as a .avif file.

Ideally, I'd be able to access the underlying file type (png/jpg) and save it in full resolution.
If anyone has any feedback regarding if advanced web scraping can extract this, please lmk.

IDK what heppn, but it was funni. !

Is this also possible for scaping Linkedin data?

I would use selenium but without a head.

who has a solution for me to SCREEN record at scale? i need to be able to record myself scrolling down a page with Puppeteer & create MP4s at scale

Too much meme video make me feel headache

Love how the example of chatgpt writing perfect code on the first try instantly debunks everyone who claims chatgpt isn't useful for programmers.

Gold. Just pure gold.

I am working on bot that monitors the button when it appear and presses it, and also reload the page every 15s but I am facing a problem (You have been over-freshening the web page. For security reasons, you are blocked) here is a way to change my IP so that I am not blocked from site

Does Python requests get blocked by Amazon

As someone with 20m a lot of this math is way off. You forgot hotels, which is by far the most expensive one. If you travel for work it can be $2000-$3000 a night for 5 star hotels which will get you to 200k a year.

Then if you have family and friends they all want help and that can be 500k+ a year. Private jets are more like 8k an hour too, especially if you cross the Atlantic

I wonder if there's an easy way to make a rotating ip network proxy...

Can't be that hard right?

NOT literally digging, but ok. Bright-App...one of those companies that makes you schedule a demo...Fuuuuuuuuuuuuuuuuuuck that.

does everyone just live with their mom too?

thanks

great video!

What's the difference between Selenium and Puppeteer?

Web scraping is nice.... but, more often than not is it prohibited by the Ts & Cs of the sites worthwhile scraping. So in the end, it all boils down to one's ethics and use case...

I think talking about LXML is worth it for the speed. If I scrape 100 websites