Scrape Data from Multiple Web Pages with Power Query

Great resource! I am curious about replacing 1 with "&PageStart&". Can you explain why we use the double quotes coupled with the double ampersand? Which language/grammar are we following here, M or HTML or something else? I just wanted to learn more coding rules so I can crack the query more freely. I would appreciate any help you could provide.

Wow, this is clever and exactly what I needed. My mind is blown !!

I still need to watch this video a few times. Our entire organization dont know this i bet

Exclusive...

Excellent tutorial, super easy to follow. That’s brilliant 👍

Thanks!

I can't get PQ or BI to extract embedded URL in a webpage table. eg email is embedded in person's name? Any ideas?

brilliant idea

Thus is AWESOME!!

I don´t know you, but I love you. thanks!

Is it possible to scrape the URL of each individual book? If yes, how can't it be done?

It's usefull. Thanks you. I am looking for silimilar data scraper software. Do you mind to show me how to work with power BI in the case with differences website please.

As always, an excellent tutorial

This is simply awesome, now I have to practice this technique.

Can you provide me with an example of using a single Power Query connection to scrape multiple HTML elements from a webpage and load them into corresponding cells?

I slightly adjusted this to scrape data from a folder full of PDF files. Excellent thanks!

How did you come up with 21610 for the list of numbers?

What if i want to pull all the data available but i dont know how many pages there is?

Great example

Hi. When I get data from web it only extracts a small fraction of the data on the web page. Why is that? I also do not have the option to add table using examples. Excel 2021

Woow this is brillant :D
i have question, this method can work in any website am i right? so, by learning this i don't have to figure way in Pyathon or other web scraper data am i right?

Thanks in a million.

The problem I have is that I want to capture the URLs and I keep getting "No CSS"

How do you get URLs from a web page?

Awesome use of M for us tiptoeing into the M Script!

I want to scrape multiple links list 😶

The website is not anymore updated

Hi! Your tutorial is very clear. However, what if the web page you are trying to access needs your credentials first? Do you know how I can go around that? Thank you!

Good afternoon, i followed your instructions however instead of producing the results from the subsequent URL pages it just mirrored the results from the first page. Any ideas? Thanks, Steve

Hi Mynda, I'm trying to scrape data from Amazon but the website is not moving forward from Accept Cookies. Any help

Hi Mynda. Thank you for sharing it. Very useful. However, Is there any way to get the actual URL since the position keeps changing whenever I refresh data in Power BI.

Beautiful. It's solved my actual problem. Thx. :)

maybe someone can help me here. I need to do something similar to this, simpler in terms of form, since on the web the table I need already has the table format, but the problem is that the url address is not modified when changing pages to see other parts of the table. I hope someone knows how to proceed in this situation. Thanks in advance.

hey, I don't have a start parameter in my url

Amazing how it is easy to scrape web pages. Thanks for this excellent tutorial.

To think I was doing this manually 🤦🏽‍♂️. Thank you, this is a huge time saver!

Wow...Easily used this tutorial to query printer settings from every Zebra printer on my LAN. Very helpful!

Great tutorial! Helped me a lot. But do you have any idea, why "Add Table Using Examples" won't work and throws this message: "This Stencil app is disabled for this browser"?

hi is it possible to scrape data to a list of url's using power query?

Is it possible to have more than one parameter?

Great video thanks this makes web scraping a lot easier. Thank you.

How can we pull data from a web page that publishes daily. Web does not have date as field.
We need the excel to be updated daily with generated data daily as a table so we have a history and date table

Mam can u please tell me how to scrape data from multiple websites which have different url format at once

Hi! Nice video. I just want to know if there is a way to extract the paragraph that is inside some products using power query also. I was trying but power only extract information in the outside, that is visible, is there a code or a formula to do this? Thanks!

When we assume that bookstore extend the number of pages in the time, how can I set it up so that query will check all pages available. I can not set it up so, because when it checks the urls which does not exists yet, it will stop sraping procedure. Is it possible to fix it somehow?

Thanks a lott!! I was wondering if the web page is updated would the loaded data in power bi update too (so basically if it's real time or not)

Thank you soooo much! You changed my life this weekend. Been struggling with Excel's limitations for years, and lost countless hours of my life sometimes without even accomplishing my goal. I only discovered the existence of Power Query last night with your video, and you blew my mind. A brilliantly well presented and comprehensive video on it too! It got me partway through my current problem, but now I'm stuck again if you can help?

I've created Query1 to gets multiple tables from each webpage with 10 records each , and includes a record ID. But each record has a link to a details page for more info for that record. The record ID is used within the URL string to get those details. Can I create a single query that collects the list of records and uses the ID to also collect the details for each record all in one go?

Also, with 30,000 records in total, it takes hours to refresh. However, as the historic records don't change, and have a historic date of filing, is there any way for future updates to only get and append the latest records (with a filing date after the last date of the previous dataset, whilst removing any duplicates, and append it to the list?

Finally, it would be great if a timestamp could be added in an additional column to denote the date when that query was run, so that I can easily see which data has been added and when. Is any of this possible with PowerQuery?

What if my URL page is a search page how can I scrap the data from it. Cause URL only has text and no tables?