Python 3 Programming Tutorial - Parsing Websites with re and urllib

Hello Sentdex, can you give a hints where i shold start to search. this is my story. I have a project in the project user can post image or text. now other user like the post and wanted to share the post to her own or him wall just like facebook. please and thank you,,

wow 1M sub

Thank you

thanks god

Amazing. Love you. You make parsing so easy to understand.

Thank you bruh ❤

Great Bruh

how to replace spaces using symbol in python using regular expressions

I tried this script on a different URL and got a Forbidden 403 error...do some websites block parsing via script?

@sentdex Hi there!! Thanx for your great tutorial! I'm a newbie on python and programming in general and I have a problem right now that's kinda like what you show here. I've extracted a table from a website (using the api) and the results come in text (csv). I get around 20 different statistics (it's sports-related) and I only need 3 of them. So I would like to eliminate all the data that I don't need and just get those 3. Would you recommend the same Library modules (re and urllib) or another module to do that? As I said, it looks to be the same kinda thing you're showing here, the difference being that I need to basically remove stats instead of text when I scrape it and just get the one I need. Thanx again for your great tutorials!!

Hi,
Thanks for these series of tutorials.l am new in this field and need help.I'm trying to write a code for scraping several pages of web and don't know how should I start?
I tried several times but hadn't true run☹I hope u help me🙏

how to save that file you have extracted ?

how do i get the full playlist, it's not in the user's profile. infact it is a totally different person but I want this guy!

import re
what was re tho? I'm trying to recall this part now and I can't remember what it is.

it does not print any thing in terminal i maybe know because "eachp"

data = urllib.parse.urlencode(values)
data = data.encode('utf-8')

These two lines. You assign different values to the same variable. How does that work?

So how does regex code work exactly? Is it one after the other? Would '.*?' yield different results than '*.?' or '?*.'?

Thanks for your video.
I have one question.. instead of specifying the sample URL in the code, would it be possible to make it via input?
what I mean is, I work with web based tools that contain same data fields with different values of course.. like support tickets lets say.
I want script where I can paste my ticket URL and then to be parsed for specific fields like ticket number, customer name, etc and populate the excel table with the parsed data
I have lot of tickets to deal with sometimes and opening all the URLs in separate tabs is just not an option so I'm trying to consolidate everything in excel file (for now) to quickly see which ticket is in what state, when they are scheduled, etc.

How would i do this in django?

It may work on p but for scraping useful stuff like links it gets tricky especially if you wanna get the href and also the value inside the tags.
I did use a library for that before but now I wanna try without.
EDIT: nvm, doing a second findall on the result of the first for further filtering does it. Also you could use those url results to traverse through all the results and filter those as well... hmm
Thanks, good tutorial.

please process json data using urllib and string slicing

content = []
paragraphs = re.findall(r'<p>(.*?)</p>', str(respData))
for eachP in str(paragraphs):
content.append(eachP)
sentence = ''.join(content)

* This just cleans the output a little more so you are not reading in like a downwards fassion

Thanks for the vid. Can anyone help me on how to send username and password to handle an authentication popup to automate it in chrome?

You are awesome!

@sentdex

values = {'s': 'basics',
'submit': 'search'}

I have tried to put some other links but It does not work, it only works with the link that you posted

Instead of importing urllib.request and urllib.parse individually, is it possible to just import urllib as a whole library?
In the same respect, since in the last vid you said you mostly only use re.findall() , can we just import re.findall instead of the whole re library module?

This is AWESOME! Thanks a lot!

Hi, great video!

I just have a question, when you do this it doesn't save the webpage as "Complete" but rather as "HTML, only". Is there a way to do Complete using urllib?

how can you do this with "google" i am not able to achieve this with google. it's just blank after execution.
but i'm curious to read the para data or any normal English data in the Html source code of google.

Hi and thank you for the great tutorial.
I have extracted my paragraphs as you said but inside the <p> tags there are so many <a href=".....">some stuff in between</a> !!
I want to some how delete the <a href="...."> junks as well . I don't know how in your work u didn't face them :D
let me know if you have any comment on this.
thanks in advance for all the great videos you have uploaded for everyone. ! :)

Could you explain how to parse HTML data which has two columns and have to go via login authtentication system

Help me plz when i run the program it gives me that error AttributeError: module 'urllib' has no attribute 'encode'

Awesome videos! Keep it up

recommendable contribution, appreciate your effort to teach others

thanks!!!!

Thank you for taking the time to make these videos... You are a great teacher

very good but how to integrate the fake id info so you can get into google with this?

Usually all give intoduction on single page website, what about the website which had 100 pages in it?? Try to make tutorial on it!!!

what happens of there's no closing (</p>) tag on the page?

been watching entire series. no clue whats going on lol. hope i can make my own tutorials one day

supper boss

Hi,

Great vedio. Wonderful explanation.

I have a small doubt.

I have to copy the website url which is currently opened in a browser using a python code instead of manually copy pasting the URL.

And assign it to the URL variable.

And use the code which is given by you in this vedio.

Please help me with the code to copy the URL using the python code.

Regards,
Jaideep.

make detailed lectures on url