Skip to content

Instantly share code, notes, and snippets.

@XDo0

XDo0/crawl_url.py

Created December 31, 2021 02:21
Show Gist options
  • Save XDo0/1958fe8f7827bb1467b2e19137e57b53 to your computer and use it in GitHub Desktop.
Save XDo0/1958fe8f7827bb1467b2e19137e57b53 to your computer and use it in GitHub Desktop.
Download ZIP
python request a url & Beautifulsoup spider
Raw
crawl_url.py
from bs4 import BeautifulSoup,SoupStrainer
import requests
def crawl_url(keyword):
url=f"https://www.google.com/search?q={keyword}"
response = requests.get(url)
soup = BeautifulSoup(response.content, 'lxml')
all_tags=soup.find_all("div",attrs={"data-version-added":True})
# all_catas=soup.find_all(class_="devsite-nav-text")
#...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment