HTML 結(jié)構(gòu)解析是 Web 爬蟲中的核心技能之一，它允許你從網(wǎng)頁中提取所需的信息。Python 提供了幾種流行的庫來幫助進(jìn)行 HTML 解析，其中最常用的是 BeautifulSoup 和 lxml。

1. 安裝必要的庫

首先，你需要安裝 requests（用于發(fā)送 HTTP 請求）和 beautifulsoup4（用于解析 HTML）。可以通過 pip 安裝：

pip install requests beautifulsoup4

2. 發(fā)送 HTTP 請求并獲取 HTML 內(nèi)容

使用 requests 庫可以輕松地從網(wǎng)站抓取 HTML 頁面：

import requestsurl = "https://www.example.com"
response = requests.get(url)# 檢查請求是否成功
if response.status_code == 200:html_content = response.text
else:print(f"Failed to retrieve page, status code: {response.status_code}")

3. 解析 HTML 內(nèi)容

接下來，使用 BeautifulSoup 解析 HTML 內(nèi)容：

from bs4 import BeautifulSoupsoup = BeautifulSoup(html_content, 'html.parser')

這里的 'html.parser' 是解析器的名字，BeautifulSoup 支持多種解析器，包括 Python 自帶的標(biāo)準(zhǔn)庫、lxml 和 html5lib。

4. 選擇和提取信息

一旦你有了 BeautifulSoup 對象，你可以開始提取信息。以下是幾種常見的選擇器方法：

• 通過標(biāo)簽名：

  titles = soup.find_all('h1')
  

• 通過類名：

  articles = soup.find_all('div', class_='article')
  

• 通過 ID：

  main_content = soup.find(id='main-content')
  

• 通過屬性：

  links = soup.find_all('a', href=True)
  

• 組合選擇器：

  article_titles = soup.select('div.article h2.title')
  

5. 遍歷和處理數(shù)據(jù)

提取到數(shù)據(jù)后，你可以遍歷并處理它們：

for title in soup.find_all('h2'):print(title.text.strip())

6. 遞歸解析

對于復(fù)雜的嵌套結(jié)構(gòu)，你可以使用遞歸函數(shù)來解析：

def parse_section(section):title = section.find('h2')if title:print(title.text.strip())sub_sections = section.find_all('section')for sub_section in sub_sections:parse_section(sub_section)sections = soup.find_all('section')
for section in sections:parse_section(section)

7. 實(shí)戰(zhàn)示例

讓我們創(chuàng)建一個完整的示例，抓取并解析一個簡單的網(wǎng)頁：

import requests
from bs4 import BeautifulSoupurl = "https://www.example.com"# 發(fā)送請求并解析 HTML
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')# 找到所有的文章標(biāo)題
article_titles = soup.find_all('h2', class_='article-title')# 輸出所有文章標(biāo)題
for title in article_titles:print(title.text.strip())

這個示例展示了如何從網(wǎng)頁中抓取所有具有 class="article-title" 的 h2 元素，并打印出它們的文本內(nèi)容。

以上就是使用 Python 和 BeautifulSoup 進(jìn)行 HTML 結(jié)構(gòu)解析的基本流程。當(dāng)然，實(shí)際應(yīng)用中你可能需要處理更復(fù)雜的邏輯，比如處理 JavaScript 渲染的內(nèi)容或者分頁等。

在我們已經(jīng)討論的基礎(chǔ)上，讓我們進(jìn)一步擴(kuò)展代碼，以便處理更復(fù)雜的場景，比如分頁、錯誤處理、日志記錄以及數(shù)據(jù)持久化。我們將繼續(xù)使用 requests 和 BeautifulSoup，并引入 logging 和 sqlite3 來記錄日志和存儲數(shù)據(jù)。

1. 異常處理和日志記錄

在爬取過程中，可能會遇到各種問題，如網(wǎng)絡(luò)錯誤、服務(wù)器錯誤或解析錯誤。使用 try...except 塊和 logging 模塊可以幫助我們更好地處理這些問題：

import logging
import requests
from bs4 import BeautifulSouplogging.basicConfig(filename='crawler.log', level=logging.INFO, format='%(asctime)s:%(levelname)s:%(message)s')def fetch_data(url):try:response = requests.get(url)response.raise_for_status()  # Raises an HTTPError for bad responsessoup = BeautifulSoup(response.text, 'html.parser')return soupexcept requests.exceptions.RequestException as e:logging.error(f"Failed to fetch {url}: {e}")return None# Example usage
url = 'https://www.example.com'
soup = fetch_data(url)
if soup:# Proceed with parsing...
else:logging.info("No data fetched, skipping...")

2. 分頁處理

許多網(wǎng)站使用分頁顯示大量數(shù)據(jù)。你可以通過檢查頁面源碼找到分頁鏈接的模式，并編寫代碼來遍歷所有頁面：

def fetch_pages(base_url, page_suffix='page/'):current_page = 1while True:url = f"{base_url}{page_suffix}{current_page}"soup = fetch_data(url)if not soup:break# Process page data here...# Check for next page linknext_page_link = soup.find('a', text='Next')if not next_page_link:breakcurrent_page += 1

3. 數(shù)據(jù)持久化：SQLite

使用數(shù)據(jù)庫存儲爬取的數(shù)據(jù)可以方便后續(xù)分析和檢索。SQLite 是一個輕量級的數(shù)據(jù)庫，非常適合小型項目：

import sqlite3def init_db():conn = sqlite3.connect('data.db')cursor = conn.cursor()cursor.execute('''CREATE TABLE IF NOT EXISTS articles (id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT NOT NULL,author TEXT,published_date DATE)''')conn.commit()return conndef save_article(conn, title, author, published_date):cursor = conn.cursor()cursor.execute('''INSERT INTO articles (title, author, published_date) VALUES (?, ?, ?)''', (title, author, published_date))conn.commit()# Initialize database
conn = init_db()# Save data
save_article(conn, "Example Title", "Author Name", "2024-07-24")

4. 完整示例：抓取分頁數(shù)據(jù)并保存到 SQLite

讓我們將上述概念整合成一個完整的示例，抓取分頁數(shù)據(jù)并將其保存到 SQLite 數(shù)據(jù)庫：

import logging
import requests
from bs4 import BeautifulSoup
import sqlite3logging.basicConfig(filename='crawler.log', level=logging.INFO)def fetch_data(url):try:response = requests.get(url)response.raise_for_status()return BeautifulSoup(response.text, 'html.parser')except requests.exceptions.RequestException as e:logging.error(f"Failed to fetch {url}: {e}")return Nonedef fetch_pages(base_url, page_suffix='page/'):conn = sqlite3.connect('data.db')cursor = conn.cursor()cursor.execute('''CREATE TABLE IF NOT EXISTS articles (id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT NOT NULL,author TEXT,published_date DATE)''')conn.commit()current_page = 1while True:url = f"{base_url}{page_suffix}{current_page}"soup = fetch_data(url)if not soup:break# Assume the structure of the site allows us to find titles easilytitles = soup.find_all('h2', class_='article-title')for title in titles:save_article(conn, title.text.strip(), None, None)next_page_link = soup.find('a', text='Next')if not next_page_link:breakcurrent_page += 1conn.close()def save_article(conn, title, author, published_date):cursor = conn.cursor()cursor.execute('''INSERT INTO articles (title, author, published_date) VALUES (?, ?, ?)''', (title, author, published_date))conn.commit()# Example usage
base_url = 'https://www.example.com/articles/'
fetch_pages(base_url)

這個示例將抓取 https://www.example.com/articles/ 上的分頁數(shù)據(jù)，保存文章標(biāo)題到 SQLite 數(shù)據(jù)庫。注意，你需要根據(jù)實(shí)際網(wǎng)站的 HTML 結(jié)構(gòu)調(diào)整 find_all 和 find 方法的參數(shù)。

既然我們已經(jīng)有了一個基本的框架來抓取分頁數(shù)據(jù)并存儲到 SQLite 數(shù)據(jù)庫中，現(xiàn)在讓我們進(jìn)一步完善這個代碼，包括添加更詳細(xì)的錯誤處理、日志記錄、以及處理動態(tài)加載的網(wǎng)頁內(nèi)容（通常由 JavaScript 渲染）。

1. 更詳細(xì)的錯誤處理

在 fetch_data 函數(shù)中，除了處理請求錯誤之外，我們還可以捕獲和記錄其他可能發(fā)生的錯誤，比如解析 HTML 的錯誤：

def fetch_data(url):try:response = requests.get(url)response.raise_for_status()soup = BeautifulSoup(response.text, 'html.parser')return soupexcept requests.exceptions.RequestException as e:logging.error(f"Request error fetching {url}: {e}")except Exception as e:logging.error(f"An unexpected error occurred: {e}")return None

2. 更詳細(xì)的日志記錄

在日志記錄方面，我們可以增加更多的信息，比如請求的 HTTP 狀態(tài)碼、響應(yīng)時間等：

import timedef fetch_data(url):try:start_time = time.time()response = requests.get(url)elapsed_time = time.time() - start_timeresponse.raise_for_status()soup = BeautifulSoup(response.text, 'html.parser')logging.info(f"Fetched {url} successfully in {elapsed_time:.2f} seconds, status code: {response.status_code}")return soupexcept requests.exceptions.RequestException as e:logging.error(f"Request error fetching {url}: {e}")except Exception as e:logging.error(f"An unexpected error occurred: {e}")return None

3. 處理動態(tài)加載的內(nèi)容

當(dāng)網(wǎng)站使用 JavaScript 動態(tài)加載內(nèi)容時，普通的 HTTP 請求無法獲取完整的內(nèi)容。這時可以使用 Selenium 或 Pyppeteer 等庫來模擬瀏覽器行為。這里以 Selenium 為例：

from selenium import webdriver
from selenium.webdriver.chrome.options import Optionsdef fetch_data_with_js(url):options = Options()options.headless = True  # Run Chrome in headless modedriver = webdriver.Chrome(options=options)driver.get(url)# Add wait time or wait for certain elements to loadtime.sleep(3)  # Wait for dynamic content to loadhtml = driver.page_sourcedriver.quit()return BeautifulSoup(html, 'html.parser')

要使用這段代碼，你需要先下載 ChromeDriver 并確保它在系統(tǒng)路徑中可執(zhí)行。此外，你還需要安裝 selenium 庫：

pip install selenium

4. 整合所有改進(jìn)點(diǎn)

現(xiàn)在，我們可以將上述所有改進(jìn)點(diǎn)整合到我們的分頁數(shù)據(jù)抓取腳本中：

import logging
import time
import requests
from bs4 import BeautifulSoup
import sqlite3
from selenium import webdriver
from selenium.webdriver.chrome.options import Optionslogging.basicConfig(filename='crawler.log', level=logging.INFO)def fetch_data(url):try:start_time = time.time()response = requests.get(url)elapsed_time = time.time() - start_timeresponse.raise_for_status()soup = BeautifulSoup(response.text, 'html.parser')logging.info(f"Fetched {url} successfully in {elapsed_time:.2f} seconds, status code: {response.status_code}")return soupexcept requests.exceptions.RequestException as e:logging.error(f"Request error fetching {url}: {e}")except Exception as e:logging.error(f"An unexpected error occurred: {e}")return Nonedef fetch_data_with_js(url):options = Options()options.headless = Truedriver = webdriver.Chrome(options=options)driver.get(url)time.sleep(3)html = driver.page_sourcedriver.quit()return BeautifulSoup(html, 'html.parser')def fetch_pages(base_url, page_suffix='page/', use_js=False):conn = sqlite3.connect('data.db')cursor = conn.cursor()cursor.execute('''CREATE TABLE IF NOT EXISTS articles (id INTEGER PRIMARY KEY AUTOINCREMENT,title TEXT NOT NULL,author TEXT,published_date DATE)''')conn.commit()current_page = 1fetch_function = fetch_data_with_js if use_js else fetch_datawhile True:url = f"{base_url}{page_suffix}{current_page}"soup = fetch_function(url)if not soup:breaktitles = soup.find_all('h2', class_='article-title')for title in titles:save_article(conn, title.text.strip(), None, None)next_page_link = soup.find('a', text='Next')if not next_page_link:breakcurrent_page += 1conn.close()def save_article(conn, title, author, published_date):cursor = conn.cursor()cursor.execute('''INSERT INTO articles (title, author, published_date) VALUES (?, ?, ?)''', (title, author, published_date))conn.commit()# Example usage
base_url = 'https://www.example.com/articles/'
use_js = True  # Set to True if the site uses JS for loading content
fetch_pages(base_url, use_js=use_js)

這個改進(jìn)版的腳本包含了錯誤處理、詳細(xì)的日志記錄、以及處理動態(tài)加載內(nèi)容的能力，使得它更加健壯和實(shí)用。