Extracting Text from HTML with Python

Your objective is to extract text from an HTML file in Python, replicating the output you'd obtain by copying the text from a browser and pasting it into a text editor.

Challenges

Regular expressions are not robust enough for poorly formed HTML. While Beautiful Soup is often recommended, it can pick up unwanted content like JavaScript and fail to interpret HTML entities.

Promising Alternative: html2text

Although it produces markdown instead of plain text, html2text handles HTML entities correctly and ignores JavaScript. However, its documentation and examples are limited.

Optimal Code for Text Extraction

The code below offers an effective solution that filters out unwanted elements and preserves HTML entities:

from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "http://news.bbc.co.uk/2/hi/health/2284783.stm"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")

# Remove scripts and styles
for script in soup(["script", "style"]):
    script.extract()

# Extract text
text = soup.get_text()

# Convert line breaks and remove whitespace
lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

Dependency

To use this code, you'll need BeautifulSoup4 installed with:

pip install beautifulsoup4