Skip to content

XPath and HTML

XPath selectors are a common source of brittle, hard-to-debug crawlers; the helpers in zavod.helpers.html add typed return values and explicit count guards on top of lxml.

Use the typed helpers in zavod.helpers.html

lxml's Element.xpath() has a loose return type. The same call can return a list of elements, a list of strings, a single string, a number, or a boolean, depending on the expression. The type stubs reflect this by typing the result as Any. A typo in the expression or an upstream HTML change can therefore produce a result of the wrong type, with the failure surfacing far away from the offending line.

The helpers in zavod.helpers.html wrap .xpath() with concrete return types and runtime checks. Prefer them over calling .xpath() directly:

Helper Returns Use for
h.xpath_elements list[Element] Any element-returning expression.
h.xpath_element Element Exactly one element; raises on zero or multiple matches.
h.xpath_strings list[str] Text-returning expressions (.../text(), string(...)).
h.xpath_string str Exactly one string result.
h.element_text str Concatenated text content, whitespace-squashed by default. 
from zavod import helpers as h

# Typed result; raises if the table is not unique.
table = h.xpath_element(doc, './/div[@id="block-content"]//table')

# Typed list of rows with a count guard.
rows = h.xpath_elements(table, "./tbody/tr", expect_exactly=12)

# Clean string extraction with whitespace squashing.
title = h.element_text(h.xpath_element(doc, ".//h1"))

For tabular content, use h.parse_html_table rather than walking rows by hand. It yields one dict[str, Element] per row, keyed by slugified header text; pair it with h.cells_to_str when string values are enough:

for row in h.parse_html_table(table):
    cells = h.cells_to_str(row)
    name = cells["full_name"]
    # ... or reach into row["full_name"] when the cell contains links or markup

h.links_to_dict is occasionally useful when a cell contains labelled anchor elements that should become a {label: href} mapping.

lxml's own .findall() is typed and a fine choice for trivial relative selectors (row.findall("./td"), el.findall(".//a")); reach for the xpath_* helpers when the selector actually needs XPath predicates, axes, or contains(...).

Write specific but semantic selectors

Selectors need to be specific enough to pick out the intended content, but semantic and concise enough to survive irrelevant HTML changes. A selector that is too loose can put the wrong text into a person's name property. A selector pinned to layout details breaks on the next restyle.

# Good:
table = h.xpath_element(doc, './/div[@id="block-content"]//table')

# Avoid:
table = h.xpath_element(doc, './/div[@id="block-content"]//div[3]//table')

Prefer .//div[contains(@class, 'abc')] over .//div[@class='abc'] because the class attribute can hold multiple whitespace-separated values. Watch for conflicting class names like abc-footer, which also match contains(@class, 'abc').

Fail loudly when a selection does not match expectations

Selections often turn out to be different from what was intended. Common symptoms:

  • the loop runs over an empty selection and the crawler silently produces no entities;
  • content added to the page matches the expression unintentionally and ends up in entity properties;
  • the wrong table is selected, and the failure surfaces several function calls later.

Errors should surface as close to the offending code as possible. The typed helpers above enforce shape (list vs. single, element vs. string). For count expectations, pass expect_exactly or assert on the returned list:

# Exactly one match expected; raises on zero or many.
table = h.xpath_element(doc, ".//table")

# Known fixed count.
rows = h.xpath_elements(table, "./tbody/tr", expect_exactly=12)

# Open-ended but must not be empty.
items = h.xpath_elements(doc, ".//h2[text()='The Section']/following-sibling::ul/li")
assert len(items) > 0, items

Detect changes in hand-parsed content

Use h.assert_dom_hash when a change to a particular HTML region needs human review, such as content extracted by hand or prose that controls crawler configuration. Pass text_only=True to ignore markup churn and monitor visible text. The change-detection guide covers monitor selection, scoping and the rules for accepting a new hash.