A useful inventory preserves how often each destination occurs while removing only distinctions the identity policy declares irrelevant. If one page contains links to /guide#install and /guide#usage, our document inventory has one guide target with count two. If it also links to /guide?edition=2, that is another key. Counting occurrences and deduplicating targets answer different questions; keep both through a count mapping.
The transformation has an order. Extract decoded href strings from anchor start events. Choose the effective base using the explicit optional base href. Resolve each reference, then apply document_key. Only after that should you compare origins and group targets. Comparing raw relative strings or grouping before normalization can split equivalent keys or merge different ones. A missing or valueless href is not an extracted reference at all, while an explicit empty href resolves to the current effective base.
This lab’s parser helper is supplied. It collects destinations at start events, including an anchor left open at end of source. That is intentional: a link count does not need the completed text-label contract of the earlier extractor. It ignores comments and non-anchor elements. It also ignores HTML base tags because the function’s caller supplies base_href; the capstone will explicitly discover a source base.
from urllib.parse import urljoin, urldefrag
base = "https://example.test/docs/page.html"
counts = {}
for href in ["../guide#install", "/guide#usage"]:
document = urldefrag(urljoin(base, href)).url
counts[document] = counts.get(document, 0) + 1
print(counts)The snippet illustrates counting after resolution and fragment removal. The build uses the full supplied document_key policy, including hostname and port handling. Reuse that tested helper instead of inventing another normalization policy in the grouping loop.
For this inventory, internal means the same canonical scheme and authority as the document. The authority already normalizes host case and the scheme’s default port. http://example.test differs from https://example.test; port 444 differs from HTTPS’s default port. A hostname that merely ends with a similar string is not the same host. These comparisons classify report rows only. They do not establish request authorization, DNS safety or SSRF defenses.
from urllib.parse import urlsplit
def origin(url):
parts = urlsplit(url)
port = parts.port
if port is None:
port = 443 if parts.scheme == "https" else 80
return parts.scheme, parts.hostname, port
print(origin("https://example.test/a") == origin("http://example.test/a"))
print(origin("https://example.test/a") == origin("https://example.test:444/a"))An omitted port receives the scheme’s default; explicit port zero remains zero. The build compares canonical authorities from the supplied key helper, which makes the same distinction.
Implement link_inventory with internal/external sorted rows and an ignored counter. Count each extracted reference that cannot resolve or whose resolved URL is rejected by the key policy as ignored, including malformed authorities, mailto and script-scheme references. An invalid declared base that makes urljoin raise produces an explicit ValueError. External means a syntactically accepted other origin, not a broken link. Internal means same-origin, not present on disk. Empty input produces empty groups; an invalid document URL raises ValueError. The capstone adds a known-page map to answer presence and reachability questions without pretending a local fixture proves live availability.