Two links can identify one source document while pointing to different places within it. https://example.test/guide#install and https://example.test/guide#usage share our document key. That is an application decision for a link inventory. It does not make the fragments meaningless: a later fragment audit could check whether those destinations exist. Keep the original reference if you will need that evidence.
Our key policy is intentionally conservative. Accept an absolute HTTP(S) URL with a nonempty ASCII hostname, no credentials, no whitespace/control characters or backslash, and a port that Python can parse in range. Reject an explicit empty port. Normalize the scheme and hostname to lowercase, remove the scheme’s default port, supply / for an empty path and remove the fragment. Preserve the path and query exactly otherwise. Bracket an IPv6 hostname when rebuilding its authority. We do not perform IDNA conversion or full hostname validation.
from urllib.parse import urlsplit, urlunsplit
parts = urlsplit("HTTPS://Example.TEST:443/Guide?q=one+two#install")
print(parts.scheme, parts.hostname, parts.port)
print(urlunsplit((parts.scheme, parts.hostname, parts.path, parts.query, "")))Do not lowercase the whole URL: /Guide and /guide may be different server resources. Do not decode %2F into /: that can change path segmentation. Do not remove a final slash, sort query pairs, delete “tracking-looking” fields or collapse repeated slashes without an application contract. Our policy also does not erase dot segments from already absolute inputs; relative resolution has its own merge behavior. Empty query/fragment delimiters may disappear when components are serialized, a deliberate limit of this key representation.
urlsplit is a component parser, not a validator. Some errors occur while splitting; a malformed port may instead raise when .port is read. Guard both operations and return None for inputs outside the build’s allowed subset. A valid-looking key is still no evidence that a host is reachable or safe to request. In particular, matching an origin or accepting an IP literal does not implement SSRF protection. This lab never resolves DNS, follows redirects or opens sockets.
Queries deserve a separate inspection operation. For form-style query syntax, parse_qsl returns ordered (name, value) pairs. With keep_blank_values=True, x=&x=2 retains both values. A dictionary conversion loses duplicate-key history, and a default parse drops blank values. A plus sign decodes as a space in form queries; %2B decodes as a literal plus. Those rules do not describe arbitrary URL paths.
from urllib.parse import parse_qsl
print(parse_qsl("tag=one&tag=two&empty=&q=a+b&plus=%2B", keep_blank_values=True))Implement document_key(url) and query_pairs(url). The second function inspects a well-formed URL and never rewrites its key. Test fragment variants, nondefault ports including zero, case-sensitive paths, duplicate/blank query values, unsupported schemes, credentials and malformed ports. Then compare keys only when your application has stated what equality means. A smaller explicit policy is easier to reason about than a function promising to “clean any URL.”