I Gave My App a URL

A bookmark importer is a pleasantly small feature. Give it a URL, fetch the page, read its title. There is barely enough work there for a form.

It is also a way for somebody to choose where your server opens a connection. That second description is just as accurate as the first, and much harder to fit into an innocent-looking text field.

I built a small Hanami application to put the two descriptions next to each other. It has no database and saves nothing. A successful import displays a title and the path the HTTP requests took to reach it. Alongside it sits a local laboratory: one server pretending to be a public website, another pretending to be an internal service, and a redirect between them.

Everything in the experiment runs on loopback. There is no real internal system to discover, and no third-party site involved.

The companion source is public. The version described here is 4822ea6.

A title from a URL

The application uses Ruby 3.4.6, Hanami and hanami-action 3.0.1, and net-http 0.9.1. Nokogiri handles the HTML. The action’s useful work comes down to this, with presentation and error handling omitted:

class Create < UrlImporter::Action
  include Deps["importer"]

  def handle(request, response)
    bookmark = importer.call(request.params[:url])
    # Present the title, final URL and request trace.
  end
end

Hanami gives the importer a home without making the action interesting. Fetching belongs in the HTTP client; interpreting the returned document belongs in the importer. The view receives text, escaped before it reaches HTML. It never receives a remote page to embed.

For an ordinary page the result is exactly as dull as it should be:

1. A normal page
   A small & useful page
2. A relative redirect
   302 -> 200

Those are the first two checks printed by the companion’s bin/demo. Following a redirect is useful. Plenty of perfectly ordinary links need one.

The page gets a vote

The third fixture returns a redirect to the other server. In the browser laboratory, the path is:

http://pages.lab.test:4567/to-internal
                 |
                 | 302
                 v
http://127.0.0.1:4568/private

The first name is resolved by an explicit lab exception. It represents a website the importer is allowed to fetch. The second server contains only fictional data and counts requests to its private endpoint.

The demo includes a deliberately naive client that follows this one redirect between its own fixtures. It does not accept arbitrary URLs and is not exposed as an application endpoint. Its result is:

3. Following a redirect without validating its destination
   Internal requests: 1

That counter is the experiment. The original page did not have to contain anything clever. It only had to tell the client where to go next.

In a hosted application, the connection originates from the server’s network position. A visitor may therefore be able to make it contact a service the visitor cannot reach directly. That is the concern behind server-side request forgery, or SSRF. OWASP’s prevention guide describes both restricted-destination and arbitrary-destination applications; which one you are building changes the policy you can enforce.

This importer accepts arbitrary public destinations, so there is no short list of trusted websites. There is still a policy: HTTP or HTTPS, no embedded credentials, default ports, and a conservative set of public unicast addresses. Loopback, private, link-local and other special-purpose ranges are excluded. IPv6 gets an explicit policy too.

Every redirect asks again

The important placement is inside the fetch loop. This is a shortened version; the full method also handles redirect limits, malformed destinations and HTTPS downgrades:

loop do
  target = @policy.resolve(url)
  status, location, body = request(target)

  # Record this hop and return the page for a non-redirect response.
  # ...

  url = URI.join(target.uri, location).to_s
end

A relative redirect becomes another absolute URL. Then the loop starts over, and the destination has to earn permission again. Returning to the same hostname does not skip that step.

The lab exception is equally specific: only http://pages.lab.test:4567 maps to loopback in the interactive lab. Enabling the lab does not grant permission to fetch the rest of the local network.

Through the real importer, the same experiment prints:

4. The same redirect through the importer
   Rejected: Only the default HTTP and HTTPS ports are accepted
   Additional internal requests: 0

The wording matters. This particular URL is refused by the port restriction, before address validation. It would be dishonest to present that output as proof that the private-address check ran. Separate tests exercise loopback and private addresses on default ports, including mixed public and private DNS answers.

What this experiment proves is narrower and useful: validation runs on the redirect destination, and the internal endpoint receives no additional request.

The address checked has to be the address used

There is another gap, even without redirects. Resolve a hostname, approve its address, then hand the hostname to an HTTP client that resolves it again: the check and the connection need not refer to the same answer.

The policy therefore returns both the parsed URI and the selected IP. It checks every address returned by its resolver and refuses the destination if any is disallowed. The transport uses the selected address explicitly:

http = Connection.new(uri.hostname, uri.port, nil)
http.ipaddr = target.ip
http.use_ssl = uri.scheme == "https"
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.verify_hostname = true

Connection is a small subclass of Net::HTTP; its extra job is bounding HTTP metadata reads. The original hostname remains the HTTP and TLS identity, while ipaddr selects the connection’s peer. Ruby documents the separate address and IP settings. Replacing the hostname itself with an IP would lose a distinction HTTPS needs.

The third argument, nil, disables environment proxy discovery. Otherwise a proxy could become another place where destination resolution happens. There are no cookies, authorization headers or automatic retries in this client.

The tests connect to a deliberately unresolvable hostname through a supplied loopback IP. The HTTPS test goes further: a temporary certificate is accepted for its own hostname, refused for a different one, and refused without its local trust anchor. Connecting to the checked IP is not an excuse to turn certificate verification off.

Permission still needs a budget

A destination passing the address policy can still send an enormous page or keep the connection occupied. The importer allows 256 KiB of HTML, three redirects, two-second connect/read/write timeouts and an eight-second deadline around the fetch, including DNS.

The body limit is checked while reading, rather than after the download:

response.read_body do |chunk|
  if body.bytesize + chunk.bytesize > @max_bytes
    raise Rejected, "The page exceeds the download limit"
  end

  body << chunk
end

A per-read timeout and a total deadline do different jobs. Occasional bytes can keep satisfying a read timeout while the operation takes much longer than intended. The outer deadline bounds that fetch as a whole.

The companion also limits HTTP metadata and HTML parser complexity. Those mechanisms have their own tests and tradeoffs; they are worth reading in the code, but they do not change where the central decision lives. Before each connection, decide whether this destination is permitted.

What the small app buys

From the companion directory, with Ruby 3.4.6 and Bundler 4.0.17 available, install the locked dependencies and run the experiment:

git clone https://git.sr.ht/~bounga/url-importer
cd url-importer
git checkout 4822ea60c8381942594687b675d3caf8a169ba37
bundle config set --local path vendor/bundle
bundle install
bin/demo

bin/demo starts and stops its own fixtures. For the browser, run bin/lab in one terminal and LAB_MODE=1 bin/server in another, both from the companion directory. The Hanami app is at http://127.0.0.1:2300. No hosts-file changes are needed.

The tests and local demonstration establish the behaviour of this particular client. They do not establish that every globally numbered address is harmless in every deployment. Routing is a property of the environment too. A hosted fetching service would need an independently enforced egress policy, access control and load limits. The address ranges also need maintenance as the IANA registries change.

That is the part of the form I wanted to make visible. The title is what comes back, but the connection is what the application agrees to do. Each redirect is another request for that agreement, and it deserves an answer of its own.