1 URLs and HTTP

To access the text of a document from the web, first obtain its URL as a string. Convert the address into a url structure using string->url. Then, open the document using get-pure-port or get-impure-port, depending on whether or not you wish to examine its MIME headers. At this point, you have a regular input port with which to process the document, as with any other file.

Currently the only supported protocols are "http", "https", and sometimes "file".

1.1 URL Structure

http://sky@www:801/cgi-bin/finger;xyz?name=shriram;host=nw#top

{-1}   {2} {3} {4}{---5---------} {6} {----7-------------} {8}

1 = scheme, 2 = user, 3 = host, 4 = port,

5 = path (two elements),  6 = param (of second path element),

7 = query, 8 = fragment

The strings inside the user, path, query, and fragment fields are represented directly as Racket strings, without URL-syntax-specific quoting. The procedures string->url and url->string translate encodings such as %20 into spaces and back again.

By default, query associations are parsed with either ; or & as a separator, and they are generated with & as a separator. The current-alist-separator-mode parameter adjusts the behavior.

An empty string at the end of the path list corresponds to a URL that ends in a slash. For example, the result of (string->url "http://racket-lang.org/a/") has a path field with strings "a" and "", while the result of (string->url "http://racket-lang.org/a") has a path field with only the string "a".

When a "file" URL is represented by a url structure, the path field is mostly a list of path elements. For Unix paths, the root directory is not included in path; its presence or absence is implicit in the path-absolute? flag. For Windows paths, the first element typically represents a drive, but a UNC path is represented by a first element that is "" and then successive elements complete the drive components that are separated by / or \.

1.2 URL Functions

An HTTP connection is created as a pure port or a impure port. A pure port is one from which the MIME headers have been removed, so that what remains is purely the first content fragment. An impure port is one that still has its MIME headers.

If str starts with "file:", then the path is always parsed as an absolute path, and the parsing details depend on file-url-path-convention-type:

• 'unix : If "file:" is followed by // and a non-/, then the first element after the // is parsed as a host (and maybe port); otherwise, the first element starts the path, and the host is "".

• 'windows : If "file:" is followed by //, then the // is stripped; the remainder parsed as a Windows path. The host is always "" and the port is always #f.

This function does not raise any exceptions.

• 'unix : Elements in URL are treated as path elements. Empty strings in the path list are treated like 'same.

• 'windows : If the first element is "" then the next two elements define the UNC root, and the rest of the elements are treated as path elements. Empty strings in the path list are treated like 'same.

The url->string procedure uses alist->form-urlencoded when formatting the query, so it is sensitive to the current-alist-separator-mode parameter for determining the association separator. The default is to separate associations with a &.

The GET method is used to retrieve whatever information is identified by URL.

The HEAD method is identical to GET, except the server must not return a message body. The meta-information returned in a response to a HEAD request should be identical to the information in a response to a GET request.

The DELETE method is used to delete the entity identified by URL.

Beware: By default, "https" scheme handling does not verify a server’s certificate (i.e., it’s equivalent of clicking through a browser’s warnings), so communication is safe, but the identity of the server is not verified. To validate the server’s certificate, set current-https-protocol to a context created with ssl-make-client-context, and enable certificate validation in the context with ssl-set-verify!.

The "file" scheme for URLs is handled only by get-pure-port, which uses open-input-file, does not handle exceptions, and ignores the optional strings.

Beware: See get-pure-port for warnings about "https" certificate validation.

The net/head library provides procedures, such as extract-field for manipulating the header.

Since web servers sometimes return mis-formatted replies, purify-port is liberal in what it accepts as a header. as a result, the result string may be ill formed, but it will either be the empty string, or it will be a string matching the following regexp:

#rx"^HTTP/.*?(\r\n\r\n|\n\n|\r\r)"

When a header argument is supplied, it is passed along to the connect procedure.

The connection is made in such a way that the port is closed before call/input-url returns, no matter how it returns. In particular, it is closed if handle raises an exception, or if the connection process is interruped by an asynchronous break exception.

• the URL scheme, such as "http";

• the proxy server address; and

• the proxy server port number.

Currently, the only proxiable scheme is "http". The default mapping is the empty list (i.e., no proxies).

1.3 URL Unit

(require net/url-unit)

The url+scheme^ signature contains current-connect-scheme, which url@ binds to a parameter. The parameter is set to the scheme of a URL when tcp-connect is called to create a connection. A tcp-connect variant linked to url@ can check this parameter to choose the connection mode; in particular, net/url supplies a tcp-connect that actually uses ssl-connect when (current-connect-scheme) produces "https".

Note that net/url does not provide the current-connect-scheme parameter.

1.4 URL Signature

(require net/url-sig)