Zorp 3.4 LTS Reference Guide

The SSL framework starts its operation by inspecting the values set in the ssl.handshake_seq attribute. When this attribute is set to SSL_HSO_CLIENT_SERVER the client side, otherwise (SSL_HSO_SERVER_CLIENT) the server side handshake is performed first.

As part of the handshake process the proxy checks if SSL is enabled on the given side (ssl.client_connection_security and ssl.server_connection_security attributes). It is not necessary for SSL to be enabled on both sides - Zorp can handle one-sided SSL connections as well (e.g., the firewall communicates in an unencrypted channel with the client, but in a secure channel with the server). If SSL is not enabled, the handshake is skipped for that side.

When SSL is needed, the proxy will cooperate with the policy layer to have all required parameters (keys, certificates, etc.) set up. This is achieved using decision points in the hash named ssl.handshake_hash which is explained later in detail.

The SSL handshake is slightly different for the client (in this case Zorp behaves as an SSL server) and the server (when Zorp behaves as an SSL client).

As an SSL server the first thing to present to an SSL client is a certificate/key pair, thus a call to the 'setup_key' callback is made. It is expected that by the time this callback returns the attributes ssl.client_local_privatekey and ssl.client_local_certificate are filled appropriately.

If peer authentication is enabled (by setting the attribute ssl.client_verify_type) a list of trusted CA certificates must be set up (stored in the hash ssl.client_local_ca_list). The list can be set up by the 'setup_ca_list' function call. Peer certificates are verified against the trusted CA list and their associated revocation lists. Revocations can be set up in the 'setup_crl_list' callback.

At the end of the verification another callback named 'verify_cert' is called which can either ACCEPT or DENY the certificate possibly overriding the verification against the local CA database.

The server-side handshake is similar to the client-side handshake previously described. The difference is the order of certificate verification. On the server side Zorp verifies the server's certificate first and then sends its own certificate for verification. This is unlike the client side where the local certificate is sent first, and then the peer's certificate is verified.

So the callbacks are called in this order: 'setup_ca_list' and 'setup_crl_list' to set up CA and CRL information, 'verify_cert' to finalize certificate validation, and 'setup_key' to optionally provide a local certificate/key pair.

A certificate name behaves as a string, and contains a DN in the following format (also known as one-line format):

/RDN=value/RDN=value/.../RDN=value/

The word RDN stands for relative distinguished name. For example the DN below:

cn=Root CA, ou=CA Group, o=Foo Ltd, l=Bar, st=Foobar State, c=US

becomes:

/C=US/ST=Foobar State/L=Bar/O=Foo Ltd/OU=CA Group/CN=Root CA/

	Warning
	The format and representation of certificate names may change in future releases.

A certifying authority may revoke the issued certificates. A revocation means that the serial number and the revocation date is added to the list of revoked certificates. Revocations are published on a regular basis. This list is called the Certificate Revocation List, also known as CRL. A CRL always has an issuer, a date when the list was published, and the expected date of its next update.

The proxy stores trusted CA certificates in a Certificate hash. This hash can be indexed by two different types. If an integer index is used, the slot specified by this value is looked up; if a string index is used, it is interpreted as a one-line DN value, and the appropriate certificate is looked up. Each slot in this hash contains an X.509 certificate.

Similarly to the certificate hash, a separate hash for storing Certificate Revocation Lists was defined. A CRL contains revocation lists associated to CAs.

Zorp is able to automatically verify the certificates received. The types of accepted certificates can be controlled separately on the client and the server side using the ssl.client_verify_type and the ssl.server_verify_type attributes. These attributes offer an easy way to restrict encrypted access only to sites having trustworthy certificates. The available options are summarized in the following table.

The ssl.server_check_subject can be used to compare the domain name provided in the Subject field of the server certificate to application level information about the server. Currently it can compare the Subject field to the domain name of the HTTP request in HTTPS communication. If the ssl.server_check_subject is set to TRUE and ssl.server_verify_type is SSL_VERIFY_REQUIRED_UNTRUSTED or SSL_VERIFY_REQUIRED_TRUSTED, the HTTP proxy using the SSL framework will deny access to the page and return an error if the Subject field does not match the domain name of the URL.

The basic protocol is as follows: the client issues a request (also called command in FTP terminology) and the server responds with the result. Both commands and responses are line based: commands are sent as complete lines starting with a keyword identifying the operation to be performed. A response spans one or more lines, each specifying the same 3-digit status code and possible explanation.

Certain commands (for example RETR, STOR or LIST) also have a data attachment which is transferred to the peer. Data attachments are transferred in a separate TCP connection. This connection is established on-demand on a random, unprivileged port when a data transfer command is issued.

Endpoint information of this data channel is exchanged via the PASV and PORT commands, or their newer equivalents (EPSV and EPRT).

The data connection can either be initiated by the client (passive mode) or the server (active mode). In passive mode (PASV or EPSV command) the server opens a listening socket and sends back the endpoint information in the PASV response. In active mode (PORT or EPRT command) the client opens a listening socket and sends its endpoint information as the argument of the PORT command. The source port of the server is usually either 20, or the port number of the Command Channel minus one.

	Example 4.2. FTP protocol sample
	220 FTP server ready USER account 331 Password required. PASS password 230 User logged in. SYST 215 UNIX Type: L8 PASV 227 Entering passive mode (192,168,1,1,4,0) LIST 150 Opening ASCII mode data connection for file list 226-Transferring data in separate connection complete. 226 Quotas off QUIT 221 Goodbye

Changing the default behavior of commands can be done by using the hash attribute request, indexed by the command name (e.g.: USER or PWD). There is a similar attribute for responses called response, indexed by the command name and the response code. The possible values of these hashes are shown in the tables below. See Section 2.1, Policies for requests and responses for details. When looking up entries of the response attribute hash, the lookup precedence described in Section 2.1.2, Response codes is used.

Example 4.3. Customizing FTP to allow only anonymous sessions

This example calls a function called pUser (defined in the example) whenever a USER command is received. All other commands are accepted. The parameter of the USER command (i.e. the username) is examined: if it is 'anonymous' or 'Anonymous', the connection is accepted, otherwise it is rejected. class AnonFtp(FtpProxy): def config(self): self.request["USER"] = (FTP_REQ_POLICY, self.pUser) self.request["*"] = (FTP_REQ_ACCEPT) def pUser(self,command): if self.request_parameter == "anonymous" or self.request_parameter == "Anonymous": return FTP_REQ_ACCEPT return FTP_REQ_REJECT

FTP servers send the list of supported features to the clients. For example, proftpd supports the following features: LANG en, MDTM, UTF8, AUTH TLS, PBSZ, PROT, REST STREAM, SIZE. Zorp can change the default behavior of Ftp features using the hash attribute features, indexed by the name of the feature (e.g.: UTF8 or AUTH TLS). The possible actions are shown in the table below. See Section 2.1, Policies for requests and responses for details.

The built-in Ftp proxies of Zorp permit the use of every feature by default.

Enabling FTPS connections

For FTPS connections to operate correctly, the FTP server and client applications must comply to the FTP Security Extensions (RFC 2228) and Securing FTP with TLS (RFC 4217) RFCs.

For FTPS connections, the AUTH TLS, PBSZ, PROT features must be accepted. Also, STARTTLS support must be properly configured. See Section 3.2, Configuring TLS and SSL encrypted connections for details.

If the proxy is configured to disable encryption between Zorp and the client, the proxy automatically removes the AUTH TLS, PBSZ, PROT features from the list sent by the server.

If STARTTLS connections are accepted on the client side (self.ssl.client_security=SSL_ACCEPT_STARTTLS), but TLS-forwarding is disabled on the server side, the proxy automatically inserts the AUTH TLS, PBSZ, PROT features into the list sent by the server. These features are inserted even if encryption is explicitly disabled on the server side or the server does not support the FEAT command, making one-sided STARTTLS support feasible.

Warning

When using inband routing with the FTPS protocol, Zorp compares the server's certificate to its hostname. The subject_alt_name parameter (or the Common Name parameter if the subject_alt_name parameter is empty) of the server's certificate must contain the hostname or the IP address (as resolved from the Zorp host) of the server (e.g., ftp.example.com). Alternatively, the Common Name or the subject_alt_name parameter can contain a generic hostname, e.g., *.example.com. Note that if the Common Name of the certificate contains a generic hostname, do not specify a specific hostname or an IP address in the subject_alt_name parameter.

	Note
	The Zorp Ftp proxy does not support the following FTPS-related commands: REIN, CCC, CDC. STARTTLS is supported in nontransparent scenarios as well.

	Example 4.4. Configuring FTPS support
	This example is a standard FtpProxy with FTPS support enabled. class FtpsProxy(FtpProxy): def config(self): self.ssl.client_connection_security = SSL_ACCEPT_STARTTLS self.ssl.server_connection_security = SSL_FORWARD_STARTTLS

The available stacking modes for this proxy module are listed in the following table. For additional information on stacking, see Section 2.3.1, Proxy stacking.

Starting with Zorp 3.3FR1, the Ftp proxy supports inband authentication as well to use the built-in authentication method of the FTP and FTPS protocols to authenticate the client. The authentication itself is performed by the ZAS backend configured for the service.

If the client uses different usernames on ZAS and the remote server (e.g., he uses his own username to authenticate to ZAS, but anonymous on the target FTP server), the client must specify the usernames and passwords in the following format:

Username:

<ftp user>@<proxy user>@<remote site>[:<port>]

Password:

<ftp password>@<proxy password>

Alternatively, all the above information can be specified as the username:

<ftp user>@<proxy user>@<remote site>[:<port>]:<ftp password>@<proxy password>

Warning

When using inband routing with the FTPS protocol, Zorp compares the server's certificate to its hostname. The subject_alt_name parameter (or the Common Name parameter if the subject_alt_name parameter is empty) of the server's certificate must contain the hostname or the IP address (as resolved from the Zorp host) of the server (e.g., ftp.example.com). Alternatively, the Common Name or the subject_alt_name parameter can contain a generic hostname, e.g., *.example.com. Note that if the Common Name of the certificate contains a generic hostname, do not specify a specific hostname or an IP address in the subject_alt_name parameter.

HTTP is a text based protocol where a client sends a request comprising of a METHOD, an URI and associated meta information represented as MIME-like headers, and possibly a data attachment. The server responds with a status code, a set of headers, and possibly a data attachment. Earlier protocol versions perform a single transaction in a single TCP connection, HTTP/1.1 introduces persistency where a single TCP connection can be reused to perform multiple transactions.

An HTTP method is a single word - usually spelled in capitals - instructing the server to apply a function to the resource specified by the URI. Commonly used HTTP methods are "GET", "POST" and "HEAD". HTTP method names are not restricted in any way, other HTTP based protocols (such as WebDAV) add new methods to the protocol while keeping the general syntax intact.

Headers are part of both the requests and the responses. Each header consists of a name followed by a colon (':') and a field value. These headers are used to specify content-specific and protocol control information.

The response to an HTTP request starts with an HTTP status line informing the client about the result of the operation and an associated message. The result is represented by three decimal digits, the possible values are defined in the HTTP RFCs.

The protocol has three variants, differentiated by their version number. Version 0.9 is a very simple protocol which allows a simple octet-stream to be transferred without any meta information (e.g.: no headers are associated with requests or responses).

Version 1.0 introduces MIME-like headers in both requests and responses; headers are used to control both the protocol (e.g.: the "Connection" header) and to give information about the content being transferred (e.g.: the "Content-Type" header). This version has also introduced the concept of name-based virtual hosts.

Building on the success of HTTP/1.0, version 1.1 of the protocol adds persistent connections (also referred to as "connection keep-alive") and improved proxy control.

Both requests and responses might have an associated data blob, also called an entity in HTTP terminology. The size of the entity is determined using one of three different methods:

	Example 4.5. Example HTTP transaction
	GET /index.html HTTP/1.1 Host: www.example.com Connection: keep-alive User-Agent: My-Browser-Type 6.0 HTTP/1.1 200 OK Connection: close Content-Length: 14 <html> </html>

HttpProxy is able to operate both in transparent and non-transparent mode. In transparent mode, the client does not notice (or even know) that it is communicating through a proxy. The client communicates using normal server-style requests.

In non-transparent mode, the address and the port of the proxy server must be set on the client. In this case the client sends proxy-style requests to the proxy.

	Example 4.6. Proxy style HTTP query
	GET http://www.example.com/index.html HTTP/1.1 Host: www.example.com Connection: keep-alive User-Agent: My-Browser-Type 6.0 HTTP/1.1 200 OK Connection: close Content-Length: 14 <html> </html>

In non-transparent mode it is possible to request the use of the SSL protocol through the proxy, which means the client communicates with the proxy using the HTTP protocol, but the proxy uses HTTPS to communicate with the server. This technique is called data tunneling.

	Example 4.7. Data tunneling with connect method
	CONNECT www.example.com:443 HTTP/1.1 Host: www.example.com User-agent: My-Browser-Type 6.0 HTTP/1.0 200 Connection established Proxy-agent: My-Proxy/1.1

Changing the default behavior of requests is possible using the request attribute. This hash is indexed by the HTTP method names (e.g.: GET or POST). The response attribute (indexed by the request method and the response code) enables the control of HTTP responses. The possible actions are described in the following tables. See also Section 2.1, Policies for requests and responses. When looking up entries of the response attribute hash, the lookup precedence described in Section 2.1.2, Response codes is used.

Example 4.8. Implementing URL filtering in the HTTP proxy

This example calls the filterURL function (defined in the example) whenever a HTTP GET request is received. If the requested URL is 'http://www.disallowedsite.com', the request is rejected and an error message is sent to the client. class DmzHTTP(HttpProxy): def config(self): HttpProxy.config(self) self.request["GET"] = (HTTP_REQ_POLICY, self.filterURL) def filterURL(self, method, url, version): if (url == "http://www.disallowedsite.com"): self.error_info = 'Access of this content is denied by the local policy.' return HTTP_REQ_REJECT return HTTP_REQ_ACCECT

Example 4.9. 404 response filtering in HTTP

In this example the 404 response code to GET requests is rejected, and a custom error message is returned to the clients instead. class DmzHTTP(HttpProxy): def config(self): HttpProxy.config(self) self.response["GET", "404"] = (HTTP_RSP_POLICY, self.filter404) def filter404(self, method, url, version, response): self.error_status = 404 self.error_info = "Requested page was not accessible." return HTTP_RSP_REJECT

Both request and response headers can be modified by the proxy during the transfer. New header lines can be inserted, entries can be modified or deleted. To change headers in the requests and responses use the request_header hash or the response_header hash, respectively.

Similarly to the request hash, these hashes are indexed by the header name (like "User-Agent") and contain an actiontuple describing the action to take.

By default, the proxy modifies only the "Host", "Connection", "Proxy-Connection" and "Transfer-Encoding" headers. "Host" headers need to be changed when the proxy modifies the URL; "(Proxy-)Connection" is changed when the proxy turns connection keep-alive on/off; "Transfer-Enconding" is changed to enable chunked encoding.

Example 4.10. Header filtering in HTTP

The following example hides the browser used by the client by replacing the value of the User-Agent header to Lynx in all requests. The use of cookies is disabled as well. class MyHttp(HttpProxy): def config(self): HttpProxy.config(self) self.request_header["User-Agent"] = (HTTP_HDR_CHANGE_VALUE, "Lynx 2.4.1") self.request_header["Cookie"] = (HTTP_HDR_POLICY, self.processCookies) self.response_header["Set-Cookie"] = (HTTP_HDR_DROP,) def processCookies(self, name, value): # You could change the current header in self.current_header_name # or self.current_header_value, the current request url is # in self.request_url return HTTP_HDR_DROP

URLs or sets of URLs can be easily rejected or redirected to a local mirror by modifying some attributes during request processing.

When an HTTP request is received, normative policy chains are processed (self.request, self.request_header). Policy callbacks for certain events can be configured with the HTTP_REQ_POLICY or HTTP_HDR_POLICY directives. Any of these callbacks may change the request_url attribute, instructing the proxy to fetch a page different from the one specified by the browser. Please note that this is transparent to the user and does not change the URL in the browser.

	Example 4.11. URL redirection in HTTP proxy
	This example redirects all HTTP GET requests to the 'http://www.balabit.com/' URL by modifying the value of the requested URL. class MyHttp(HttpProxy): def config(self): HttpProxy.config(self) self.request["GET"] = (HTTP_REQ_POLICY, self.filterURL) def filterURL(self, method, url, version): self.request_url = "http://www.balabit.com/" return HTTP_REQ_ACCEPT

Example 4.12. Redirecting HTTP to HTTPS

This example redirects all incoming HTTP connections to an HTTPS URL. class HttpProxyHttpsredirect(HttpProxy): def config(self): HttpProxy.config(self) self.error_silent = TRUE self.request["GET"] = (HTTP_REQ_POLICY, self.reqRedirect) def reqRedirect(self, method, url, version): self.error_status = 301 #self.error_info = 'HTTP/1.0 301 Moved Permanently' self.error_headers="Location: https://%s/" % self.request_url_host return HTTP_REQ_REJECT

Zorp differentiates between two request types: server requests and proxy request. Server requests are sent by browsers directly communicating with HTTP servers. These requests include an URL relative to the server root (e.g.: /index.html), and a 'Host' header indicating which virtual server to use.

Proxy requests are used when the browser communicates with an HTTP proxy. These requests include a fully specified URL (e.g.: http://www.example.com/index.html).

As there is no clear distinction between the two request types, the type of the request cannot always be accurately detected automatically, though all common cases are covered.

Requests are handled differently in transparent and non-transparent modes.

A transparent HTTP proxy (transparent_mode attribute is TRUE) is meant to be installed in front of a network where clients do not know about the presence of the firewall. In this case the proxy expects to see server type requests only. If clients communicate with a real HTTP proxy through the firewall, proxy type requests must be explicitly enabled using the permit_proxy_requests attribute, or transparent mode has to be used.

The use of non-transparent HTTP proxies (transparent_mode attribute is FALSE) must be configured in web browsers behind the firewall. In this case Zorp expects proxy requests only, and emits server requests (assuming parent_proxy is not set).

Parent proxies are non-transparent HTTP proxies used behind Zorp. Two things have to be set in order to use parent proxies. First, select a router which makes the proxy connect to the parent proxy, this can be either InbandRouter() or DirectedRouter(). Second, set the parent_proxy and parent_proxy_port attributes in the HttpProxy class. Setting these attributes results in proxy requests to be emitted to the target server both in transparent and non-transparent mode.

The parent proxy attributes can be set both in the configuration phase (e.g.: config() event), or later on a per-request basis. This is possible because the proxy re-connects.

Example 4.13. Using parent proxies in HTTP

In this example the MyHttp proxy class uses a parent proxy. For this the domain name and address of the parent proxy is specified, and a service using an InbandRouter is created. class MyHttp(HttpProxy): def config(self): HttpProxy.config(self) self.parent_proxy = "proxy.example.com" self.parent_proxy_port = 3128 def instance(): Service("http", MyHttp, router=InbandRouter()) Listener(SockAddrInet('10.0.0.1', 80), "http")

In non-transparent mode it is possible to let Zorp process ftp:// URLs, effectively translating HTTP requests to FTP requests on the fly. This behaviour can be enabled by setting permit_ftp_over_http to TRUE and adding port 21 to target_port_range. Zorp currently supports passive mode transfers only.

There are cases when the HTTP proxy must return an error page to the client to indicate certain error conditions. These error messages are stored as files in the directory specified by the error_files_directory attribute, and can be customized by changing the contents of the files in this directory.

Each file contains plain HTML text, but some special macros are provided to dynamically add information to the error page. The following macros can be used:

It is generally recommended not to display error messages to untrusted clients, as they may leak confidential information. To turn error messages off, set the error_silent attribute to TRUE, or strip error files down to a minimum.

	Note
	The language of the messages can be set using the config.options.language global option, or individually for every Http proxy using the language parameter. See Appendix 4, Global options of Zorp for details.

HTTP supports stacking proxies for both request and response entities (e.g.: data bodies). This is controlled by the request_stack and response_stack attribute hashes. See also Section 2.3.1, Proxy stacking.

There are two stacking modes available: HTTP_STK_DATA sends only the data portion to the downstream proxy, while HTTP_STK_MIME also sends all header information to make it possible to process the data body as a MIME envelope. Please note that while it is possible to change the data part in the stacked proxy, it is not possible to change the MIME headers - they can be modified only by the HTTP proxy. The possible parameters are listed in the following tables.

Please note that stacking is skipped altogether if there is no body in the message.

Certain webserver applications may return data entities in 205 responses. This is explicitly prohibited by the RFCs, but Zorp permits such responses for interoperability reasons.

Starting with version 3.3FR1, Zorp supports category-based URL filtering using a regularly updated database.

Configuring URL-filtering in HTTP

The URLs and domains in the database are organized into thematic categories like adult, news, jobsearch, etc.

To enable url-filtering, set the enable_url_filter and enable_url_filter_dns options to TRUE. The enable_url_filter_dns option is needed only to ensure that a domain or URL is correctly categorized even when it is listed in the database using its domain name, but the client tries to access it with its IP address (or vice-versa).

	Note
	URL-filtering is handled by the Zorp Http proxy, without the need of using ZCV. The URL-filtering capability of Zorp is available only after purchasing the url-filter license option. Updates to the URL database are automatically downloaded daily from the BalaBit website using the zavupdate utility.

Access to specific categories can be set using the url_category option, which is a hash indexed by the name of the category. The following actions are possible:

Action	Description
HTTP_URL_ACCEPT	Permit access to the URL.
HTTP_URL_REJECT	Reject the request. The error code and reason for the rejection can be specified in the optional second and third arguments. See Section Configuring URL-filtering in HTTP for details.
HTTP_URL_REDIRECT	Redirect the connection to the URL specified in the second argument.

Table 4.13.  Action codes for URL filtering

Example 4.14. URL-filtering example

The following example blocks several categories and accepts the rest. For a complete list of categories, see Section List of URL-filtering categories. class MyHTTPUrlFilter(HttpProxy): def config(self): HttpProxy.config(self) self.enable_url_filter=TRUE self.enable_url_filter_dns=TRUE self.url_category['adult']=(HTTP_URL_REJECT, (403, "Adult website",)) self.url_category['porn']=(HTTP_URL_REJECT, (403, "Porn website",)) self.url_category['malware']=(HTTP_URL_REJECT, (403, "Site contains malware",)) self.url_category['phishing']=(HTTP_URL_REJECT, (403, "Phishing site",)) self.url_category['warez']=(HTTP_URL_REJECT, (403, "Warez site",)) self.url_category['*']=(HTTP_URL_ACCEPT,) The following example redirects access to online gaming sites to a dummy website. class MyHTTPUrlFilter(HttpProxy): def config(self): HttpProxy.config(self) self.enable_url_filter=TRUE self.enable_url_filter_dns=TRUE self.url_category['onlinegames']=(HTTP_URL_REDIRECT, "http://example.com") self.url_category['*']=(HTTP_URL_ACCEPT,)

Customizing the URL database

To customize the database, you have to manually edit the relevant files of the database. The URL database is located on the Zorp hosts under the /etc/zorp/urlfilter/ directory. Every thematic category is subdirectory containing two files called domains and urls. These files contain the list of domains (e.g., example.com) and URLs (e.g., example.com/news/) that fall into the specific category. Optionally, the subdirectory may contain a third file called expressions, where more complex rules can be defined using regular expressions.

• To to allow access (whitelist) to a domain or URL, add it to the domains or urls file of the whitelist category. Do not forget to configure your Http proxies to permit access to the domains of the whitelist category.

  	Warning
  	Deleting a domain from a category is not equivalent to whitelisting. Deleted domains will be re-added to their original category after the next database update.

• To add a new URL or domain to an existing category, create a new subdirectory under /etc/zorp/urlfilter/, create the domains and urls files for this new category, and add the domain or URL (without the http://www. prefix) to the domains or urlsfile. Zorp will automatically add these sites to the specific category after the next daily database update, or when the zufupdate command is executed.

• To create a new category, create a new subdirectory under /etc/zorp/urlfilter/, create the domains and urls files for this new category, and add domains and URLs to these files. Do not forget to configure your Http proxies to actually use the new category.

	Warning
	Manual changes to the URL database are not applied automatically, they become effective only after the next daily database update, or when the zufupdate command is executed.

	Note
	Manual changes are automatically merged with the original database during database updates. If you are using the URL-filter database on several Zorp hosts and modify the database manually, make sure to copy your changes to the other hosts as well.

The syntax of the IMAP protocol is strictly defined, both the client and the server is either reading a complete line or a sequence of octets prefixed with the length of the sequence.

Request lines start with the tag, followed by a command verb identifying the operation. Each command might have one or more arguments separated by spaces. Each argument has an associated type, one of: ATOM, LITERAL, STRING, LIST. The type further specifies the syntax how these arguments are represented.

A response from the server might be sent directly in response to a request, or unilaterally whenever the server implementation feels it appropriate. The response includes a response verb with zero or more arguments. Note that there might be more response verbs returned for a single command and the response verbs have no direct relationship with the request verb.

Content (e.g.: mail bodies) are transferred as literals embedded in commands and responses. There is no separate bulk transfer mode in the protocol like in POP3 or SMTP. This results in extremely large request/response sizes.

Each message might have one or more associated message flags like '\Deleted' or '\Seen'.

IMAP defines four protocol states. Most commands are valid only in certain states. IMAP has the following states:

IMAP is similar to other protocols in the sense that a connection is authenticated once, at the beginning of the communication. Before authentication is performed only a limited set of commands can be issued, for example AUTHENTICATE and LOGIN.

Each IMAP operation requires a current mailbox which is similar to the current working directory on UNIX systems. Without a selected mailbox, only a limited set of commands can be issued, for example SELECT, CREATE or REMOVE.

Once a mailbox is selected using the SELECT command, further operations become available, like FETCH or STORE.

Example 4.16. IMAP protocol sample

* OK newmail IMAP server ready A001 CAPABILITY * CAPABILITY IMAP4 IMAP4rev1 ACL QUOTA LITERAL+\ MAILBOX-REFERRALS NAMESPACE UIDPLUS ID\ NO_ATOMIC_RENAME UNSELECT CHILDREN\ MULTIAPPEND SORT THREAD=ORDEREDSUBJECT\ THREAD=REFERENCES IDLE STARTTLS LISTEXT\ LIST-SUBSCRIBED ANNOTATEMORE A001 OK Completed A002 LOGIN user password A002 OK User logged in A003 SELECT INBOX * FLAGS (\Answered \Flagged \Draft \Deleted \Seen) * OK [PERMANENTFLAGS (\Answered \Flagged \Draft\ \Deleted \Seen \*)] * 1094 EXISTS * 3 RECENT * OK [UNSEEN 1092] * OK [UIDVALIDITY 1047554575] * OK [UIDNEXT 36885] A003 OK [READ-WRITE] Completed A004 FETCH 1 RFC822 * 1 FETCH (RFC822 {12} 123456789012 ) A004 OK Completed A005 LOGOUT * BYE LOGOUT received A005 OK Completed

Responses to IMAP requests come in two types: tagged and untagged. When a client issues a request, the server responds with a single tagged response, which may be preceeded by a number of untagged response lines. In the example above, the client issues a tagged A001 CAPABILITY command to ask the server for the supported capabilities. The server replies with the untagged * CAPABILITY IMAP4 ... line, listing the capabilities, and the tagged A001 OK Completed line, indicating that the request was successfully completed.

Changing the default behaviour of requests is possible using the request attribute. This hash is indexed by the IMAP command name.

The response attribute is indexed as follows: The response attribute hash is a three-dimensional hash, indexed by the command name for which the response is sent; the type of the response (TAGGED or UNTAGGED); and the response name. Untagged responses are accepted when there is a command in the pending queue (i.e. no tagged response arrived to it yet). The following constants are defined for the response types:

The proxy looks up the hash value corresponding to the IMAP command name as the key. If the hash contains no entry for a command, the "*" entry is used. If there is no "*" entry in the hash, the command is denied.

The possible actions are described in the following tables.

For calling a method, the hash must contain a tuple containing two values. The first value is IMAP_REQ_POLICY and the second is the function to call. The function must return with one of the IMAP_REQ_* values (excluding IMAP_*_POLICY), displayed in the table above.

The function is called with three arguments (apart from 'self'): the command tag, the command name, and its arguments. The representation of arguments used by IMAP is described in Section 4.8.2.4, The IMAP command structure in policies.

If the proxy is to answer instead of the server, the action tuples must contain the following three items: The value IMAP_REQ_RESPOND; the STRING to be sent back followed by a command tag, and a LIST containing untagged lines to be sent back to the client.

For example, to reply to every CAPABILITY request on behalf of the server:

	Example 4.17. Rewriting IMAP capability response
	self.request["CAPABILITY"] = (IMAP_REQ_RESPOND, "OK CAPABILITY completed", ("[IMAP4rev1]", ))

There are other methods to control which CAPABILITYs are known by the client. There is a separate capability hash for this, indexed by the name of the capabilities. The valid values are listed below.

This hash has nothing to do with capabilities known by the proxy; it defines which answers can arrive to the client for a CAPABILITY command.

Modifying the IMAP greeting string

The IMAP greeting string can be modified (rewritten) by the proxy to hide sensitive information about the server. This can be realized as a rule defined as a tuple containing the following three items:

• The value IMAP_REQ_REWRITE;

• a default return value (e.g.: IMAP_REQ_ACCEPT);

• and a string.

	Example 4.18. Changing the greeting string in IMAP
	def config(self): ... self.response["GREETING", "UNTAGGED", "OK"] = / (IMAP_REQ_REWRITE, IMAP_REQ_ACCEPT, "Welcome to Zorp IMAP proxy") ...

In the IMAP protocol the user can assign flags to mails (or other objects). For example, a flag is assigned to a message to indicate that it has been read (\Seen), it can be marked as important mail or it can be indicated that it has already been answered (\Answered). The usable flags are not predefined in the protocol, IMAP clients can assign any flags they desire.

Flags can be controlled similarly to requests and responses using the flag hash. It is a normative hash indexed by the name of the flag (case sensitive). The common practice is to accept any flags by default and explicitly drop unneeded flags. The possible actions related to flags are shown in the table below.

When using functions in policies to evaluate IMAP commands, the commands are represented as a recursive tuple of tuples having the following structure. Every command is a tuple of length 3, containing the tag of the command, the name of the command and a tuple containing the arguments.

The following values are possible as arguments (IMAP command structure in the policy layer):

Of course, lists can contain other lists recursively.

When processing IMAP responses where a number argument precedes the response name (e.g.: 1094 EXISTS), the number counts as the first argument.

Below are some examples how the different argument types are used in the IMAP protocol.

Example 4.19.  IMAP arguments in use

Issued command: a0001 FETCH 1,2 RFC822 The command as processed by the Zorp IMAP proxy: tag: "a0001" command: "FETCH" arguments: ((',', (1, '1'), (2, '2')), 'RFC822'); where (',', (1, '1'), (2, '2') is a comma separated list, (1, '1') is an integer, and RFC822 is a string. Issued command: a0002 FETCH 1:2 RFC822 The command as processed by the Zorp IMAP proxy: tag: a0002 command: FETCH arguments: ((1, 2), 'RFC822'); where (1, 2) is a range. Received response: * 1 FETCH (RFC822 <literal>) The command as processed by the Zorp IMAP proxy: command: FETCH arguments: ((1, '1'), ('(', 'RFC822', '<LITERAL>')); where <LITERAL> is a literal represented by a string.

IMAP supports stacking proxies into different levels of the IMAP communication. Stacking is controlled by the stack attribute hash. See also Section 2.3.1, Proxy stacking. There are three stacking modes available, described in the table below.

LDAP is a request/response based binary protocol. The client can connect to the server on a channel at TCP/389 port and send REQUESTs. The client can request several operations in parallel. The following operations can be performed:

The protocol operates according to the following general scheme:

The LDAP protocol is described using ASN.1 (Abstract Syntax Notation), and is typically transferred using the Basic Encoding Rules, a subset of ASN.1.