14.1. Procedure – Enabling TLS-encryption for VNC connections

Purpose: 

To enable TLS-encryption in a VNC connection policy, complete the following steps.

Note

Some vendors may use custom protocol elements and TLS encryption, that does not have available documentation. For this reason, these cannot be audited by PSM. Independetly from the vendors, only those custom features are supported that are described in the RFC 6143. Regarding encryption, only the completely TLS-encapsulated streams can be processed where the TLS encryption process was started before the VNC protocol handshake.

Prerequisites: 

Depending on your requirements, one or more of the following might be needed:

  • An X.509 certificate and its private key. PSM can display this certificate to the peers on the client and server side. You can also use different certificates for the client and server sides. Use your own PKI system to generate these certificates, as they cannot be created on PSM. Note that the Common Name of the certificate must contain the domain name or the IP address of PSM. otherwise the clients might reject the certificate.

  • To generate certificates on-the-fly for a connection, a signing certificate authority is required. For details on creating a signing CA, see Procedure 7.12, Signing certificates on-the-fly.

  • To require the peers of PSM to have an X.509 certificate signed by a specific Certificate Authority, a list of the trusted certificate authorities is needed. For details on creating a trusted CA list, see Procedure 7.11, Verifying certificates with Certificate Authorities.

Balabit recommends using 2048-bit RSA keys (or stronger).

Steps: 

  1. Navigate to VNC Control > Connections and select the connection policy in which you want to enable TLS.

    Figure 14.1. VNC Control > Connections — Enabling TLS-encryption for VNC connections

    VNC Control > Connections — Enabling TLS-encryption for VNC connections
  2. Set the encryption settings used between the client and PSM in the Client-side transport security settings section.

    To require encryption, select TLS. When the connection is encrypted, PSM has to show a certificate to the peer.

  3. Select the certificate to show to the peers.

    • To use the same certificate for every peer, complete the following steps.

      1. Generate and sign a certificate for PSM in your PKI system, and export the certificate and its private key.

      2. Select Use the same certificate for each connection.

      3. Select Private key for host certificate, click and upload the private key.

      4. Select X.509 host certificate, click and upload the certificate.

    • To use a separate certificate for every connection, complete the following steps. Note when using this option, the client must show a certificate to PSM, and the Client-side transport security settings > Peer certificate validation > Only accept certificates authenticated by the trusted CA list option must be set.

      1. Create a certificate authority that will be used to sign the certificates that PSM shows to the peer. For details, see Procedure 7.12, Signing certificates on-the-fly.

      2. Select Generate certificate on-the-fly.

      3. In the Signing CA field, select the certificate authority to use.

  4. Select how PSM should authenticate the peers.

    • To permit connections from peers without requesting a certificate, select No certificate is required.

    • To permit connections only from peers having valid certificate that was signed by a specific CA, complete the following steps.

      1. Create a list of trusted Certificate Authorities that will be used to validate the certificates of the peers. For details on creating a trusted CA list, see Procedure 7.11, Verifying certificates with Certificate Authorities.

      2. Select Only accept certificates authenticated by the trusted CA list.

      3. In the Trusted CA field, select the certificate authority list to use.

  5. Set the encryption settings used between PSM and the server in the Server-side transport security settings section.

    To require encryption, select TLS. When the connection is encrypted, PSM has to show a certificate to the peer.

  6. Select the certificate to show to the server.

    • If the server does not require a certificate from PSM, select None.

    • To use the same certificate for every peer, complete the following steps.

      1. Generate and sign a certificate for PSM in your PKI system, and export the certificate and its private key.

      2. Select Use the same certificate for each connection.

      3. Select Private key for host certificate, click and upload the private key.

      4. Select X.509 host certificate, click and upload the certificate.

    • To use a separate certificate for every connection, complete the following steps. Note when using this option, the client must show a certificate to PSM, and the Client-side transport security settings > Peer certificate validation > Only accept certificates authenticated by the trusted CA list option must be set.

      1. Create a certificate authority that will be used to sign the certificates that PSM shows to the peer. For details, see Procedure 7.12, Signing certificates on-the-fly.

      2. Select Generate certificate on-the-fly.

      3. In the Signing CA field, select the certificate authority to use.

  7. Select how PSM should authenticate the peers.

    • To permit connections from peers without requesting a certificate, select No certificate is required.

    • To permit connections only from peers having valid certificate that was signed by a specific CA, complete the following steps.

      1. Create a list of trusted Certificate Authorities that will be used to validate the certificates of the peers. For details on creating a trusted CA list, see Procedure 7.11, Verifying certificates with Certificate Authorities.

      2. Select Only accept certificates authenticated by the trusted CA list.

      3. In the Trusted CA field, select the certificate authority list to use.

  8. Click .

    Expected result: 

    The encryption settings are applied to the connection policy.