HTTP Client

Many AuthStates establish outgoing connections to HTTP/S servers, TAN-authentication plug-ins (TAN-authentication plug-ins), WS-Trust authentication plug-ins (WS-Trust authentication AuthStates) and X.509 authentication plug-in (X.509 authentication AuthState). The HTTP client properties are available in these auth states

Available HTTP clients / types

There are two distinct variations of the HTTP client available in nevisAuth: A "global" auth HTTP client and "auth state / instance" HTTP client.

Auth "global" HTTP client

The auth HTTP client is the default nevisAuth HTTP client provided, using a shared configuration for all connections. It is configured via system properties in the nevisAuth env.conf file with the same property names as the instance based httpclient, but with the prefix ch.nevis.auth.* (example: ch.nevis.auth.httpclient.tls.trustStoreRef).

The benefit of using this HTTP client is that it simplifies configuration for "everything using it" as the configuration only needs to be managed in one place. As it's preferable to have fine-granular control of how auth states are configured, normally auth states do not use this client.

Accessing the auth HTTP client via code for custom auth states or groovy script states can be achieved the following way:

Auth HTTP client

import ch.nevis.esauth.util.httpclient.api.Http;
 
Http.get().url("nevis.net").build().send();

AuthState / Instance HTTP client

The "instance" HTTP client is used by most auth states and allows a per-auth-state configuration of the HTTP client in the esauth4.xml file. For example, adding the httpclient.tls.keyObjectRef property to the auth state configures the key object references for this specific auth state. The benefit of using this client is having fine-granular control over its behaviour and configuration, for example configuring individual trust stores only used by this client.

Intance HTTP client

import ch.nevis.esauth.util.httpclient.api.HttpClient;
import ch.nevis.esauth.util.httpclient.api.HttpClients;
import ch.nevis.esauth.util.httpclient.api.Http;

Properties properties = new Properties();
HttpClient httpClient = HttpClients.create(properties);
Http.get().url("nevis.net").build().send(httpClient); 

When to use which HTTP client?

Deciding which HTTP client to use is of interest when you're using script states or custom auth states.

The following table will give you some hints on which one to pick depending on your use case.

• If you want to trust root-CA signed certificates use the auth HTTP client.
• If you want to manage the truststore yourself, pick the instance HTTP client and set the trustStoreRef property.

When relying heavily on the auth HTTP client, be aware that you'll likely need to bump the connection pool properties, since all the auth states using it will occupy more connection resources.

Packaging

The HTTP client is located in the nevisauth-commons-<version>.jar Java archive.

JVM default truststore handling

Usually, using the JVM truststore makes sense when wanting to establish HTTPS connections to endpoints using Root-CA signed certificates which are present in the JVM truststore.

If you want to use the default JVM truststore instead of supplying your own or managing certificates yourself, the way to achieve this is not to configure the trustStoreRef property of the HTTP client. This will lead the HTTP client to fall back to the JVM truststore.

The HTTP client is handling usage of the default truststore supplied by the JVM in a special way as the JVM vendor supplied truststore is usually not directly accessible from nevisAuth.

To solve this issue, the HTTP client will attempt to directly load the certificates supplied by the JVM from the default file system path they are located at. If for some reason a JVM vendor does not use the default path where these certificates are usually stored, the path can be specified using the -Dch.nevis.auth.httpclient.tls.defaultTrustStore JVM system property. (This will be applied to both "global" auth and instance based http clients). See System Properties

nevisAuth will load the JVM default trustore first and afterwards what was specificed in the -Djavax.net.ssl.trustStore.

note

Certificates are stored per alias in memory, so aliases from -Djavax.net.ssl.trustStore will override ceritificates from the JVM default truststore in case their alias matches. This behaviour only applies if there was no direct truststore configuration applied on the HttpClient. If the HttpClient truststore is directly configured for the HTTP client nothing else will be loaded, except what was specified there.

In some cases it is possible that the JVM default truststore contains a certificate which is not correct for certain uses-cases, and interferes with the one provided in -Djavax.net.ssl.trustStore. This can happen if those certificates have different aliases. Note, that nevisAuth cannot control how certificates are picked during the ssl handshake. In case you have this issue, configure the truststore used in the -Djavax.net.ssl.trustStore directly on the HttpClient level. In this case the JVM default truststores will not load, and there will be no interference.

Overwritten default JVM truststore in nevisAdmin4 setups

nevisAuth instances managed by nevisAdmin4 will usually overwrite the JVM default truststore used by setting the -Djavax.net.ssl.trustStore system property in the env.conf file. This simplifies the Nevis setup as nevisAdmin4 will use automatic key management to ensure Nevis components are able to communicate with each other by automatically adding the certificates for other components such as nevisIDM, nevisAdmin4 and so on to this configured truststore. However, as a downside, the default JVM truststore is no longer directly accessible.

Response parsing

Http response decoding by default uses UTF-8 charset. You can change that manually if required, see example.

HttpClient configuration

The properties are grouped into:

• Key- and truststore properties
• Proxy properties
• Connection properties
• Authorization properties
• General properties

Key- and truststore properties

• httpclient.tls.hostnameVerification (Boolean, optional, true)

  Default value: true

  Defines whether to verify the hostname in the certificate presented by the HTTP server against the HTTP server hostname. By default, the hostname verification is enabled. If set to false, the hostname verification will not be performed.

• httpclient.tls.keyObjectRef (String, optional, -)

  This property configures the key material to use when validating the access token. The access token is decrypted using the configured private key. The referenced keystore must be defined inside a KeyStore element of the nevisAuth esauth4.xml configuration.

  Format

  <keystore-reference>/<keyobject-reference>
  Example

  DefaultKeyStore/DefaultSigner

  Variable substitution is only allowed for the whole property value, for example: ${keystore-and-keyobj-ref}

  Unique key objects

  If the key object name is unique in the esauth4.xml file, it’s possible to bypass the key store as part of the property value; e.g. DefaultSigner. If a non-unique key object is referenced, nevisAuth will throw a BadConfigurationException during startup.

• httpclient.tls.trustStoreRef (String, optional, JVM default)

  The keystore reference that is being used as truststore. The referenced keystore must be defined inside a KeyStore element of the nevisAuth esauth4.xml configuration.

  If this value is not supplied, the default JVM truststore is used.

• httpclient.tls.trustAll (Boolean, optional, false)

  Default value: false

  If this property is set to true the certificate validation is disabled and all certificates are automatically trusted.

  It's not recommended to use this property in production environments.

Proxy properties

• httpclient.proxy.host (String, URL, optional, -)

  Hostname of outbound proxy. If this property is not specified, the system assumes that no proxy must be used and ignores the other proxy-related properties.

• httpclient.proxy.port (Integer, optional, -)

  Port of outbound proxy.

• httpclient.proxy.username (String, optional, -)

  The name of the user to be used to authenticate against the outbound proxy.

• httpclient.proxy.password (String, optional, -)

  The password to be used to authenticate against the outbound proxy.

Connection properties

• httpclient.connection.pool.maxTotal (Integer, optional, 200)

  The maximum number of total open connections in the connection pool.

• httpclient.connection.pool.maxPerRoute (Integer, optional, 100)

  The maximum number of connections per route in the connection pool.

• httpclient.connection.pool.timeout (String, optional, 0)

  Default value: 0

  The HTTP client pool keep-alive timeout value. With the default value of 0 there is no timeout (Infinite).

  The property allows configuring the timout in string notation for hours minutes and seconds.

  Supported format

  [n]H[n]M[n.n]S
  Examples

  15s
  1m
  1h3m12.305s

• httpclient.connection.timeout (String, optional, 30)

  Default value: 30

  The HTTP client connection timeout value. With the default value of 30 seconds. The connection timeout defines how long the communication between a client and a server is allowed to take. The property is applied to the following: getting a connection from the pool (reuqest connection timeout), connecting to the specified url (connection timeout), receiving response from the target (socket timeout).

  The property allows configuring the timout in string notation for hours minutes and seconds.

  Supported format

  [n]H[n]M[n.n]S
  Examples

  15s
  1m
  1h3m12.305s

Authorization properties

Basic authorization

• httpclient.authorization.basic.username (String, optional, -)

  The username to be used for basic authorization.

• httpclient.authorization.basic.password (String, optional, -)

  The password to be used for basic authorization.

SecToken authorization

• httpclient.authorization.basic.sectoken.userId (String, optional, -)

  The user ID that should be used for the generation of the SecToken

• httpclient.authorization.basic.sectoken.profileId (String, optional, -)

  The profile ID that should be used for the generation of the SecToken

• httpclient.authorization.basic.sectoken.roles (String, optional, -)

  A list of roles separated by ,. The user roles that should be used for the generation of the SecToken

info

SecTokens are valid for 30 minutes when they expire a new one will be generated automatically. To avoid any issues it will be done way before the expiration.

General properties

• httpclient.followRedirects (Boolean, optional, false)

  Default value: false

  If set to true, the HTTP client will follow HTTP redirects according to the HTTP specification. The following HTTP status codes will result in an automatic redirect of HEAD and GET methods only. POST and PUT methods will not be automatically redirected as requiring user confirmation.

  • 302 Moved Temporarily
  • 301 Moved Permanently
  • 307 Temporary Redirect

• httpclient.forwardCookies (Boolean, optional, false)

  Default value: false

  If set to true, the HTTP client will forward obtained cookies.

Logging

For analyzing the HTTP client configuration or the outgoing HTTP traffic, you can configure a logger with name HttpClient on DEBUG.

- name: HttpClient
  level: DEBUG

System properties

• -Dch.nevis.auth.httpclient.tls.defaultTrustStore (String, optional, $JAVA_HOME/jre/lib/security/cacerts)

  Default value: $JAVA_HOME/jre/lib/security/cacerts

  Special JVM system property allows to overwrite the default path of the JVM truststore used by the HTTP client when no explicit truststore is configured. It is usually not necessary to set this property unless specific JVM vendors or setups use different non-standard paths for storing the root-CA certificates. This will be applied to both "global" auth and instance based http clients. The path must be supplied as absolute path starting from the file system root!

info

entity() This property exists because the HTTP client does not have access to the JVM default truststore if it is overwritten by setting the system property -Djavax.net.ssl.trustStore which is done when configuring the nevisAuth instance in nevisAdmin4 with automatic key management in place.

caution

Do not set this property unless nevisAuth instructs you to do so. nevisAuth will generate the following error message in the log in case the path is wrong: Failed to load the Java default truststore certificates from ... to trust root-CA signed official certificates This property is not intended to provide a custom-managed default truststore, use the httpclient.tls.trustStoreRef property instead.

Programmatic configuration

caution

Prefer the usage of property based configuration, and only rely on programmatic configuration if it is necessary that configuration is created by programmatic means instead from a static configuration integrators have access to.

The configuration of a Http Client can also be created programatically, with the builder method of the HttpClientConfiguration class.

Example programmatic configuration:

Create

HttpClientConfiguration httpClientConfiguration = HttpClientConfiguration.builder()
        .tls(TlsConfiguration.builder()
                .hostnameVerification(true)
                .trustAll(false)
                .withTrustStoreConfiguration(KeyStoreConfiguration.builder()
                        .path("/path/to/truststore.jks")
                        .password("password".toCharArray())
                        .build())
                .withKeyStoreConfiguration(KeyStoreConfiguration.builder()
                        .path("/path/to/keystore.jks")
                        .password("password".toCharArray())
                        .alias("alias")
                        .build())
                .build())
        .authorization(AuthorizationConfiguration.builder()
                .secTokenConfiguration(SecTokenConfiguration.builder()
                        .userId("userId")
                        .hostname("hostname")
                        .profileId("profileId")
                        .roles(Set.of("admin"))
                        .build())
                .build())
        .connectionPoolTimeout(Duration.ofMinutes(5))
        .connectionTimeout(Duration.ofMinutes(5))
        .followRedirects(true)
        .forwardCookies(true)
        .maxPerRoute(100)
        .maxTotal(100)
        .proxy(ProxyConfiguration.builder()
                .host("https://proxy")
                .port(3128)
                .password("password".toCharArray())
                .username("username")
                .build())
        .build();

HttpClient httpClient = HttpClients.create(httpClientConfiguration);

HttpClient API documentation

Http

This factory class will generate HTTP objects:

ch.nevis.esauth.util.httpclient.api.Http

public abstract class Http {

    /**
     * Create a builder object that can be used to build a HTTP GET request.
     */
    public static HttpRequestBuilder get();

    /**
     * Create a builder object that can be used to build a HTTP DELETE request.
     */
    public static HttpRequestBuilder delete();
    
    /**
     * Create a builder object that can be used to build a HTTP POST request.
     */
    public static HttpRequestBuilder post();

    /**
     * Create a builder object that can be used to build a HTTP PUT request.
     */
    public static HttpRequestBuilder put();

    /**
     * Create a builder object that can be used to build a HTTP PATCH request.
     */
    public static HttpRequestBuilder patch();

    /**
     * Create a builder which can be used to create an entity object that can bee built into an HTTP request
     */
    public static EntityBuilder entity();
}

HttpRequestBuilder

This builder class will help building HTTP requests. Instances of it can be obtained from the Http factory class.

ch.nevis.esauth.util.httpclient.api.HttpRequestBuilder

public interface HttpRequestBuilder {

    /**
    * Build the URI of the request in various ways.
    * Note that subsequent calls will override each other and that url parameters can be built into the request via
    * the HttpRequestBuilder # urlParameter(String, String) method.
    */
    HttpRequestBuilder url(String url);
    HttpRequestBuilder url(URI uri);
    HttpRequestBuilder url(URL url);

    /**
    * Builds an entity into the HTTP request. It is only allowed for the following HTTP methods: POST, PATCH, PUT
    */
    HttpRequestBuilder entity(Entity entity);

    /**
     * Builds a header into the request.
     * Can be called multiple times, all headers will be applied to the request.
     */
    HttpRequestBuilder header(String key, String value);

    /**
     * Builds an Instant header into the request.
     * Can be called multiple times, all headers will be applied to the request.
     */
    HttpRequestBuilder header(String key, Instant value);

    /**
     * Builds a parameter into the URI of the request.
     * Can be called multiple times, all parameters will be appended.
     */
    HttpRequestBuilder urlParameter(String key, String value);

    /**
     * Builds the request.
     * The instance is immutable, once this is called, no operations on it is allowed.
     */
    HttpRequest build();
}

HttpRequest

Once a HttpRequest object is built, it can be executed with one of the methods found on the the HttpRequest class itself:

ch.nevis.esauth.util.httpclient.api.HttpReqest

public interface HttpRequest {
    /**
     * Executes the HttpRequest with the Auth "global" HTTP Client, shared among all threads and AuthStates.
     *
     * In case fine-tuned configuration or isolation is required for the request, consider creating a custom HTTP Client.
     *
     * Note that the returned HttpResponse contains the content of the HTTP response already read into
     * memory.
     */
    HttpResponse send() throws IOException;

    /**
     * Executes the HttpRequest with the provided HTTP client instance.
     *
     * In case the HTTP client is not preserved afterwards, consider not creating it in the first place, and
     * executing requests instead with the defeault Auth HTTP Client, via calling just send().
     *
     * Note that the returned HttpResponse contains the content of the HTTP response already read into
     * memory.
     */
    HttpResponse send(HttpClient httpClient) throws IOException;
}

HttpResponse

Once a HTTP request is executed, the API will return a HttpResponse object.

ch.nevis.esauth.util.httpclient.api.HttpResponse

public interface HttpResponse {

    /**
     * All headers of the HTTP request.
     * Header keys do not need to be unique.
     * See <a href="https://www.rfc-editor.org/rfc/rfc7231#page-33">Request Header fields</a>.
    */
    Map<String, List<String>> headers();

    /**
     * Note that header names do not need to be unique, but in practice they often are, hence this method.
     * @return the first header found for a given name.
     */
    String header(String headerKey);

    /**
     * Parses and returns a header value to Instant from the response.
     *
     * The input must comply with the standards: RFC1123, RFC1036, ASCTIME.
     * Which support the following formats respectively:
     * - "EEE, dd MMM yyyy HH:mm:ss zzz"
     * - "EEE, dd-MMM-yy HH:mm:ss zzz"
     * - "EEE MMM d HH:mm:ss yyyy
     * 
     * HTTP servers do usually follow one of these standards when they issue time values in headers.
     */
    Instant headerDate(String headerKey);

    /**
     * HTTP code of the response.
     */
    int code();

    /**
     * The reason the accessed HTTP Server associated with the response, typically set in error responses.
     */
    String reasonPhrase();

    /**
     * Bytes of the body of the HTTP response, or empty if no body was in the response.
     */
    Optional<byte[]> body();

    /**
     * Stringified UTF-8 body of the HTTP response, or an empty String if no body was in the response.
     * In case different encoding is expected, call `body()` instead and parse it accordingly.
     */
    String bodyAsString();

    /**
     * Stream of the body of the HTTP response, empty if no body was in the response.
     * Note that the body is always read into memory, the returned stream does not equal to the response body the system created.
     */
    Optional<InputStream> bodyAsStream();
}

HttpClients

HTTP clients can be created at this factory. Creating a client is recommended if it can be stored for longer period time or if a client with custom configuration is needed.

ch.nevis.esauth.util.httpclient.api.HttpClients

/*
* If possible, avoid creating new HTTP client instances for executing a single HTTP request, 
* use HttpRequest # send() for that purpose, which will use the default Auth HTTP Client.
*/
public abstract class HttpClients {

    /**
     * Creates an HTTP Client with all the default configuration values.
     * It has the same effect as calling HttpClients.create(new Properties())
     *
     * Calling this method will create a new HTTP Client, so it does not equate to using the Auth "global"
     * HTTP client.
     */
    public static HttpClient createWithDefaults() throws BadConfigurationException;

    /**
     * Creates an HTTP Client with the provided properties used as configuration.
     * All properties intended for the HTTP Client MUST start with the prefix 'httpclient',
     * otherwise they will be ignored. Furthermore, non-existent configuration started with this prefix
     * will result in a BadConfigurationException thrown.
     * All configuration properties are optional, defaults will be used in case they are not provided.
     *
     * <p>Example:
     * <code>
     *     httpclient.forwardCookies=true
     *     httpclient.tls.trustStoreRef=mytruststore
     *     httpclient.connection.timeout=1m
     * </code>
     */
    public static HttpClient create(Properties properties) throws BadConfigurationException;

    /**
     * Creates an HTTP Client from preparsed configuration.
     */
    public static HttpClient create(HttpClientConfiguration config) throws HttpClientCreationException;

    /**
     * This creator simplifies the creation of HTTP Clients in ScriptState scripts, where type check is not always convenient.
     */
    public static HttpClient create(Map<Object, Object> propertiesMap) throws BadConfigurationException;
}

HttpClient

Create HTTP clients with the HttpClients factory class.

ch.nevis.esauth.util.httpclient.api.HttpClient

/**
 * Represents an HTTP Client that can be used to execute HTTP requests.
 *
 * It is recommended to store instances of this interface once created.
 * It is also recommended not to create new HTTP client instances for executing a single HTTP request, 
 * use HttpRequest # send() for that purpose, which will use the default Auth "global" HTTP Client.
 *
 * Once not needed, the HTTP Client should be closed, via HttpClient # close() to free the associated system
 * resources.
 */
public interface HttpClient extends Closeable {
    HttpResponse execute(HttpRequest request) throws IOException;
}

URIBuilder

A helper class that can be used to build java.net.URI objects concisely with the builder pattern.

ch.nevis.esauth.util.httpclient.api.uri.URIBuilder

public interface URIBuilder {

    /**
     * Builds the scheme of the URI.
     * It is recommended to use URIBuilder # http() and URIBuilder # https() creators for HTTP and HTTPs.
     */
    URIBuilder scheme(String scheme);

    /**
     * Builds an IPv4 address in dotted-decimal form, or a registered name.
     */
    URIBuilder host(String host);

    /**
     * Builds 'localhost' as a URIBuilder # host(String) into the URI.
     */
    URIBuilder localhost();

    /**
     * Builds a port number in decimal.
     */
    URIBuilder port(int port);

    /**
     * Appends a path to the URI - can be called multiple times to append multiple paths.
     * Provided paths must be prefixed with '/'.
     */
    URIBuilder path(String path);

    /**
     * Builds all query parameters into the URI.
     * This method is mutually exclusive with building URIBuilder # query(String). It can work together with
     * URIBuilder # parameter(String, String) though, both will be added to the parameters.
     */
    URIBuilder parameters(Map<String, String> parameters);

    /**
     * Builds a single query parameter into the URI.
     * This method is mutually exclusive with building URIBuilder # query(String). It can work together with
     * URIBuilder # parameters(Map) though, both will be added to the parameters.
     */
    URIBuilder parameter(String key, String value);

    /**
     * Builds the whole query into the URI.
     *
     * This method is mutually exclusive with building URIBuilder # parameter(String, String) and
     * URIBuilder # parameters(Map).
     */
    URIBuilder query(String query);

    /**
     * Builds the fragment into the URI. The fragment identifier component of a URI allows indirect
     * identification of a secondary resource.
     * See <a href="https://www.rfc-editor.org/rfc/rfc3986#section-3.5">Fragment</a>.
     */
    URIBuilder fragment(String fragment);

    /**
     * Finish building the URI. Once called, this instance may not be built upon again.
     *
     * In case the URI could not be built, an IllegalArgumentException is thrown wrapping the
     * URISyntaxException.
     */
    URI build();

    /**
     * Start building a URI with no pre-builds.
     * It is recommended to use URIBuilder # http() and URIBuilder # https() creators for HTTP and HTTPs.
     */
    static URIBuilder create();

    /**
     * Start building a URI by pre-building http or https as the scheme.
     */
    static URIBuilder http();
    static URIBuilder https();

    /**
     * Start building a URI various ways, by pre-building the provided uri.
     */
    static URIBuilder from(String uri);
    static URIBuilder from(URI uri);
}

EntityBuilder

A helper class that can be used to build entity objects, that can be built into HTTP requests.

ch.nevis.esauth.util.httpclient.api.EntityBuilder

public interface EntityBuilder {

    /**
     * Builds the content type the entity.
     * Can be called once, subsequent calls will override the already built value.
     */
    EntityBuilder contentType(String contentType);

    /**
     * Builds the String charset of the entity. It will only be considered, if the
     * EntityBuilder # contentType(String) is built into the entity.
     * Can be called once, subsequent calls will override the already built value.
     */
    EntityBuilder charset(Charset charset);

    /**
     * Builds the {@link Charset} of the entity. It will only be considered, if the
     * EntityBuilder # contentType(String) is built into the entity.
     * Can be called once, subsequent calls will override the already built value.
     */
    EntityBuilder charset(String charset);

    /**
     * Builds the String content of the body of the entity.
     * Can be called once, subsequent calls will override the already built value.
     *
     * his method is mutually exclusive with EntityBuilder # formParameter(String, String) and
     * EntityBuilder # stream(ByteArrayInputStream), calling multiple of them will result in an
     * IllegalStateException!
     */
    EntityBuilder content(String content);

    /**
     * Builds the Stream content of the body of the entity.
     * <p>Can be called once, subsequent calls will override the already built value.
     *
     * This method is mutually exclusive with EntityBuilder # formParameter(String, String)} and
     * EntityBuilder # content(String), calling multiple of them will result in an
     * IllegalStateException!
     */
    EntityBuilder stream(ByteArrayInputStream is);

    /**
     * Builds a form parameter into the body of the entity.
     * Can be called multiple times, all form parameters will be built into the request.
     *
     * This method is mutually exclusive with EntityBuilder # content(String) and
     * EntityBuilder # stream(ByteArrayInputStream), calling multiple of them will result in an
     * IllegalStateException!
     */
    EntityBuilder formParameter(String key, String value);

    /**
     * Builds the entity.
     * The instance is immutable, once this is called, no operations on it is allowed, so only call this once everything
     * is built correctly via the EntityBuilder (this) interface.
     */
    Entity build();
}