Troubleshooting

This chapter describes some known issues and how to solve them.

Performance optimization notice

• To keep a high-performance functioning of regexp, remove irrelevant wildcard patterns. like .*. For example, instead of .*someValue.*, you can use directly someValue.
• Enable keep-alive connections between the proxy and the application.

Authentication with a client certificate and OpenSSL 1.1.1

As of release 3.14.3.0, authentication with a client certificate possibly does not work as expected anymore. This is due to the upgrade of nevisProxy to OpenSSL version 1.1.1. The problems may occur if you have set the parameter ClientCert of the IdentityCreationFilter to "optional" or "optional_no_ca".

There are two possible workarounds to fix this problem:

Via the IdentityCreationFilter

If you want to fix this problem via the IdentityCreationFilter, set the filter parameter StoreClientCert to "true".

This fix works without any performance loss, but is not so secure: A user using a smart card will not be logged out automatically if he removes the smart card from the slot. To avoid this, use the crypto API of the browsers to notify the application of card removal events. Make the application act accordingly.

Via the ApacheConfigFilter

If you want to fix this problem via the ApacheConfigFilter, configure the filter as follows:

<filter>
    <filter-name>ClientCertApacheConfigFilter</filter-name>
    <filter-class>ch:nevis::navajo::apglue::httpd_${HTTPD_LIB_VERSION}_x::servlet::ApacheConfigFilter</filter-class>
    <filter-lib>/opt/nevisproxy/webapp/WEB-INF/lib/libApache${HTTPD_LIB_VERSION}_Servlet.so.1</filter-lib>
    <init-param>
        <param-name>
            LocationConfig
        </param-name>
        <param-value>
            SSLVerifyClient optional_no_ca
            SSLVerifyDepth 2
        </param-value>
    </init-param>
</filter

Pay attention to the following points:

• Set the value of the parameter SSLVerifyClient to "optional", if the parameter ClientCert of the IdentityCreationFilteris also set to "optional" (<SSLVerifyClient> field).
• Give the parameter SSLVerifyDepth the same value as the parameter ClientCert of the IdentityCreationFilter (<SSLVerifyDepth> field).
• Map the filter to the same URL as the one set in the IdentityCreationFilter.

This is the most secure workaround. But it can slow down performance, especially if you use a smart card. To optimize performance, increase the value of the attribute MaxKeepAliveRequests and/or KeepAliveTimeout in the navajo.xml configuration file.

Depending on the smart card used, the user may be forced to enter its password after the keep-alive timeout expires.

Lost POST request body in LuaFilter in case of chunked transfer-encoding

The LuaFilter sometimes omits the body of a chunked POST request. This may happen if the LuaFilter comes before the IdentityCreationFilter in the filter chain.

To solve this problem, add the modifier flag RESET_BUFFERED_BODY to the configuration of the respective LuaFilter:

<init-param>
    <param-name>ModifierFlags</param-name>
    <param-value>+RESET_BUFFERED_BODY</param-value>
</init-param>

Increase the number of open file descriptors

Under heavy load, nevisProxy may log the following error message:

2021 04 28 12:48:31.306 4.5.0.0rc2 Apache     06027.140088593544960 3-ERROR :  AH02179: apr_socket_accept: (client socket) [APAC-0005]
2021 04 28 12:48:31.769 4.5.0.0rc2 Apache     125211.140091297666880 3-ERROR :  AH03490: scoreboard is full, not at MaxRequestWorkers. Increase ServerLimit. [APAC-0005]

This could happen if the number of open file descriptors is too low. To solve this issue, there is two possible solution:

1. Adapt it only for the nevisProxy service: as this change does not survive a reinstallation or upgrade, we only recommend this solution for testing purposes.
2. Increase it on system level: this is the recommended way for hosts running a nevisProxy instance. Requires a reboot of the host machine but is permanent from the instance view.

Service level

To increase it only for the nevisProxy service, adapt the /usr/lib/systemd/system/[email protected] file by setting the LimitNOFILE parameter in the [Service] section, for example:

...
ExecStop=/opt/nevisproxy/bin/nevisproxy %i stop
KillMode=control-group
 
LimitNOFILE=10000
 
[Install]
WantedBy=multi-user.target

Then reload the new configuration by running the command systemctl daemon-reload. As this method is not permanent (nevisProxy reinstallation or upgrade can overwrite this file), it is only recommended for testing purposes.

System level

Another solution is to increase it for the whole system by setting the parameter DefaultLimitNOFILE in the /etc/systemd/system.conf file:

...
#DefaultLimitRSS=
DefaultLimitNOFILE=10000
#DefaultLimitAS=
...

This modification requires reboot to take effect. The new value can be verified after restarting the nevisProxy instance, it should log the resource limitations:

2021 05 03 16:12:42.114 5.5.5.5    NavajoStar 29674.140620893026112 5-NOTICE:           using 'RLIMIT_FSIZE' hardlimit='-1' softlimit='-1'
2021 05 03 16:12:42.114 5.5.5.5    NavajoStar 29674.140620893026112 5-NOTICE:           using 'RLIMIT_NOFILE' hardlimit='10000' softlimit='10000'
2021 05 03 16:12:42.114 5.5.5.5    NavajoStar 29674.140620893026112 5-NOTICE:           using 'RLIMIT_STACK' hardlimit='-1' softlimit='8388608'

Or by calling the cat /proc/<pid>/limits command with the parent or child process of nevisProxy, for example:

10:55 $ cat /proc/38619/limits
Limit                     Soft Limit           Hard Limit           Units    
Max cpu time              unlimited            unlimited            seconds  
Max file size             unlimited            unlimited            bytes    
Max data size             unlimited            unlimited            bytes    
Max stack size            8388608              unlimited            bytes    
Max core file size        0                    unlimited            bytes    
Max resident set          unlimited            unlimited            bytes    
Max processes             63335                63335                processes
Max open files            10000                10000                files    
Max locked memory         65536                65536                bytes    
Max address space         unlimited            unlimited            bytes    
Max file locks            unlimited            unlimited            locks    
Max pending signals       63335                63335                signals  
Max msgqueue size         819200               819200               bytes    
Max nice priority         0                    0                   
Max realtime priority     0                    0                   
Max realtime timeout      unlimited            unlimited            us

OpenSSL compatibility

Because nevisProxy usually uses the most recent OpenSSL version, it is possible that some backends will not be accessible anymore due to a stricter security policy. For example, because a cipher algorithm has been disabled or some key size has been restricted.

Backend problems

If a backend cannot be reached anymore, check the possible cause of it in navajo.log. Usually an SSL/TLS error should be logged. For example:

2016 02 08 20:46:14.011 isi3web OpenSSLHan 18203.140737049229056.471b-14549607721132811-152c26b8412 3-ERROR : OpenSSL-failure: 140737049229056:error:14082174:SSL routines:SSL3_CHECK_CERT_AND_ALGORITHM:dh key too small:s3_clnt.c:3448:0x0a

In this case, the dh-key (Diffie-Hellman) provided by the backend is too small. You can now either update your backend or remove the DH-algorithm from the SSLCipherSuites of the HttpsConnectorServlet (e.g., "!aNULL:!aDSS:!MD5:!kEDH+AES:RSA+AES:!kEDH+3DES:RSA+3DES").

In any case, it is better to keep the backends up to date. If you run a java-based backend, you can set the following java-property:

• javax.net.debug=ssl,handshake

This will provide more information on the backend on why it refuses the connection. Common cases are:

• Starting from java8, SSLv3 and MD5 are not supported anymore. In this case you should update your certificates.
• Starting from java8, the Diffie-Hellman key must be at least 1024 bytes.

If for some reason you can't update the backend with new ciphers, you can set the SSLCipherSuites of the HttpsConnectorServlet to: DEFAULT:@SECLEVEL=0

And most probably you will have to allow TLSv1.1 as well in the parameter SSLProtocol: -all +TLSv1.2 +TLSv1.3 +TLSv1.1

For security reasons this is not recommended and you should keep your backend up to date with the best available ciphers.

Frontend problems

If the proxy can't start (due to a certificate with not valid ciphers) you get an error like this:

2023 05 09 10:21:40.731 isi3web    Apache     101577.140468476995008 0-ERROR :  SSL Library Error: error:0A00018E:SSL routines::ca md too weak [APAC-0054]

In that case we strictly recommend to create new certificates. If this is not possible, you can set the SSLCipherSuite in navajo.xml to:

SSLCipherSuite="DEFAULT:@SECLEVEL=0"

Depending on the certificate you may have to allow TLSv1.1 as well in the parameter SSLProtocol:

SSLProtocol="-all +TLSv1.2 +TLSv1.3 +TLSv1.1"

For security reasons this is not recommended and you should update your certificate as soon as possible.

Slow response body processing with ModSecurity

When ModSecurity is configured to inspect the response body, it analyses the body from the beginning each time it receives a new chunk of data. If the response body is split in small chunks, many iterations are performed, and this can slow down the processing of the response. You can check the size of the response chunks by raising the log level of your Http(s)ConnectorServlet to 9, and searching for calls to sendResponseBody, for example:

2024 02 06 11:16:27.233 9.9.9.0    IW4HttpsCo 124587.139693714491136.1dc193d4417d54839d192617ed6307c0.2aafcb04a6a23f7a 9-DBG_HI:        HttpConnectorServlet_1_0::sendResponseBody (self=0x5594a69b6b20) {
2024 02 06 11:16:27.233 9.9.9.0    IW4HttpsCo 124587.139693714491136.1dc193d4417d54839d192617ed6307c0.2aafcb04a6a23f7a 9-DBG_HI:           HttpConnectorServlet_1_0::sendResponseBody: (HttpsConnectorServlet) 8192 bytes read from backend
2024 02 06 11:16:27.234 9.9.9.0    IW4HttpsCo 124587.139693714491136.1dc193d4417d54839d192617ed6307c0.2aafcb04a6a23f7a 9-DBG_HI:           HttpConnectorServlet_1_0::sendResponseBody: (HttpsConnectorServlet) 6591 bytes read from backend
2024 02 06 11:16:27.234 9.9.9.0    IW4HttpsCo 124587.139693714491136.1dc193d4417d54839d192617ed6307c0.2aafcb04a6a23f7a 9-DBG_HI:           HttpConnectorServlet_1_0::sendResponseBody: (HttpsConnectorServlet) 1127 bytes read from backend

For the BackendConnectorServlet, set the log level to 8, and search for calls to receiveResponse, for example:

2024 02 06 11:33:59.283 9.9.9.0    NPBackendS 138232.140332129797888.effd052c1de1f2f1e20687bdd29e5d9f.4f31efa4fbf0c2d0 8-DBG_ME:  HttpBackend_1_0::receiveResponse: (BackendConnectorServlet) starting reading response
2024 02 06 11:33:59.283 9.9.9.0    NPBackendS 138232.140332129797888.effd052c1de1f2f1e20687bdd29e5d9f.4f31efa4fbf0c2d0 8-DBG_ME:  HttpBackend_1_0::receiveResponse: (BackendConnectorServlet) 8192 bytes read from backend
2024 02 06 11:33:59.284 9.9.9.0    NPBackendS 138232.140332129797888.effd052c1de1f2f1e20687bdd29e5d9f.4f31efa4fbf0c2d0 8-DBG_ME:  HttpBackend_1_0::receiveResponse: (BackendConnectorServlet) 1884 bytes read from backend
2024 02 06 11:33:59.284 9.9.9.0    NPBackendS 138232.140332129797888.effd052c1de1f2f1e20687bdd29e5d9f.4f31efa4fbf0c2d0 8-DBG_ME:  HttpBackend_1_0::receiveResponse: (BackendConnectorServlet) finished reading response, 10076 bytes read in total

The Http(s)ConnectorServlet and BackendConnectorServlet read up to 8 kB of data at once. If many smaller chunks are reported in the logs, analysing large response bodies with ModSecurity can be slow.

There are two options to speed up the processing:

• either configure your backend server to send larger chunks of data when possible;
• or configure a LuaFilter to buffer the response body into larger chunks before it is analysed by ModSecurity. See the example examples/various/LuaFilter_buffering.example in your nevisProxy installation folder, and adapt the buffer size to reduce the number of chunks, for example to 8192 bytes. Then map the LuaFilter between the ModSecurityFilter and the servlet.