Understanding JDBC Internals & Timeout Configuration posted 11 months ago in Dev

Understanding JDBC Internals & Timeout Configuration

 

 

 

An application with a proper JDBC timeout can cut down the failure time. In this article we would like to talk about different kinds of timeout values and recommended timeout application methods when you import values from DBMS.

Web Application Server became unresponsive after a DDos attack one day

(This is a close reconstitution of an actual event.)

The entire service did not work normally after a DDos attack. The network was disconnected because L4 was not working, which caused WAS to be inoperable as well. Shortly afterwards, the security team blocked all DDos attacks, and restored the network back to normal. Yet, WAS was still not working.

Through the ThreadDump of WAS, the service team was able to confirm that WAS had stopped during API call from JDBC. After 20 minutes, WAS was still in WAITING status and the service was still not working. About 30 minutes had passed when an exception suddenly occurred, and the service was restored.

Why was WAS in WAITING status for 30 minutes when QueryTimeout value was set to 3 seconds, and why did WAS start working again after 30 minutes?

You can find the answer if you understand how the JDBC Timeout works.

Why Do We Need to Know about the JDBC Driver?

When there is a performance issue or an error, WAS and DBMS are the two important tiers we pay attention to. In NHN, WAS and DBMS are generally handled by different departments, so each department tries to figure out this situation by focusing on their own area of expertise. When this happens, you get a blind spot between WAS and DBMS, that does not receive much attention. For Java applications, the blind spot would be between DBCP and JDBC. In this article we will focus on JDBC.

What is a JDBC Driver?

JDBC is a standard API that you use to access the DBMS in Java applications. There are 4 types of JDBC drivers(Wikipedia) defined by Sun. NHN mainly uses the type 4. JDBC type 4 driver is written entirely in Java (pure Java) and communicates with a DBMS using sockets in Java applications.

 

Figure 1: JDBC Type 4.

Type 4 drivers process byte stream via sockets, and have the same basic operations as a network library likeHttpClient. This uses up a lot of CPU resources and loses response timeout, while sharing the same error points with other network libraries. If you have used HttpClient before, then you must have encountered errors from not setting the timeout value. Type 4 driver may have the same error (a hang occurs) if the socket timeout value is not set properly.

Let's learn about how to configure the socket timeout value for JDBC driver, and what needs to be considered.

 

Timeout Class at WAS - DBMS Communication

 

 

Figure 2: Timeout Class.

Figure 2 above shows a simplified version of the timeout class when WAS and DBMS are communicating.

The higher level timeout is dependent on the lower level timeout. The higher level timeout will operate normally only if the lower level timeout operates normally as well. If the JDBC driver socket timeout does not work properly, then higher level timeouts such as statement timeout and transaction timeout will not work properly either.

We have received a lot of comments that said:

Even after the statement timeout was configured, the application still did not recover from the error because the statement timeout did not work at the time of network failure.

The statement timeout does not handle the timeouts at the time of network failure. Statement timeout does only one thing: restricts the operation time of 1 statement. Handling timeout to prevent network failure must be done by JDBC Driver.

The JDBC driver's socket timeout is affected by the OS's socket timeout configuration. This would explain why JDBC connection hang recovers 30 minutes after the network connection failure, even when the JDBC driver's socket timeout is not configured. 

DBCP Connection Pool is located on the left side of Figure 2. You can see that the timeout classes and DBCP are separated. DBCP is in charge of creating and managing connections, and is not involved in processing timeouts. When a connection is created within DBCP or a validation query is sent to check the validity of the connection, the socket timeout does affect these processes but does not affect the application directly.

However, when getConnection() is called to DBCP from the application logic, then you can specify the timeout until the application acquires the connection. However, this has nothing to do with the JDBC's connect timeout. 

 

Figure 3: Timeout for Each Levels.

What is Transaction Timeout?

Transaction timeout is a timeout valid in frameworks (Spring, EJB container) or at the application level.

Transaction timeout can be an unfamiliar concept. Simply put, transaction timeout is "Statement Timeout * N (number of statements being processed) + @ (garbage collection, etc.)." Transaction timeout is used to limit the total statement processing time to the maximum amount allowed.

For example, if it takes 0.1 second to process 1 statement, processing a few statements would not be a problem, but processing 100,000 statements would take 10,000 seconds (approx. 7 hours). Statement timeout can be used here.

EJB CMT (Container Managed Transaction) would be a typical example of actual implementations. EJB CMT varies in its implementation methods and operating process depending on developers. NHN does not use EJB Container, so transaction timeout of Spring Framework would be the most common example. In Spring, you may use XML as shown below or use @Transactional from Java source codes, for configuration.

 

<tx:attributes>
        <tx:method name="…" timeout="3"/>
</tx:attributes>

 

 

Statement timeout provided by Spring is very simple. It records the starting time and the elapsed time for each transaction, and checks the elapsed time when an event occurs. If the timeout is abnormal, it generates an exception. 

In Spring, the connection is stored in, and used from ThreadLocal. This is called Transaction Synchronization. When a connection is saved in ThreadLocal, the starting time and the timeout time of the transaction is also recorded. When a statement is being created by using the proxy connection, the elapsed time is checked to generate an exception.

The EJB CMT implementation is done in a similar way. The structure itself is very simple. If the transaction timeout is very important but the container or the framework you are using does not provide this feature, you could implement it yourself without major problems. There is no standard API for transaction timeout.

Lucy 1.5 and 1.6 Framework does not have a transaction timeout feature, but you can get the same result by using Transaction Manager from Spring. 

If the processing time of the statement (5 or less) is 200 ms and the processing time of other business logics or framework operation is 100 ms, the transaction timeout time should be set to 1,100 ms ((200 * 5) + 100) or more.

What is Statement Timeout?

It is a limitation on how long a statement should run. It sets the timeout value for the statement, which is a JDBC API. The JDBC driver processes the statement timeout based on this value. Statement timeout is configured via java.sql.Statement.setQueryTimeout(int timeout), which is a JDBC API. In recent developing environments, the developers rarely configure the statement timeout value directly through Java source codes, but often configure it by using the framework.

To use iBatis as an example, the default value can be configured by using @defaultStatementTimeoutvalue in sqlMapConfig/settings of sql-map-config.xml. By using @timeout value, you can configure statement, select, insert and update syntax of sql-map.xml separately.

When MangedDatasource of Lucy 1.5 and 1.6 is used, the queryTimeout option can be used to get a statement of which timeout is configured at the datasource level.

The statement timeout time is configured based on the features of each application, so there is no recommended configuration value. 

Statement Timeout Execution Process for JDBC Driver

Statement timeout works differently per DBMS and driver. The way it works is similar between Oracle and MS SQLServer. It is also similar between MySQL and CUBRID.

 

QueryTimeout for Oracle JDBC Statement

 

1. Creates a statement by calling Connection.createStatement().
2. Calls Statement.executeQuery().
3. The statement transmits the Query to Oracle DBMS by using its own connection.
4. The statement registers a statement to OracleTimeoutPollingThread (1 for each classloader) for timeout process.
5. Timeout occurs.
6. OracleTimeoutPollingThread calls OracleStatement.cancel().
7. Sends a cancel message through the connection and cancels the query being executed.

 

Figure 4: Query Timeout Execution Process for Oracle JDBC Statement.

QueryTimeout for JTDS (MS SQLServer) Statement

 

1. Creates a statement by calling Connection.createStatement().
2. Calls Statement.executeQuery().
3. The statement transmits the Query to MS SqlServer by using the internal connection.
4. The statement registers a statement in TimerThread for timeout process.
5. Timeout occurs.
6. TimerThread calls up TsdCore.cancel() inside the JtdsStatement object.
7. Sends a cancel message through the ConnectionJDBC and cancels the query being executed.

 

Figure 5: QueryTimeout Execution Process for JTDS (MS SQLServer) Statement.

QueryTimeout for MySQL JDBC Statement (5.0.8)

 

1. Creates a statement by calling Connection.createStatement().
2. Calls Statement.executeQuery().
3. The statement transmits the Query to MySqlServer by using the internal connection.
4. The statement creates a new timeout-execution thread for timeout process.
5. For version 5.1.x, it changes to assign 1 thread for each connection.
6. Registers the timeout execution to the thread.
7. Timeout occurs.
8. The timeout-execution thread creates a connection that has the same configurations as the statement.
9. Transmits the cancel Query (KILL QUERY "connectionId“) by using the connection.

 

Figure 6: QueryTimeout Execution Process for MySQL JDBC Statement (5.0.8).

QueryTimeout for CUBRID JDBC Statement

 

1. Creates a statement by calling Connection.createStatement().
2. Calls Statement.executeQuery().
3. The statement transmits the Query to CUBRID DBMS by using the internal connection.
4. The statement creates a new timeout-execution thread for timeout process.
5. Registers the timeout execution to the thread.
6. Timeout occurs.
7. The timeout-execution thread creates a connection that has the same configurations as the statement.
8. Transmits the cancel message using the connection.

 

Figure 7: QueryTimeout Execution Process for CUBRID JDBC Statement.

 

What is Socket Timeout for JDBC Driver?

JDBC driver type 4 uses the socket to connect to the DBMS, and the connection timeout process between the application and the DBMS is not carried out by the DBMS.

Socket timeout value for JDBC driver is necessary when the DBMS is terminated abruptly or an network error has occured (equipment malfunction, etc.). Because of the structure of TCP/IP, there are no means for the socket to detect network errors. Therefore, the application cannot detect any disconnection with the DBMS. If the socket timeout is not configured, then the application may wait for the results from the DBMS indefinitely. (This connection is also called a "dead connection.") To prevent dead connections, a timeout must be configured for the socket. Socket timeout can be configured via JDBC driver. By setting up the socket timeout, you can prevent the infinite waiting situation when there is a network error and shorten the failure time.

It is not recommended to use the socket timeout value to limit the statement execution time. So the socket timeout value must be higher than the statement timeout value. If the socket timeout value is smaller than the statement timeout value, as the socket timeout will be executed first, and the statement timeout value becomes meaningless and will not be executed.

Socket timeout has 2 options listed below, and their configurations vary by driver.

 

• Timeout at socket connection: Time limit for Socket.connect(SocketAddress endpoint, int timeout)
• Timeout at socket reading/writing: Time limit for Socket.setSoTimeout(int timeout)

 

By checking the source for CUBRID, MySQL, MS SQL Server (JTDS) and Oracle JDBC, we confirmed that all the drivers we checked use the 2 APIs above.

How to configure SocketTimeout is as explained below.

JDBC Driver	connectTimeout	Default	Unit	Application Method
socketTimeout	Default	Unit
MySQL Driver	connectTimeout	0	ms	Specify the option in the DriverURL. Format: jdbc:mysql://[host:port],[host:port].../[database] [?propertyName1][=propertyValue1][&propertyName2][=propertyValue2]... Example: jdbc:mysql://xxx.xx.xxx.xxx:3306/database?connectTimeout=60000&socketTimeout=60000
socketTimeout	0	ms
MS-SQL Driver jTDS Driver	loginTimeout	0	sec	Specify the option in the DriverURL. Format: jdbc:jtds:<server_type>://<server>[:<port>][/<database>][;<property>=<value>[;...]] Example: jdbc:jtds:sqlserver://server:port/database;loginTimeout=60;socketTimeout=60
socketTimeout	0	sec
Oracle Thin Driver	oracle.net.CONNECT_TIMEOUT	0	ms	Not possible with the driverURL. Must be delivered to the properties object via OracleDatasource.setConnectionProperties() API. When DBCP is used, use the following APIs: BasicDatasource.setConnectionProperties() BasicDatasource.addConnectionProperties()
oracle.jdbc.ReadTimeout	0	ms
CUBRID Thin Driver	No separate configuration	5,000	ms	Not possible with the driverURL. Timeout occurs in 5 seconds. Note 1: When timeout occurs with althost option specified in the URL, it can be connected to the designated host. Note 2: C API can be used to state the login_time option in ms in the URL.

• Note 1: The default value for connectTimeout and socketTimeout is "0," which means that the timeout does not occur.
• Note 2: You can also configure through properties without directly using the separate API of DBCP. 

 

When you configure properties, pass on the character string where the key value is “connectionProperties”, and the format value is “[propertyName=property;]*”. The following example shows configuring properties through xml in iBatis.

 

<transactionManager type="JDBC">
  <dataSource type="com.nhncorp.lucy.db.DbcpDSFactory">
     ....
     <property name="connectionProperties" value="oracle.net.CONNECT_TIMEOUT=6000;oracle.jdbc.ReadTimeout=6000"/>  
  </dataSource>
</transactionManager>

OS Level SocketTimeout Configuration

If the socket timeout or the connect timeout is not configured, most of the time, applications cannot detect network errors. So, until the applications are connected or are able to read data, they will wait indefinitely. However, if you look at the actual issues NHN services encountered, the problems were often resolved after the applications (WAS) tried to reconnect to the network 30 minutes after. This is because the OS can also configure socket timeout time. Linux servers used by NHN have set the socket timeout to 30 minutes. This checks the network connection at the OS level. Because the KeepAlive checking cycle for NHN's Linux servers is 30 minutes, even when socket timeout is set to 0, the DBMS network connection problems caused by network issues do not surpass 30 minutes.

Generally, the application hangs from network issues when the application is calling Socket.read(). However, depending on the network composition or the error type, it can rarely be in waiting status while running Socket.write(). When the application calls Socket.write(), the data is recorded to the OS kernel buffer and then the right to control is returned to the application immediately. Thus, as long as a valid value is recorded to the kernel buffer, Socket.write() is always successful. However, if the OS kernel buffer is full due to a special network error, even Socket.write() can be put into waiting status. In this case, the OS tries to resend the packet for a certain amount of time, and generates an error when it reaches the limit. In NHN's Linux server environment, the timeout for this situation is set to 15 minutes.

I have explained the internal operations of JDBC so far. I hope that this will help you with the correct timeout configuration and reducing errors.

If you have more questions or any good information related to JDBC, please leave your comments below.

Lastly, I have listed some of the frequently asked questions below.

FAQ

Q1. I configured the query timeout by using Statement.setQueryTimeout(), but it does not work as expected when there is a network error.

 ➔ Query Timeout only works when it is connected to the socket correctly. Therefore, it cannot be used to solve an exceptional situation with a network error. To be prepared for network errors, socket timeout in JDBC driver must be configured.

 

Q2. How are transaction timeout, statement timeout and JDBC driver socket timeout related to the DBCP configuration values?

➔ When the connection is acquired from DBCP to JDBC, nothing but waitTimeout is affected.

 

Q3. If JDBC SocketTimeout is configured, wouldn't the connections that stayed in idle status for a long time in DBCP be closed?

➔ No. The socket option is applied when the actual data is being written or read, so it does not affect the connections in idle status in DBCP. The socket option can have certain effect when new connections that lack in inside of DBCP are created, old idle connections are removed, or the validation is checked, but this does not cause any significant issues unless the network has an error.

 

Q4. How long should SocketTimeout be set to?

➔As I have mentioned in the main article above, it must be much bigger than the statement timeout, and there is no recommended value. Socket timeout value for the JDBC driver becomes effective after a network error occurs. A careful configuration for the value cannot prevent such the errors from happening, but sometimes shortens the time that the network is disabled (if the network is restored right away).

 

 

By Woon Duk Kang, Software Engineer at Web Platform Development Lab, NHN Corporation.