Dilaver DEMİREL
Servlet's Non-Blocking Adventure
Our expectation is always to be able to do more work faster with fewer resources. To achieve this, we examine what consumes system resources and where bottlenecks occur.
At its core, the process takes place between the client and the server. Both the server and the client have responsibilities for the speed of operations. If things aren't going well on the server, if it's waiting for another resource to complete the process, we lose speed, resource consumption increases, or if the client is running on a slow network, it cannot quickly receive the data from the server and complete the process.
For the servlet-based HTTP protocol, it's necessary to evaluate client-server communication in two stages.
- Communication between client and server
- Communication between server and servlet
Up to Tomcat 6 (servlet 2.5), the default HTTP connector operates synchronously. This means one thread is dedicated per request. 100 concurrent users require 100 active threads. In this case, an HTTP connector with 100 threads will be unable to handle the 101st request and will hold it. This model is called threads per connection.
Tomcat 6's new HTTP connector, Http11NioProtocol, has made the way connections are handled more efficient and different by using the NIO (non-blocking IO) principle. The new connector has a two-level thread pool. The first level contains threads that accept and maintain connections. The purpose of these threads is to receive connection requests and pass them on to the second-level worker threads. When a thread between a connection and a request becomes idle (i.e., when request processing has begun), it is returned to the first-level pool to accept new requests. This model, called "thread per request," allows for managing high user connections with a fixed thread pool. By sharing threads, the model enables concurrent handling of more user requests.
The first-level threads are called "Acceptor Threads" and their size is determined by default based on CPU cores.
The second-level threads are called "Request Processing Threads" and handle and process HTTP requests.
To understand how this model works in more detail, the Java NIO API can be examined.
Here, the NIO structure only makes the communication between the client and the server non-blocking. However, the communication between the server and the servlet still works in blocking mode.
After the request is passed to the 2nd level worker threads, the process naturally reverts to blocking mode because read/write operations are performed on the "HttpInputStream" and "HttpOutputStream". Or for operations such as waiting for database queries.
Looking at the stages in this process again:
The client opens a connection to the server.
The server receives the connection with the 1st level acceptor threads.
To process the request, it passes the process to the request processor thread managed by the servlet container; this thread is naturally blocked.
In this case, if the request processor thread pool size is 50, the application can only process and fulfill 50 requests concurrently. As seen, this situation creates another bottleneck.
This problem has been solved with the Async feature introduced in Servlet 3.0.
The Async feature basically works like this: due to bottlenecks caused by blocking of request processor threads, the request processor thread pool could only process a certain number of operations concurrently. The Async feature, however, allows processor threads to transfer their work to other threads besides the servlet container running in the background, thus freeing the processor thread from being blocked.
However, this solution creates a problem: the number of concurrently running threads in the system doesn't decrease. Although the application gains high responsiveness, it significantly consumes the system's capacity.
Furthermore, another problem that causes thread blocking is this: after the request processor thread completes the process in the background, it writes the response to the output stream to send it to the client. The client must receive this data and release the thread. If the client is running on a slow network, the data retrieval time will be longer. This also causes the thread to be blocked.
@WebServlet(urlPatterns = {
"/asyncservlet"
}, asyncSupported = true)
public class AsyncServlet extends HttpServlet {
/* ... Same variables and init method as in SyncServlet ... */
@Override
public void doGet(HttpServletRequest request,
HttpServletResponse response) {
response.setContentType("text/html;charset=UTF-8");
final AsyncContext acontext = request.startAsync();
acontext.start(new Runnable() {
public void run() {
String param = acontext.getRequest().getParameter("param");
String result = resource.process(param);
HttpServletResponse response = acontext.getResponse();
/* ... print to the response ... */
acontext.complete();
}
}
Up to this point, we can increase the capacity to handle concurrent requests in our applications with two different features:
By increasing the thread pool capacity of the Tomcat NIO Connector; this will provide a significant benefit even if our application is not asynchronous.
By using the asynchronous feature with Servlet 3.0.
The blocking problem caused by InputStream and OutputStream has been solved with the ReadListener and WriteListener introduced in Servlet 3.1. Listeners are set to ServletInput/OutputStream objects. When the stream is available to write or read, the write/read operation is performed through the listeners. In this way, the thread can be assigned to other tasks without waiting in idle state depending on the availability of the write/read operation. And thus, the blocking problem at this stage is eliminated.
@WebServlet(urlPatterns={"/asyncioservlet"}, asyncSupported=true)
public class AsyncIOServlet extends HttpServlet {
@Override
public void doPost(HttpServletRequest request,
HttpServletResponse response)
throws IOException {
final AsyncContext acontext = request.startAsync();
final ServletInputStream input = request.getInputStream();
input.setReadListener(new ReadListener() {
byte buffer[] = new byte[4*1024];
StringBuilder sbuilder = new StringBuilder();
@Override
public void onDataAvailable() {
try {
do {
int length = input.read(buffer);
sbuilder.append(new String(buffer, 0, length));
} while(input.isReady());
} catch (IOException ex) { ... }
}
@Override
public void onAllDataRead() {
try {
acontext.getResponse().getWriter()
.write("...the response...");
} catch (IOException ex) { ... }
acontext.complete();
}
@Override
public void onError(Throwable t) { ... }
});
}
}
Up to this point, we have seen the evolution of Servlets. With NIO support, we can develop applications that can do more work with fewer threads.
Reactive architectures have emerged, offering a different approach to solving these problems. In Java, reactive architectures have their own request accepting and processing mechanisms, separate from servlet containers.