Resin's TCP-ring protocol for distributed sessions can improve reliability and scalability over JDBC-based distributed sessions. Because sessions are always duplicated on separate servers, TCP-ring sessions do not have a single point of failure. JDBC-based sessions are vulnerable to the session database going down. As the number of servers increases, JDBC-based sessions can start overloading the backing database. With TCP-ring sessions, each additional server shares the backup load, so the main scalability issue reduces to network bandwidth. In combination with sticky-session load balancing, TCP-ring sessions can approach zero overhead when the majority of requests read session state without modification.

Configuration

TCP-Ring configuration must tell each host the servers in the TCP-ring and it must enable the tcp-store in the session configuration. Because session configuration is specific to a virtual host and a web-application, each web-app needs tcp-store enabled individually. It's not possible to enable tcp-store for an entire site.

Most sites using Resin's load balancing will already have the <srun> configured. Each <srun> block corresponds to a host, including the current host. Since TCP-ring sessions uses Resin's srun protocol, each host must listen for srun requests.

...
<http-server>
  <srun id='a' host='192.168.0.1' srun-index='1'/>
  <srun id='b' host='192.168.0.2' srun-index='2'/>
  <srun id='c' host='192.168.0.3' srun-index='3'/>
  <srun id='d' host='192.168.0.4' srun-index='4'/>
  ...
  <web-app id='myapp'>
    ...
    <session-config>
      <tcp-store/>
    </session-config>
  </web-app>
</http-server>
...

Usually, hosts will share the same resin.conf. Each host will be started with a different -server xx to select the correct block. On Unix, startup will look like:

resin-2.0.x> bin/httpd.sh -conf conf/resin.conf -server c start

On Windows, Resin will generally be configured as a service:

resin-2.0.x> bin/httpd -conf conf/resin.conf -server c -install-as ResinC

always-save-session

Resin's distributed sessions needs to know when a session has changed in order to save the new session value. Although Resin can detect when an application calls HttpSession.setAttribute, it can't tell if an internal session value has changed. The following Counter class shows the issue:

package test;

public class Counter implements java.io.Serializable {
  private int count;

  public int nextCount() { return count++; }
}

Assuming a copy of the Counter is saved as a session attribute, Resin doesn't know if the application has called nextCount. If it can't detect a change, Resin will not backup the new session, unless always-save-session is set. When always-save-session is true, Resin will back up the session on every request.

... <web-app id='/foo'> ... <session-config> <tcp-store/> <always-save-session/> </session-config> ... </web-app>

In contrast to JDBC-based sessions, Resin will ignore the always-load-session flag for TCP-based sessions. Because of the more sophisticate TCP-ring protocol, always-load-session is not needed.

Serialization

Resin's distributed sessions relies on Java serialization to save and restore sessions. Application object must implement java.io.Serializable for distributed sessions to work.

Protocol Examples

Session Request

To see how TCP-ring sessions work, consider a case where the load balancer sends the request to a random host. Host C owns the session but the load balancer gives the request to Host A. In the following figure, the request modifies the session so it must be saved as well as loaded.

The session id encodes the owning host. The example session id, ca8MbyA, decodes to an srun-index of 3, mapping to Host C. Host A must know the owning host for every cookie so it can communicate with the owning srun. The example configuration defines all the sruns Host A needs to know about. If Host C is unavailable, Host A can use its configuration knowledge to use Host D as a backup for ca8MbyA instead..

When the request first accesses the session, Host A asks Host C for the serialized session data (2:load). Since Host A doesn't cache the session data, it must ask Host C for an update on each request. For requests that only read the session, this TCP load is the only extra overhead, i.e. they can skip 3-5. The always-save-session flag, in contrast, will always force a write.

At the end of the request, Host A writes any session updates to Host C (3:store). If always-save-session is false and the session doesn't change, this step can be skipped. Host A sends the new serialized session contents to Host C. Host C saves the session on its local disk (4:save) and saves a backup to Host D (5:backup).

Sticky Session Request

Smart load balancers that implement sticky sessions can improve TCP-ring performance. In the previous request, Resin's TCP-ring sessions maintain consistency for dumb load balancers or twisted clients like the AOL browsers. The cost is the additional network traffic for 2:load and 3:store. Smart load-balancers can avoid the network traffic of 2 and 3.

Host C decodes the session id, caaMbyA. Since it owns the session, Host C gives the session to the servlet with no work and no network traffic. For a read-only request, there's zero overhead for TCP-ring sessions. So even a semi-intelligent load balancer will gain a performance advantage. Normal browsers will have zero overhead, and bogus AOL browsers will have the non-sticky session overhead.

A session write saves the new serialized session to disk (2:save) and to Host D (3:backup). always-save-session will determine if Resin can take advantage of read-only sessions or must save the session on each request.

Failover

Since the session always has a current copy on two servers, the load balancer can direct requests to the next server in the ring. The backup server is always ready to take control. The failover will succeed even for dumb load balancers, as in the non-sticky-session case, because the srun hosts will use the backup as the new owning server.

In the example, either Host C or Host D can stop and the sessions will use the backup. Of course, the failover will work for scheduled downtime as well as server crashes. A site could upgrade one server at a time with no observable downtime.

Recovery

When Host C restarts, possibly with an upgraded version of Resin, it needs to use the most up-to-date version of the session; its file-saved session will probably be obsolete. When a "new" session arrives, Host C loads the saved session from both the file and from Host D. It will use the newest session as the current value. Once it's loaded the "new" session, it will remain consistent as if the server had never stopped.

No Distributed Locking

Resin's TCP-ring sessions does not lock sessions. For browser-based sessions, only one request will execute at a time. Since browser sessions have no concurrently, there's no need for distributed locking. However, it's a good idea to be aware of the lack of distributed locking.

Conclusion

Although reliability generally will end up costing some performance, the trick for a good implementation is to increase reliability with a minimal cost. In some environments, using JDBC distributed sessions or simple file-based persistent sessions will improve a site's robustness with low-enough cost. Because of its scalability, it's more likely that Resin's TCP-ring distributed sessions will be a better choice for most deployment configurations.

Copyright © 1998-2002 Caucho Technology, Inc. All rights reserved. Resin® is a registered trademark, and HardCoretm and Quercustm are trademarks of Caucho Technology, Inc.