Hung JBoss/Tomcat instances and "ignored" shutdown requests

I kept hearing about how people were using "kill" in Tomcat shutdown scripts, as they'd shutdown an instance but it would still be running. Further attempts to shutdown after the initial one would say something to the effect of "already shut down" or some-such.

I hadn't been experiencing this, and said so publicly-- at which point I promptly started to experience this locally when running certain unit tests.

It sorta bugged me, and having a semi-reproducible set up, I figured what the hell, might as well troubleshoot.

For Schlitz and giggles, I cycled httpd after causing a "hung" process, and what do you know, JBoss finished shutting down.

I think this is something in the way folks have AJP set up. I'm playing with some settings, namely the "connectionTimeout" attribute in the Tomcat AJP connector config, and the "ping" attribute for mod_proxy.

I'll update this when I have a chance to see if one of them worked-- I'm troubleshooting while doing real work, so I don't shut down often. =]


Well, so far, after adding the connectionTimeout attribute to the AJP connector in deploy/jbossweb.sar/server.xml, I haven't had the shutdown problem.

I'm unconvinced so far, but so far, so good.

Here are some mod_proxy docs:

microsoft-sqljdbc.jar deployment errors on JBoss

While deploying your application on jboss you may run into a zip error type deal, something like this:

Caused by: java.lang.Error: Error visiting DelegatingHandler@1463636387[path=cf.ear/railo.war/WEB-INF/lib/microsoft-sqljdbc.jar context=file:/home/coldshen/deploy2/ real=file:/home/coldshen/deploy2/cf.ear/railo.war/WEB-INF/lib/microsoft-sqljdbc.jar]

Which means the VFS deployer just choked on the (IMO) malformed microsfot jdbc driver jar [Ed Note: yes, I left the spelling error in there, cuz I'm cool like that].

See this thread for long, technical discussion:

And the solution is to edit your vfs.xml file (given the context location above, the path is /home/coldshen/deploy2, you'll have to edit yours) and add this entry to the VFSCache bean:

<value><inject bean="VfsNamesExceptionHandler"/></value>

Which basically tells JBoss that there's a fscked-up jar in, or under, that directory.

The vfs.xml file should be located at: {jboss_home}/server/{your_server}/conf/bootstrap/vfx.xml

deploying your EAR or WAR on JBoss 5 from another location on disk

Sometimes it is a pain in the ass to keep your crap in the deploy directory. There used to be docBase and whatnot, but that nifty stuff is ignored mostly by JBoss 5, since it's got it's own nifty way of doing things.

So, if you'd like to easily keep your code separate from your appserver (mostly a developer's want-- on production, most this crap doesn't matter), you can do this:

Edit {jboss_home}/server/{your_server}/conf/bootstrap/profile-repository.xml to have something like this:

<bean name="SerializableDeploymentRepositoryFactory" class="org.jboss.system.server.profileservice.repository.SerializableDeploymentRepositoryFactory">
<property name="storeRoot">${jboss.server.base.dir}</property>
<property name="attachmentsRoot">${}/attachments</property>
<property name="applicationURIs">
<array elementClass="">
<property name="serializer"><inject bean="AttachmentsSerializer"/></property>
<property name="deploymentFilter"><inject bean="DeploymentFilter" /></property>
<property name="mainDeployer"><inject bean="MainDeployer"/></property>
<property name="checker"><inject bean="StructureModificationChecker"/></property>

See that line? : file:///path/to/your/deploy/dir ?

Yup, that's the magic line.

Your deploy directory should just contain the WARs or EARs, and they should be named like "yourapp.war" or "yourapps.ear". They can be exploded or not. Heh. "Exploded". I like that. Anyways...

That's it. JBoss5 should now deploy "stuff" from wherever you specified.

Speeding up JBoss 5 applicaiton deployment

I've mentioned before the trick of turning off the AOP scanner to quicken deployment, but I've found another way, that I like a lot better.

The delays in starting up are worst for CF apps, because they've got so many classes, and so you may have noticed that the more classes you've saved, the longer it takes.

For a while I had a script that cleared out the cfclasses folder before each start, but that was soooo hackish. More of a kludge, actually. Fine for development tho.

I've found that if you use a jboss-structure.xml file in your META-INF (I put it in my ear/META-INF), jboss will use it to determine what and where to deploy stuff, along with class paths and whatnot. This disables the recursive scanning jboss normally does, and makes startup times acceptable again.

Example 1 (ear file with libs in each war):


        <path name="railo.war" />
            <path name="WEB-INF" />
            <path name="/railo.war/WEB-INF/lib" suffixes=".jar" />


Example 2 (ear file with consolidated libs folder):


        <path name="widgets.war" />
            <path name="WEB-INF" />
            <path name="/lib" suffixes=".jar" />

        <path name="sprockets.war" />
            <path name="WEB-INF" />
            <path name="/lib" suffixes=".jar" />


Testing jgroups communication (multicasting)

Basically we're following this:

The main tools are included in the "all" JBoss server lib dir-- jgroups.jar (ex:{jbossdir}/servers/all/bin/jgroups.jar).

CD into that directory, and try this stuff:

Make sure your machine is set up correctly for IP multicast. There are 2 test programs that can be used to detect this: McastReceiverTest and McastSenderTest. Start McastReceiverTest, e.g.

java -cp ./jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr -port 5555

Then start McastSenderTest:

java -cp ./jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr -port 5555

If you want to bind to a specific network interface card (NIC), use -bind_addr, where is the IP address of the NIC to which you want to bind. Use this parameter in both sender and receiver.

You should be able to type in the McastSenderTest window and see the output in the McastReceiverTest. If not, try to use -ttl 32 in the sender. If this still fails, consult a system administrator to help you setup IP multicast correctly. If you are the system administrator, look for another job :-)

Other means of getting help: there is a public forum on JIRA

for questions. Also consider subscribing to the javagroups-users mailing list to discuss such and other problems.

2.9. The instances still don't find each other !

In this case we have to use a sledgehammer (running only under JDK 1.4. and higher): we can enable the above sender and receiver test to use all available interfaces for sending and receiving. One of them will certainly be the right one... Start the receiver as follows:

java -cp ./jgroups.jar org.jgroups.tests.McastReceiverTest1_4 -mcast_addr -use_all_interfaces

The multicast receiver uses the 1.4 functionality to list all available network interfaces and bind to all of them (including the loopback interface). This means that whichever interface a packet comes in on, we will receive it. Now start the sender:

java -cp ./jgroups.jar org.jgroups.tests.McastSenderTest1_4 -mcast_addr -use_all_interfaces

The sender will also determine the available network interfaces and send each packet over all interfaces.

This test can be used to find out which network interface to bind to when previously no packets were received. E.g. when you see the following output in the receiver:

java  -cp ./jgroups.jarorg.jgroups.tests.McastReceiverTest1_4 -mcast_addr -bind_addr
            Socket=, bind interface=/
            dd [sender=]
            dd [sender=]
            dd [sender=]

you know that you can bind to any of the 192.168.168.{1,2,4} interfaces to receive your multicast packets. In this case you would need to modify your protocol spec to include bind_addr= in UDP, e.g. "UDP(mcast_addr=;bind_addr="

Setting up multicasting can be a PITA if you don't control your network. Luckily, we do, so it's no big deal, but there are a lot of things to check if it's not working.

I'd start with verifying that multicast forwarding is enabled on your NIC, and in your configs (this varies by OS, but google is quite helpful). Then move onto the rest of your network (firewalls, routers, etc.).

Start with the firewall: On a CentOS 5 box, you'll need to tweak the iptables config (/etc/sysconfig/iptables) to allow UDP traffic on the right interfaces.

Here's an example (put it under the existing udp stuff in sysconfig/iptables, which BTW, should also give you your UDP mcast addr (...-p udp --dport 5555 -d 2xx.0.0.x)):

-A RH-Firewall-1-INPUT -p udp -i eth3 -m udp --dport 4000:6000 -j ACCEPT

The above says "open ports 4000:6000 on the eth3 interface", basically. For my purposes, eth3 is a NIC dedicated to back-end private network traffic only. Having a separate NIC for inter-server communication is really tits.

After you add that line, you'll need to reload iptables:

iptables-restore < /etc/sysconfig/iptables

Depending on how you decided to have your nodes talk to each other, you may need to configure some stuff in jgroups-channelfactory-stacks.xml (jgroupschannel-factory.sar) - see this. I used something along these lines:

<FD timeout="10000" max_tries="5" shun="true"></FD>

Configuring jgroups for jboss AS 5 is easy as pie, here's some info for params and whatnot:

Isolation and JGroups ChannelFactory and Shared Transport in JBoss AS 5 and Buddy Replication and Session Data, although I had better links a while ago. Eh, those should get you going.

Yesh, just a copy/paste entry, mostly, but what the hell.

Configuring JBoss AS 5 JMS for MySQL

To round-out the JBoss AS 5 clustering stuff, I figured I'd add a little how-to for configuring JMS to use MySQL.

This is covered in the docs, but what the hell (the example DS config files leaves out transaction stuffs, which produces an warning like this: DataSource connection transaction isolation should be READ_COMMITTED, but it is currently REPEATABLE_READ.).

{jbossdir} = your jboss dir {serverdir} = your server dir {jbossdir}/server/{serverdir}

Firstly, create a mysql datasource file:{serverdir}/deploy/mysql-ds.xml

Fill it with something along these lines:


Delete the {serverdir}/deploy/hsqldb-ds.xml file.

Then delete the {serverdir}/deploy/messaging/hsqldb-persistence-service.xml file.

Copy {jbossdir}/docs/examples/jms/mysql-persistence-service.xml file to {serverdir}/deploy/messaging/mysql-persistence-service.xml.

If you're using clustering, be sure to set Clustered to true in mysql-persistence-service.xml.

Um, guess that's it.

Errors when upgrading to JBoss 5 AS

There seem to be a few changes, even from 5.0.0.CR1 and CR2 to 5.0.1.GA.

First off, I was able to have copies of xerces in my WEB-INF/lib folders before I upgraded (didn't need them there, JBoss has it's own), and they throw an error now. I assume some sorta classloading change took place, probably for the better, so right on. Removing 'em solves that.

Secondly, I had to be sure my EAR META-INF/application.xml files were using the 5 standard, while they were are 1.2 before. Here's an example of an application.xml for 5:

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns=""

<description>jasperserver 3.0</description>
<module id="WebModule_1">

Updating to that standard was pretty easy, you can be sure it validates by using an XML editor that rocks, but I just used logic (what's the same, what's different).

That's about all I've come across so far.

Installing JasperServer 3 on JBoss AS 5

JasperServer is a kick-ass reporting application.

Running it under JBoss 5 isn't too hard, but you've got to tweak some stuff to do so (or at least I did).

First off, I like to use an ear, so create the jasperserver.ear folder in your chosen deploy folder (ex. jboss/server/default/deploy/jasperserver.ear)

Next create the META-INF folder (ex. jboss/server/default/deploy/jasperserver.ear/META-INF)

Add this as application.xml (ex. jboss/server/default/deploy/jasperserver.ear/META-INF/application.xml):

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns=""

<display-name>jasperserver 3</display-name>
<module id="WebModule_1">

Now move the jasperserver.war file into your ear directory, and rename it to, and unzip it. This should leave you with a folder like so (ex. jboss/server/default/deploy/jasperserver.ear/jasperserver.war)

Delete the file.

Edit the jasperserver.war/WEB-INF/web.xml file, and remove the datasources that you don't need (unless you want to use the sugarCRM stuff or example foodmart stuff)

Remove these:

<description>SugarCRM example database</description>

<description>Foodmart database</description>

delete the jasperserver.war/WEB-INF/application.xml (we're using the one in the ear, and the original throws some weird namespace error for me).

I had to delete xercesImpl-2.6.2.jar from the WEB-INF/lib folder for the GA version of JBoss 5 (didn't get in the way for the RCs, but there you go-- classloading changes, I assume).

Copy the js-mysql-ds.xml file from the {}/scripts into your deploy folder (ex. jboss/server/default/deploy/js-mysql-ds.xml)

Remove the foodmart and sugarCRM datasources from that file too.

Copy the mysql-connector.jar (if you don't already have one) from {}/scripts/drivers to your server lib directory (ex. jboss/server/default/lib).

Create or update your jasperserver database (scripts are in the {}/scripts directory).

Edit the jboss/server/default/deploy/js-mysql-ds.xml file and set the correct username and password.

Start your jboss server.

Load Balancing your JBoss AS 5 cluster with Apache 2 and AJP

Most of this information comes from this page:

Make sure that at least following modules are loaded (uncomment this in httpd.conf)

LoadModule proxy_module modules/
    LoadModule proxy_balancer_module modules/
    LoadModule proxy_ajp_module modules/

Add those lines (for the balancer) in APACHE_HOME/conf/httpd.conf (replace the node information with your node information, as many as are in this cluster:

<Proxy balancer://mycluster>
Order deny,allow
Allow from all
BalancerMember ajp://node1.mycluster.lan:8009/ route=node1 timeout=15
BalancerMember ajp://node2.mycluster.lan:8109/ route=node2 timeout=15
ProxySet stickysession=JSESSIONID
## to test session replication, comment out below
ProxySet lbmethod=bytraffic

You'll need to add a proxypass, or a rewrite, if you prefer. I like a combination of both (you'll need mod_rewrite for this, but it allows apache to serve static content itself, and only bother jboss when it needs processing (CFML here; change .cfm to .jsp for jsp, etc.). Here is a virtual host configuration:

<VirtualHost *:80>
DocumentRoot /path/to/static/content/
## log locations and whatnot omitted

# jack throughput up a bit in this case (caveats)
ProxyIOBufferSize 19000
LimitRequestFieldsize 18000

ProxyRequests Off

# our rewrite conditions
RewriteEngine on
RewriteCond %{REQUEST_fileNAME} !^\/myapplication
RewriteRule ^\/(.+)\.cfm(.+)? /myapplication/$1.cfm$2 [PT,QSA,L]
<Location /myapplication>
ProxyPass balancer://mycluster/myapplication/
ProxyPassReverseCookiePath /myapplication /

You may want to verify that JSESSIONID is being set as a cookie at this point-- if not, you may need to remove/change the ProxyPassReverseCookiePath attribute of the location directive (JBoss defaults to the application context name for the cookie path now).

Assuming you would like to load balance by server load or by server traffic, and/or want jboss to be able to do sticky sessions, and you followed my instructions for clustering JBoss 5, you won't need to add this to each of your jbossweb.sar/server.xml files (replacing the node1 with the node name for each nodes's server.xml file):

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1">

I guess I should go into the sticky sessions bit, here: There are many ways to set up your load balancing. You can replicate your sessions across servers (JBoss has a nice way of doing this through a database, be sure you're not using the HSQL db in production, as the docs say), or use sticky sessions (repeat requests go to the same node), or a combination.

You can do round robin, which basically just sequentially hands requests to each node (request 1 to node 1, request 2 to node 2, ect.), or you can do it by weight, where you shunt more traffic to the beefier servers.

A lot depends on what you're after, and what you have to work with, etc.

Without session replicating, your clients will lose any session information if a node goes down (assuming you're using the bytraffic lbmethod). Meaning they're logged out of that nifty webapp you are trying to provide five nines uptime (or whatever) for.

There is overhead associated with keeping the session for each client stored, essentially, on all the nodes.

There are a lot of ways to mitigate that overhead tho, so you should be able to provide at least one or two failovers before losing anyone's session (see info on "buddies" and whatnot (basically "cell" replicator groups)).

I could go on for a long time, but it's late, and this is enough for now, I figure.


Clustering JBoss AS 5, the easy way

There are a couple of simple steps to get JBoss clustering working in a way that is easy to deploy, and with minimal editing of files.

  1. Download JBoss AS, and extract it.
  2. Copy the "all" server (jboss/server/all) to "nodetemplate" (jboss/server/nodetemplate)
  3. Install Railo or JBoss into the "nodetemplate" server (you may want to use a central location for Railo or ColdFusion libraries, see the railo EAR example)
  4. Edit the server.xml file (jboss/server/nodetemplate/deploy/jbossweb.sar/server.xml) and add this to the "Engine" tag: jvmRoute="node${jboss.messaging.ServerPeerID}" (Ex: <Engine name="jboss.web" defaultHost="localhost" jvmRoute="node${jboss.messaging.ServerPeerID}"> )
  5. Change the JMS datasource database
  6. Edit jboss/server/nodetemplate/messaging/messaging-service.xml and messaging-jboss-beans.xml and change the "MessageSucker" password.
  7. Create a user named "jboss", if you don't have one already. Be sure it's a system account (not a real user account)
  8. Copy the "nodetemplate" directory to "node1" (Ex. cp -R nodetemplate node1)
  9. Give ownership of the jboss folder to the jboss user
  10. Create a startup script for your nodes in the jboss/bin directory, called "jbossnode".
  11. Put this in there, and edit the settings (ip addresss, and perhaps your multicast address (jgroups.bind_addr))
    jboss.messaging.ServerPeerID will be a unique node number, for use with load balancing later on, and to avoid node conflicts
    (this script isn't good for automatic startup without editing, eventually I'll get a new one posted)
    ## jboss home
    ## node ips

    if [ $# -eq 0 ]; then
        echo "please specify node number and an action (start|stop) - Example: jbossnode 2 start"
        exit 0

    if [ $1 -eq 1 ]; then
    if [ $1 -eq 2 ]; then

    if [ $2 = "start" ]; then

    sudo -u jboss $jboss_home/bin/ -c node$instance_number -b $instance_host \
    -Djboss.messaging.ServerPeerID=$instance_number -Djgroups.bind_addr= \
    -Djgroups.tcpping.initial_hosts=[7800] &

    if [ $2 = "stop" ]; then
    sudo -u jboss $jboss_home/bin/ -s $instance_host -S

    echo "$2ing node $instance_number ($instance_host)"
  12. make it executable: ./chmod +x jbossnode
  13. start node 1 like this: ./jbossnode 1 start
  14. start node 2 like this: ./jbossnode 2 start
  15. stop node 1 like this: ./jbossnode 1 stop
  16. stop node 2 like this: ./jbossnode 2 stop

Create a CFM file containing the following:

mySession = getPageContext().getRequest().getSession(true);
ival = mySession.getAttribute("simplesession.counter");
if (NOT isDefined("ival")){
    ival = 1;
} else {
    ival = ival + 1;
mySession.setAttribute("simplesession.counter", ival);
<br /><strong>#session.URLToken#</strong>
<br />
<cfdump var="#mySession.getID()#">
Number of refreshes:
<cfdump var="#ival#">
<cfdump var="#cookie#">
<cfif NOT structKeyExists(cookie,"JSESSIONID")>
NO JSESSIONID! Are you proxy/rewriting? Did you remember to change the cookie path (ProxyPassReverseCookiePath)?<br />
<cfdump var="#session#">

This should display something like:

V4VJ3xd4NRn82z6OkPncww__.node1 (emphasis added)

Plus the dumps. Refresh a couple times and verify your sessions are working. Play around with the balance-manager stuff if you want.

Oh, guess you should read the load balancing entry first!

Assuming you already did that, and set up sticky sessions and whatnot, you can bring various nodes up or down and hit that page to see what node you are being served from.

Some helpful links:

More later

Getting around non SES wildcard problem with CF on Tomcat/JBoss

I'm messing around with Railo and when I tried this blog I hit a big fat error.

Seems that without the .cfm/* wildcard mapping, the index.cfm/somedate/somepost type stuff won't work.

To get around it, I'd heard someone say something about URL rewriting.

I manually tried just adding a ? after the index.cfm, and viola, things seem to work, so then I added this rewrite rule to my apache virtual host configuration:

RewriteRule ^\/(.+)\.cfm\/(.+)? ajp://$1.cfm\?$2 [P,L]

pretty straightforward; I just replace the .cfm/somedate with .cfm?/somedate (which I had tested earlier, neh?).

Seems like everything is running fine now. So there you go, future people who may or may not read this, but probably at least my future self.

Deploying Railo as an EAR under JBoss 5

If you've done it with CF, all you need to do is copy a couple of files and make a couple of edits, but basically it goes like this:

Download the railo.war archive.

Create a sub-directory under the jboss server's deploy directory called railo.ear.

In the railo.ear directory, create a sub-directory called META-INF

Create a file called application.xml within the META-INF directory. Here is an example application.xml:

<?xml version="1.0" encoding="UTF-8"?>
<application xmlns=""

<module id="WebModule_1">

Move the railo.war file to the railo.ear directory, and rename it to, and unzip it there.

You should now have a sub-directory called railo.war in the railo.ear directory.


Fix the nested zip errors [link to fix] by adding a file in WEB-INF/jboss-classloading.xml that contains:

<?xml version="1.0" encoding="UTF-8"?>
<classloading xmlns="urn:jboss:classloading:1.0"

Start up jboss, browser to your /railo context (If you've got a default jboss install, it will be: http://localhost:8080/railo), and bada-bing, you should be singing right along.

Running Railo 3 on JBoss 5

Edit: Better way of fixing the classloading errors (credit to for the solve, and other good stuff):

The correct way to fix this issue is to make JBoss isloate classes loaded for different applications, thereby removing the possibiiity of conflicts. On JBoss 5 you can do this if you add a file in WEB-INF/jboss-classloading.xml that contains:

<?xml version="1.0" encoding="UTF-8"?>
<classloading xmlns="urn:jboss:classloading:1.0"


FWIW, I like to keep everything in an EAR, and use an application.xml file to define the context, etc..

Eventually I'll post an entry on creating an EAR deployment with CF/Railo, as it's pretty easy.

Running CF8 on JBoss 5

It's unsupported, but I was able to run CF8 under JBoss 5 by disabling the JSafe JCE stuff:

You can do this by setting the "coldfusion.disablejsafe" System property to "true" by adding the following to ColdFusion's JVM arguments: "-Dcoldfusion.disablejsafe=true"

I'm running the 64 bit version under OS X 10.5, and it hauls, man! JBoss 5 starts a good bit quicker too. I'm sooooo happy.

Running JasperServer on JBoss 4.0.5GA

Since I once again got JasperServer going on JBoss, I figured I'd document how.

Follow the install instructions for the most part, but you will need to A: create a mysql-ds.xml (for example) something like this (put it in the deploy/ directory):

<?xml version="1.0" encoding="UTF-8"?>

<!-- ===================================================================== -->
<!-- -->
<!-- JBoss Server DataBase Configuration -->
<!-- -->
<!-- ===================================================================== -->




And you'll need to modify the WEB-INF/applicationContext.xml file-- replace the datasource bean def with this:

<bean id="dataSource" class="org.springframework.jndi.JndiObjectFactoryBean">
        <property name="jndiName" value="java:${metadata.hibernate.dataSource.jndiName}"/>

Then, create a WEB-INF/jboss-web.xml file:


And you're in (probably not quite like flynn). So- pretty much, we unzipped the .war into the deploy directory, and made the above config changes, and, fingers croxxed, it starts up!

More Entries

BlogCFC was created by Raymond Camden. This blog is running version Contact Blog Owner