Testing jgroups communication (multicasting)

Basically we're following this:

http://www.jgroups.org/manual/html/ch02.html#ItDoesntWork

The main tools are included in the "all" JBoss server lib dir-- jgroups.jar (ex:{jbossdir}/servers/all/bin/jgroups.jar).

CD into that directory, and try this stuff:

Make sure your machine is set up correctly for IP multicast. There are 2 test programs that can be used to detect this: McastReceiverTest and McastSenderTest. Start McastReceiverTest, e.g.

java -cp ./jgroups.jar org.jgroups.tests.McastReceiverTest -mcast_addr 224.10.10.10 -port 5555
        

Then start McastSenderTest:

java -cp ./jgroups.jar org.jgroups.tests.McastSenderTest -mcast_addr 224.10.10.10 -port 5555
        

If you want to bind to a specific network interface card (NIC), use -bind_addr 192.168.0.2, where 192.168.0.2 is the IP address of the NIC to which you want to bind. Use this parameter in both sender and receiver.

You should be able to type in the McastSenderTest window and see the output in the McastReceiverTest. If not, try to use -ttl 32 in the sender. If this still fails, consult a system administrator to help you setup IP multicast correctly. If you are the system administrator, look for another job :-)

Other means of getting help: there is a public forum on JIRA

for questions. Also consider subscribing to the javagroups-users mailing list to discuss such and other problems.

2.9. The instances still don't find each other !

In this case we have to use a sledgehammer (running only under JDK 1.4. and higher): we can enable the above sender and receiver test to use all available interfaces for sending and receiving. One of them will certainly be the right one... Start the receiver as follows:

java -cp ./jgroups.jar org.jgroups.tests.McastReceiverTest1_4 -mcast_addr 228.8.8.8 -use_all_interfaces
        

The multicast receiver uses the 1.4 functionality to list all available network interfaces and bind to all of them (including the loopback interface). This means that whichever interface a packet comes in on, we will receive it. Now start the sender:

java -cp ./jgroups.jar org.jgroups.tests.McastSenderTest1_4 -mcast_addr 228.8.8.8 -use_all_interfaces
        

The sender will also determine the available network interfaces and send each packet over all interfaces.

This test can be used to find out which network interface to bind to when previously no packets were received. E.g. when you see the following output in the receiver:

java  -cp ./jgroups.jarorg.jgroups.tests.McastReceiverTest1_4 -mcast_addr 228.8.8.8 -bind_addr 192.168.168.4
            Socket=0.0.0.0/0.0.0.0:5555, bind interface=/192.168.168.4
            dd [sender=192.168.168.4:5555]
            dd [sender=192.168.168.1:5555]
            dd [sender=192.168.168.2:5555]
        

you know that you can bind to any of the 192.168.168.{1,2,4} interfaces to receive your multicast packets. In this case you would need to modify your protocol spec to include bind_addr=192.168.168.2 in UDP, e.g. "UDP(mcast_addr=228.8.8.8;bind_addr=192.168.168.2):..."

Setting up multicasting can be a PITA if you don't control your network. Luckily, we do, so it's no big deal, but there are a lot of things to check if it's not working.

I'd start with verifying that multicast forwarding is enabled on your NIC, and in your configs (this varies by OS, but google is quite helpful). Then move onto the rest of your network (firewalls, routers, etc.).

Start with the firewall: On a CentOS 5 box, you'll need to tweak the iptables config (/etc/sysconfig/iptables) to allow UDP traffic on the right interfaces.

Here's an example (put it under the existing udp stuff in sysconfig/iptables, which BTW, should also give you your UDP mcast addr (...-p udp --dport 5555 -d 2xx.0.0.x)):

-A RH-Firewall-1-INPUT -p udp -i eth3 -m udp --dport 4000:6000 -j ACCEPT

The above says "open ports 4000:6000 on the eth3 interface", basically. For my purposes, eth3 is a NIC dedicated to back-end private network traffic only. Having a separate NIC for inter-server communication is really tits.

After you add that line, you'll need to reload iptables:

iptables-restore < /etc/sysconfig/iptables

Depending on how you decided to have your nodes talk to each other, you may need to configure some stuff in jgroups-channelfactory-stacks.xml (jgroupschannel-factory.sar) - see this. I used something along these lines:

<FD timeout="10000" max_tries="5" shun="true"></FD>

Configuring jgroups for jboss AS 5 is easy as pie, here's some info for params and whatnot:

Isolation and JGroups ChannelFactory and Shared Transport in JBoss AS 5 and Buddy Replication and Session Data, although I had better links a while ago. Eh, those should get you going.

Yesh, just a copy/paste entry, mostly, but what the hell.

Load Balancing your JBoss AS 5 cluster with Apache 2 and AJP

Most of this information comes from this page:

http://wiki.jboss.org/wiki/UsingMod_proxyWithJBoss

Make sure that at least following modules are loaded (uncomment this in httpd.conf)

LoadModule proxy_module modules/mod_proxy.so
    LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
    LoadModule proxy_ajp_module modules/mod_proxy_ajp.so

Add those lines (for the balancer) in APACHE_HOME/conf/httpd.conf (replace the node information with your node information, as many as are in this cluster:

<Proxy balancer://mycluster>
Order deny,allow
Allow from all
BalancerMember ajp://node1.mycluster.lan:8009/ route=node1 timeout=15
BalancerMember ajp://node2.mycluster.lan:8109/ route=node2 timeout=15
ProxySet stickysession=JSESSIONID
## to test session replication, comment out below
ProxySet lbmethod=bytraffic
</Proxy>

You'll need to add a proxypass, or a rewrite, if you prefer. I like a combination of both (you'll need mod_rewrite for this, but it allows apache to serve static content itself, and only bother jboss when it needs processing (CFML here; change .cfm to .jsp for jsp, etc.). Here is a virtual host configuration:

<VirtualHost *:80>
ServerName your.domain.name
DocumentRoot /path/to/static/content/
## log locations and whatnot omitted

# jack throughput up a bit in this case (caveats)
ProxyIOBufferSize 19000
LimitRequestFieldsize 18000

ProxyRequests Off

# our rewrite conditions
RewriteEngine on
RewriteCond %{REQUEST_fileNAME} !^\/myapplication
RewriteRule ^\/(.+)\.cfm(.+)? /myapplication/$1.cfm$2 [PT,QSA,L]
<Location /myapplication>
ProxyPass balancer://mycluster/myapplication/
ProxyPassReverseCookiePath /myapplication /
</Location>
</VirtualHost>

You may want to verify that JSESSIONID is being set as a cookie at this point-- if not, you may need to remove/change the ProxyPassReverseCookiePath attribute of the location directive (JBoss defaults to the application context name for the cookie path now).

Assuming you would like to load balance by server load or by server traffic, and/or want jboss to be able to do sticky sessions, and you followed my instructions for clustering JBoss 5, you won't need to add this to each of your jbossweb.sar/server.xml files (replacing the node1 with the node name for each nodes's server.xml file):

<Engine name="jboss.web" defaultHost="localhost" jvmRoute="node1">
</Engine>

I guess I should go into the sticky sessions bit, here: There are many ways to set up your load balancing. You can replicate your sessions across servers (JBoss has a nice way of doing this through a database, be sure you're not using the HSQL db in production, as the docs say), or use sticky sessions (repeat requests go to the same node), or a combination.

You can do round robin, which basically just sequentially hands requests to each node (request 1 to node 1, request 2 to node 2, ect.), or you can do it by weight, where you shunt more traffic to the beefier servers.

A lot depends on what you're after, and what you have to work with, etc.

Without session replicating, your clients will lose any session information if a node goes down (assuming you're using the bytraffic lbmethod). Meaning they're logged out of that nifty webapp you are trying to provide five nines uptime (or whatever) for.

There is overhead associated with keeping the session for each client stored, essentially, on all the nodes.

There are a lot of ways to mitigate that overhead tho, so you should be able to provide at least one or two failovers before losing anyone's session (see info on "buddies" and whatnot (basically "cell" replicator groups)).

I could go on for a long time, but it's late, and this is enough for now, I figure.

Peace.

Clustering JBoss AS 5, the easy way

There are a couple of simple steps to get JBoss clustering working in a way that is easy to deploy, and with minimal editing of files.

  1. Download JBoss AS, and extract it.
  2. Copy the "all" server (jboss/server/all) to "nodetemplate" (jboss/server/nodetemplate)
  3. Install Railo or JBoss into the "nodetemplate" server (you may want to use a central location for Railo or ColdFusion libraries, see the railo EAR example)
  4. Edit the server.xml file (jboss/server/nodetemplate/deploy/jbossweb.sar/server.xml) and add this to the "Engine" tag: jvmRoute="node${jboss.messaging.ServerPeerID}" (Ex: <Engine name="jboss.web" defaultHost="localhost" jvmRoute="node${jboss.messaging.ServerPeerID}"> )
  5. Change the JMS datasource database
  6. Edit jboss/server/nodetemplate/messaging/messaging-service.xml and messaging-jboss-beans.xml and change the "MessageSucker" password.
  7. Create a user named "jboss", if you don't have one already. Be sure it's a system account (not a real user account)
  8. Copy the "nodetemplate" directory to "node1" (Ex. cp -R nodetemplate node1)
  9. Give ownership of the jboss folder to the jboss user
  10. Create a startup script for your nodes in the jboss/bin directory, called "jbossnode".
  11. Put this in there, and edit the settings (ip addresss, and perhaps your multicast address (jgroups.bind_addr))
    jboss.messaging.ServerPeerID will be a unique node number, for use with load balancing later on, and to avoid node conflicts
    (this script isn't good for automatic startup without editing, eventually I'll get a new one posted)
    #!/bin/sh
    ## jboss home
    jboss_home="/path/to/jboss"
    ## node ips
    node_ip_1=10.0.1.8
    node_ip_2=10.0.2.1

    if [ $# -eq 0 ]; then
        echo "please specify node number and an action (start|stop) - Example: jbossnode 2 start"
        exit 0
    fi

    if [ $1 -eq 1 ]; then
    instance_number=1
    instance_host=$node_ip_1
    fi
    if [ $1 -eq 2 ]; then
    instance_number=2
    instance_host=$node_ip_2
    fi

    if [ $2 = "start" ]; then

    sudo -u jboss $jboss_home/bin/run.sh -c node$instance_number -b $instance_host \
    -Djboss.messaging.ServerPeerID=$instance_number -Djgroups.bind_addr=127.0.0.1 \
    -Djgroups.tcpping.initial_hosts=127.0.0.1[7800] -Djava.net.preferIPv4Stack=true &

    fi
    if [ $2 = "stop" ]; then
    sudo -u jboss $jboss_home/bin/shutdown.sh -s $instance_host -S
    fi

    echo "$2ing node $instance_number ($instance_host)"
  12. make it executable: ./chmod +x jbossnode
  13. start node 1 like this: ./jbossnode 1 start
  14. start node 2 like this: ./jbossnode 2 start
  15. stop node 1 like this: ./jbossnode 1 stop
  16. stop node 2 like this: ./jbossnode 2 stop

Create a CFM file containing the following:

<cfscript>
mySession = getPageContext().getRequest().getSession(true);
ival = mySession.getAttribute("simplesession.counter");
if (NOT isDefined("ival")){
    ival = 1;
} else {
    ival = ival + 1;
}
mySession.setAttribute("simplesession.counter", ival);
</cfscript>
<cfoutput>
CFSESSIONID:
<br /><strong>#session.URLToken#</strong>
<br />
JSESSIONID:
<cfdump var="#mySession.getID()#">
Number of refreshes:
<cfdump var="#ival#">
Cookies:
<cfdump var="#cookie#">
<cfif NOT structKeyExists(cookie,"JSESSIONID")>
NO JSESSIONID! Are you proxy/rewriting? Did you remember to change the cookie path (ProxyPassReverseCookiePath)?<br />
</cfif>
Session:
<cfdump var="#session#">
</cfoutput>

This should display something like:

V4VJ3xd4NRn82z6OkPncww__.node1 (emphasis added)

Plus the dumps. Refresh a couple times and verify your sessions are working. Play around with the balance-manager stuff if you want.

Oh, guess you should read the load balancing entry first!

Assuming you already did that, and set up sticky sessions and whatnot, you can bring various nodes up or down and hit that page to see what node you are being served from.

Some helpful links:

http://docs.hp.com/en/5991-5855/ar01s07.html

http://www.yorku.ca/dkha/jboss/docs/ClusterTesting.htm

http://www.redhat.com/docs/manuals/jboss/jboss-eap-4.3/doc/messaging/JBoss_Messaging_User_Guide/html-single/index.html

http://wiki.jboss.org/wiki/UsingMod_proxyWithJBoss

http://www.jboss.org/file-access/default/members/jbossas/freezone/docs/Clustering_Guide/beta500/html_single/index.html

More later

BlogCFC was created by Raymond Camden. This blog is running version 5.9.3.000. Contact Blog Owner