Using GlassFish 3.1.2.2 with 2 servers, 2 nodes, 2 instances and deployed simple MDB to cluster (Conventional cluster with master broker). I can access on server1 and server2 via port 23700. Shutdown server1 and try to access from server2 and get NAM1007 : Problem with membership change notification. Exception occurred : java.lang.NullPointerException. If I look at ports on server2 23700 is not listening. I had to deploy an EAR with an MDB to get server1 listening on the ORB ports.
It appears that shutting down the master broker gracefully also shuts down the ORB listeners of the other instances. What happens if you want to do rolling maintenance?
Second scenario, I powered off master broker without GlassFish shutdown and ORB port stayed up on server2, but after message was delivered it was not removed from the queue. Stand alone client config:
System.setProperty("com.sun.appserv.iiop.endpoints","glassfish1.corp.local:23700,glassfish2.corp.local:23700");
System.setProperty("com.sun.iiop.loadbalancingpolicy", "ic-based");
System.setProperty("com.sun.corba.ee.transport.ORBWaitForResponseTimeout", "5000");
System.setProperty("com.sun.corba.ee.transport.ORBTCPConnectTimeouts","100:500:100:500");
System.setProperty("com.sun.corba.ee.transport.ORBTCPTimeouts","500:2000:50:1000");