GF4 cluster stuck starting up with an EJB Timer deployed

Hi,

I came across a weird problem starting up a GF4 two node cluster.

As I said, I set up a two node cluster working quite smooth. As I was pretty much interested on how the EJB timers behave deployed on this environment, I set up GF4 cluster with an external DB for the Ejb timers and I deployed a simple EJB Timer. The timer worked nicely on the cluster as it should be once it was deployed, but I played with the cluster to see how the EJB stands up.

1/ I shutdown the node where the timer was running.
2/ EJB timer failed over to the other node successfully and the timer continues ticking.
3/ I started up the stopped node, though I had no success!.
4/ Timer is running okay.

After this, I decided to restart the entire cluster (two nodes) but what i stroke me was that no node restarted. Both nodes were stuck at some stage, it seemed to me related with the timers database but I am not fully sure.

[2014-06-24T16:55:03.149+0100] [glassfish 4.0] [INFO] [] [javax.enterprise.system.container.ejb.org.glassfish.ejb.persistent.timer] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1403625303149] [levelValue: 800] [[
==> Restoring Timers ... ]]

[2014-06-24T16:55:03.253+0100] [glassfish 4.0] [INFO] [] [org.eclipse.persistence.session.file:/opt/glassfish4/glassfish/nodes/genesis2/ntcs-test2/applications/ejb-timer-service-app/WEB-INF/classes/___EJB__Timer__App] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1403625303253] [levelValue: 800] [[
EclipseLink, version: Eclipse Persistence Services - 2.5.0.v20130507-3faac2b]]

[2014-06-24T16:55:04.599+0100] [glassfish 4.0] [INFO] [jts.startup_msg] [javax.enterprise.system.core.transaction.com.sun.jts.CosTransactions] [tid: _ThreadID=1 _ThreadName=main] [timeMillis: 1403625304599] [levelValue: 800] [[
JTS5014: Recoverable JTS instance, serverId = [100]]]

I SELECTed the EJB timer database and timerid was there and the node processes looked stuck at that point and they not show up as started on the DAS. At this stage, I decided to kill the node process, undeploy the EJB application and restart the DAS. After that I could start up the cluster again but without the EJB timer application.

Does anybody idea about this behavior?

Cheers

EJB timer tested:

@SuppressWarnings("unused")
@Schedule(second="*/1", minute="*", hour="8-23", dayOfWeek="Mon-Fri",
dayOfMonth="*", month="*", year="*", info="MyTimer")
private void scheduledTimeout(final Timer t) {
System.out.println("@Schedule called at: " + new java.util.Date());
}