Hi,
sometimes when our backend server (called via RMI) is under heavy load and the application has increasing response times I see a lot of orb related threads with the following stack trace:
Thread "p: thread-pool-1; w: 201" thread-id: 61,539 thread-state: TIMED_WAITING Waiting on lock: com.sun.corba.ee.impl.encoding.BufferQueue@3ac37396
at: java.lang.Object.wait(Native Method)
at: com.sun.corba.ee.impl.encoding.BufferManagerReadStream.underflow(BufferManagerReadStream.java:134)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_1.grow(CDRInputStream_1_1.java:113)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.read_octet_array(CDRInputStream_1_0.java:719)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.getConvertedChars(CDRInputStream_1_0.java:2340)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.internalReadString(CDRInputStream_1_0.java:594)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.readStringOrIndirection(CDRInputStream_1_0.java:576)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.read_codebase_URL(CDRInputStream_1_0.java:1956)
at: com.sun.corba.ee.impl.encoding.CDRInputStream_1_0.read_value(CDRInputStream_1_0.java:1062)
at: com.sun.corba.ee.impl.encoding.CDRInputObject.read_value(CDRInputObject.java:531)
at: com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl$14.read(DynamicMethodMarshallerImpl.java:384)
at: com.sun.corba.ee.impl.presentation.rmi.DynamicMethodMarshallerImpl.readArguments(DynamicMethodMarshallerImpl.java:451)
at: com.sun.corba.ee.impl.presentation.rmi.ReflectiveTie._invoke(ReflectiveTie.java:172)
at: com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatchToServant(CorbaServerRequestDispatcherImpl.java:528)
at: com.sun.corba.ee.impl.protocol.CorbaServerRequestDispatcherImpl.dispatch(CorbaServerRequestDispatcherImpl.java:199)
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequestRequest(CorbaMessageMediatorImpl.java:1624)
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:1486)
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleInput(CorbaMessageMediatorImpl.java:990)
at: com.sun.corba.ee.impl.protocol.giopmsgheaders.RequestMessage_1_2.callback(RequestMessage_1_2.java:214)
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.handleRequest(CorbaMessageMediatorImpl.java:742)
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.dispatch(CorbaMessageMediatorImpl.java:539)
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
One orb related RUNNABLE thread does the following:
Thread "p: thread-pool-1; w: 182" thread-id: 9,636 thread-state: RUNNABLE
at: java.lang.reflect.Array.get(Native Method)
at: com.sun.corba.ee.impl.orbutil.copyobject.ClassCopierFactoryArrayImpl$1.doCopy(ClassCopierFactoryArrayImpl.java:86)
at: com.sun.corba.ee.impl.orbutil.copyobject.ClassCopierBase.copy(ClassCopierBase.java:134)
at: com.sun.corba.ee.impl.orbutil.copyobject.ClassCopierBase.copy(ClassCopierBase.java:123)
...
at: com.sun.corba.ee.impl.protocol.CorbaMessageMediatorImpl.doWork(CorbaMessageMediatorImpl.java:2324)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.performWork(ThreadPoolImpl.java:497)
at: com.sun.corba.ee.impl.orbutil.threadpool.ThreadPoolImpl$WorkerThread.run(ThreadPoolImpl.java:540)
I just wanted to ask if this is normal behaviour or if there is something we can do to improve the performance? As far as I can see in the orb monitoring of the threadpool we don't hit any max values:
- currentbusythreads < numberofavailablethreads
- but numberofworkitemsinqueue > 0 (is this related to the threads in state TIMED_WAITING?)
Regards
Bernd