Although they appear somewhat complex, lock chains are fairly straightforward. The primary limitation is the process needs to be running in the shell. Hi PH , We have already enabled Verbose GC for our SOA and OSB domains . -XX:-UseGCOverheadLimit -Xverify:none -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:/opt/soauser/gclogs/SOA_MS1-`date '+%Y-%m-%d-%H:%M:%S'`.log Attached is the startup script and set domain In the simplest case, thread A is waiting for thread B, while thread B is waiting for thread A. Source
idx=0x28 tid=18983 prio=5 alive, in native, daemon "(GC Worker Thread 8)" id=? The request originates from a RMI and invoking the following Oraclle BPEL EJB: oracle.soa.management.internal.ejb.impl.FacadeFinderBean_4vacyo_FacadeFinderBeanImpl_WLSkel My next recommendation is that you revisit your current Oracle SOA / BPEL audit level and see So you mean to say the issue is caused because of JVM trashing , and the JVM is trashed because of improper GC tuning on SOA domain ? What is this plane and what are the rockets on the back used for? http://stackoverflow.com/questions/14529848/jrockit-getting-stuck
Also as I see from the thread dumps the problem is in the AbstractBundlePesistenceManager syncronised methods and the fact that there is only one instance of Oracle PM per repository. 4) Start of Managed Server log with envt details: starting weblogic with Java version: java version "1.6.0_33" Java(TM) SE Runtime Environment (build 1.6.0_33-b03) Oracle JRockit(R) (build R28.2.4-14-151097-1.6.0_33-20120618-1634-linux-x86_64, compiled mode) Starting WLS with memory leak (OldGen growing over time).
I sent the wrong thread dump. It can be as simple healthy pattern like multiple threads executing requests to the same set of servlet or application layer or it can be a symptom of bottleneck if multiple That variable is of type BundleCache. How To Analyze Thread Dump In Weblogic Using Samurai Tool You can follow my article below for the detail: http://javaeesupportpatterns.blogspot.ca/2012/02/prstat-linux-how-to-pinpoint-high-cpu.htmlRegards, P-H phcharbonneau Reply | Threaded Open this post in threaded view ♦ ♦ | Report Content as Inappropriate ♦ ♦
Hi PH , Thanks for the one round of analysis and really appreciate your work . Weblogic Kernel Default Self Tuning Waiting For Lock Like Show 0 Likes(0) Actions 5. Show Andrey Adamovich added a comment - 07/Oct/09 09:59 1) No, we haven't used 1.6, but as far as I noticed from the code nothing really changed in bundle db pm We are working on creating several scenarios that would/could duplicate this situation.
Unless the STDERR was redirected to a log file, the thread dump cannot be saved when issued via kill signals. Stuck Executethread For Queue 'weblogic.kernel.default (self-tuning)' From the operating system's point of view the Java threads are just like any application threads. FYI OSB Thread Dump are always the most complex to analyze given its high level of abstraction. i noticed that call on a number of your stack traces.
Stack trace: Thread-15 "[STUCK] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'"
We have restarted OSB_MS1 and later we observed the transactions are being flowed PROD environment . this contact form Analyze the locking and try to establish a order of locking or to avoid the locks entirely. Thanks, Kiran phcharbonneau Reply | Threaded Open this post in threaded view ♦ ♦ | Report Content as Inappropriate ♦ ♦ Re: Stuck Threads Leading to all servers hung . Stack Overflow Questions Developer Jobs Documentation beta Tags Users current community help chat Stack Overflow Meta Stack Overflow your communities Sign up or log in to customize your list. How To Resolve Stuck Threads In Weblogic
Furthermore, a process consists of many concurrently running threads. Thread #8 is attempting to do the following: OSB is called to execute a business function. If there is no additional thread waiting for a lock held by C, then A->B->C is a complete and open lock chain. have a peek here Yes , We have done this ((-XX:-UseGCOverheadLimit) ) because support suggested us to do explicitly disable the HotSpot 1.6 GC overhead limit detector when we have OOM issues in the domain
Analyze: I've got some thread dumps and solved the problem by commenting these thread dumps. Thread dump analysis is never easy, so many problem patterns. Weblogic.socket.muxer Threads Blocked All the transactions were failed and console page was still hung . Usually there are many "benign" threads which are harmless and we should ignore these to narrow down to the actual thread that is causing an issue.For ex.
All of these data points can show what is getting executed by the thread even if we haven't written the code or truly understand the implementation. "[ExecuteThread: '4' for queue: 'weblogic.kernel.Default JCA Shrink Frequency Datasource Shrink Frequency Datasource Inactive Connection Timeout Important: if you have set the "initial connection" to "0″ on the jca adapter pool and the shrinking frequencies respect the following: phcharbonneau Reply | Threaded Open this post in threaded view ♦ ♦ | Report Content as Inappropriate ♦ ♦ Re: Stuck Threads Leading to all servers hung . Fat Lock Weblogic How frequent are you observing such failure on SOA domain?
I'm particulary interested in these STUCK Threads below. Or its possible its waiting for an event to happen and continues in same state and stack appearance. Blocked Chains A blocked lock chain is made up of a lock chain whose head thread is also part of another lock chain, which can be either open or deadlocked. Check This Out Re: Blocked trying to get lock: [email protected][fat lock] Mauro Flores Feb 17, 2011 5:06 PM (in response to Joe Weinstein-Oracle) Joe Weinstein wrote: Your driver bug case might be the best
Use -Dweblogic.SocketReaders=4 to tweak the number of Muxer threads. SOA managed server #3 is the worst of the 3 but problem is also observed from the 2 others >> Java Heap OldGen space is maxed out! A thread can allocate small objects in its own TLA without synchronizing with other threads. Verify GC or Memory is not a bottleneck Check that Finalizer thread is not blocked.
There is another level of tagging based on a thread getting returned to the thread pool or not within a defined time period. idx=0x2c tid=3544 prio=5 alive, in native, daemon "(Code Generation Thread 1)" id=4 idx=0x30 tid=3545 prio=5 alive, in native, native_waiting, daemon "(Code Optimization Thread 1)" id=5 idx=0x34 tid=3546 prio=5 alive, in native, Let take the following thread stack. Administrator Hi Kiran, Sorry for delay on this one.
This thread stack taken from a thread dump. Open Chains Open lock chains represent a straight dependency, thread A is waiting for B which is waiting for C, and so on. We are not seeing any exceptions, but just a lot of stuck threads. This code is the culprit and on drilling down further, this was firing a bad performing SQL which had a filter with a function that did not hit the index :
The method name gives us a general idea of what the invocation is about. RMI intra communication etc. Posted by Sabha Parameswaran at 2:58 PM Email ThisBlogThis!Share to TwitterShare to FacebookShare to Pinterest Labels: Analysis, Bottleneck, Java, Performance, Thread Dumps, WebLogic, WLS 9 comments: JaiDecember 20, 2012 at 6:40 Almost invariably, the thread stack is in English which lets the user read the package, classname and method call.
Enabling traces is a "process" in the environment where this happens. Why does smartphone GPS find its position much faster than a GPS module? Need your more analysys on OSB and SOA logs and need to find RCA , Let me know if you need anything else . Will it not be possible to move synronisation tasks directly to that class?
File size will depend how much diagnostic data was turned on, last time of reset etc. - Please identify the source of the Thread orabpel.invoke.pool-4.thread-63 e.g. This problem is very critical to us. Anytime or specific times?