There were a lot of emergencies in my practice with a very simple cause: one of the threads hung. And the single hanging thread isn’t so dangerous per se.
Theory of finding memory leaks is pretty simple. I would formulate it as follows:
There is a lot of articles about that. I remember a really great article: http://olex.openlogic.com/wazi/2009/how-to-fix-memory-leaks-in-java. I recommend it to everyone and thank it’s author.
If you got memory leak while doing performance testing, you are lucky. Usually, it’s very simple. Even if you use J2EE technologies, I mean, say, 80% of code, which is running on JRE, isn’t your team code (I mean container, various libraries, etc). I said simple, as you can reproduce it. Of course, there could be more difficult cases (JDK native code, lack of sources, etc.). But anyway, opportunity to reproduce is a serious bid for victory.
I was needed to clarify several things on Hibernate according to my work (what is a query cache? does it have regions? what is update timestamps cache? does it have regions or not?)
In general, there is a lot of articles, papers, notes, manuals, etc. One of the excellent examples is http://tech.puredanger.com/2009/07/10/hibernate-query-cache/.
So we got some understanding on query cache and update timestamps cache after reading: there is a query cache with query and bound variables as keys and update timestamps cache, that keeps a record corresponding to each table (was it modified or not later than query result was cached).
As you know, we need memory dump for a lot of things: memory leak analysis, memory footprint analysis (high memory footprint could be a problem sometimes) and to check/understand some details about our application.
However, in some circumstances heap dump generation can last tens minutes or even more. And all this time application is not available (yeah, we could set up a cluster, of course, but sometimes, it isn’t done). So that was my case, a server, which I had to get a memory dump from, was not clustered, heap size was about several gigabytes, jmap was used (jmap –F –dump:format=b…..), dump was stored to disk.
Next simple ideas helped me speed up dump generation more than 10 times:
I’m going to tell you about one more emergency in our production environment.
So, we have a server with Glassfish, with 2 jdbc Oracle connection pools. One of these pools was usable, another wasn’t. Just was kept as a reminder of something ;). All external requests to this server were remote jdbc calls.
The symptoms of this emergency were:
I use JD-GUI for decompilation to java now (http://java.decompiler.free.fr/). There is also JD-Eclipse, plugin for Eclipse.
It’s very nice tool. In general, it solves all my problems in this area. Yeah, of course, several times I got some methods decompiled not very good. And, of course, that were the most important methods for me at that moment. 🙂
However, there was more significant inconvinience for me (in both JD-GUI and JD-Eclipse, as they use the same core). Line numbers of decompiled code are not the same as line numbers of source code.