performance stories

technical essays about performance, life and everything ;)

Yet another hanging…

I’m going to tell you about one more emergency in our production environment.

So, we have a server with Glassfish, with 2 jdbc Oracle connection pools. One of these pools was usable, another wasn’t. Just was kept as a reminder of something ;). All external requests to this server were remote jdbc calls.

The symptoms of this emergency were:

  1. It was impossible to ping usable jdbc pool (GF console -> Resources -> JDBC -> Connection Pools -> General -> ping). I mean ping request was just hanging;
  2. Oracle DBAs saw no problems with sessions, connections, etc.;
  3. Unusable pool could be pinged easily;
  4. The servers, which were using this server, hung up in a special way: their http request handlers pools exhausted gradually, then the servers appeared hanging for some time, then were in working state again, then again from the beginning (exhausted, hung, working).

I had thread dumps as usual


Such a Nested Monitor Lockout

There is a very nice tutorial:

I like it because there are pretty clear descriptions of lockouts. Each of lockouts has its own name (Nested Monitor Lockout, Deadlock, Reentrance Lockout). Before I read this article, I called the all situations with hanging endlessly threads very simple – “Deadlock”.    🙂

There are also methods to prevent deadlock. I remember that hadn’t answered this question at one of my interview (fortunately, there was no impact on result).

And it was very funny to meet Nested Monitor Lockout ( in practice in a couple of days.

More exactly, that was on production environment.

How to restore line numbers after decompilation…

I use JD-GUI for decompilation to java now ( There is also JD-Eclipse, plugin for Eclipse.

It’s very nice tool. In general, it solves all my problems in this area. Yeah, of course, several times I got some methods decompiled not very good. And, of course, that were the most important methods for me at that moment. 🙂

However, there was more significant inconvinience for me (in both JD-GUI and JD-Eclipse, as they use the same core). Line numbers of decompiled code are not the same as line numbers of source code.

And this complicates debug process very much (breakpoints, etc).

%d bloggers like this: