performance stories

technical essays about performance, life and everything ;)

Month: May, 2012

Yet another hanging java.net.SocketInputStream.socketRead0…

I’m going to tell you about one more emergency in our production environment.

So, we have a server with Glassfish, with 2 jdbc Oracle connection pools. One of these pools was usable, another wasn’t. Just was kept as a reminder of something ;). All external requests to this server were remote jdbc calls.

The symptoms of this emergency were:

  1. It was impossible to ping usable jdbc pool (GF console -> Resources -> JDBC -> Connection Pools -> General -> ping). I mean ping request was just hanging;
  2. Oracle DBAs saw no problems with sessions, connections, etc.;
  3. Unusable pool could be pinged easily;
  4. The servers, which were using this server, hung up in a special way: their http request handlers pools exhausted gradually, then the servers appeared hanging for some time, then were in working state again, then again from the beginning (exhausted, hung, working).

I had thread dumps as usual

Advertisements

Such a Nested Monitor Lockout

There is a very nice tutorial: http://tutorials.jenkov.com/java-concurrency/index.html.

I like it because there are pretty clear descriptions of lockouts. Each of lockouts has its own name (Nested Monitor Lockout, Deadlock, Reentrance Lockout). Before I read this article, I called the all situations with hanging endlessly threads very simple – “Deadlock”.    🙂

There are also methods to prevent deadlock. I remember that hadn’t answered this question at one of my interview (fortunately, there was no impact on result).

And it was very funny to meet Nested Monitor Lockout (http://tutorials.jenkov.com/java-concurrency/nested-monitor-lockout.html) in practice in a couple of days.

More exactly, that was on production environment.

%d bloggers like this: