How to speed up java heap dump generation…

by perfstories

As you know, we need memory dump for a lot of things: memory leak analysis, memory footprint analysis (high memory footprint could be a problem sometimes) and to check/understand some details about our application.

However, in some circumstances heap dump generation can last tens minutes or even more. And all this time application is not available (yeah, we could set up a cluster, of course, but sometimes, it isn’t done). So that was my case, a server, which I had to get a memory dump from, was not clustered, heap size was about several gigabytes, jmap was used (jmap –F –dump:format=b…..), dump was stored to disk.

Next simple ideas helped me speed up dump generation more than 10 times:

  1. Generate dump to mounted RAM. To implement this do the following preparation:
    • mkdir /mnt/tmpfs
    • chmod 777 /mnt/tmpfs
    • mount –t tmpfs –o size=<~all java process memory (not only heap) and even better to have some reserve> tmpfs /mnt/tmpfs
  2. Let’t generate core-file:
    gdb –pid=<java_pid>
    gcore <core_file_name>
    detach
    quit
  3. Actually, our application became alive after (2).
  4. But, you better need to get heap dump from core-file on the same machine:

    jmap -dump:format=b,file=/mnt/tmpfs/ <heap_dump_file> <path to java>/java <core_file_name>.

    Note about dependencies, only step 3 above depends on step 2 above. So you could, say, don’t mount RAM and just generate core-file online to get heap dump offline after that. Anyway this optimization will decrease your application downtime.

    Also, if you just mount RAM and will generate heap dump as you did before, you’ll speed up too. I mean, you need tell jmap to store heap dump on mounted RAM, as you understood.

    In some cases it’s possible to perform convertation (core-file to heap dump) later.

    For example, as I read in “Troubleshooting Guide for Java SE 6 with HotSpotVM” (http://www.oracle.com/technetwork/java/javase/tsg-vm-149989.pdf, глава 2.2), like this guide a lot ;), it’s possible to open core-file in Visual VM (unfortunately, this feature is available on Linux and Solaris only) and generate heap dump and thread dump by/from Visual VM. I tried that on Linux. But, anyway, it’s better to use a server, where core file was collected from.

    Advertisements