Some time ago our performance testing team run into a trouble with our Java application.

They we running a performance test where multiple automatic client applications make a lot of requests to the server. It is a usual routine for high-performance applications which allows to estimate system capacity and to gather some statistics.

Occasionally, automatic clients hung. Nothing was presented in system logs. No faults, no errors, nothing. Taking into account that all this was happening on a virtual machine on the other side of the globe, it was really hard to realize what's wrong. A couple of hours was spent checking and rechecking the configuration, the app installation etc.

In the end, I configured a client to run in JVM debug mode and connected to it with remote debugger (a great Java feature). After digging deep into the stack it turned out that the main application thread was waiting somewhere in the AWT code. After getting AWT sources I realized that something was wrong with The X Window System. Despite client application doesn't show UI in automatic mode, it still calls some AWT methods. Restart of the X server helped.

The cause of the problem was unusual: the X server was running, which lead to no exception was thrown from inside AWT, but apparently was hung, which lead to the client was waiting infinitely for connection to the server.