Be aware of SUN Oracle Networking Properties or how to prevent hanging java.net.SocketInputStream.socketRead0……

by perfstories

There were a lot of emergencies in my practice with a very simple cause: one of the threads hung. And the single hanging thread isn’t so dangerous per se.

However, this hanging thread can:

  1. Keep some lock and other threads will hang trying to acquire this lock. After that some pool (say, a connection pool or a pool of http request handlers) can become exhausted. And, of course, this is fatal to the server (see examples in my previous posts: here and here);
  2. Make program running by unusual branch of the code proving some bugs (see here, for example).

Further, there could be a lot of causes, why the first thread hung. One of them is a lack of socket read timeout.

I told you how to set socket read timeout for Oracle JDBC Driver.

Let me share yet several useful options, which prevent a lot of emergencies in our production. These are SUN Oracle networking properties: http://docs.oracle.com/javase/6/docs/technotes/guides/net/properties.html.

So if you like me will use 2 options (-Dsun.net.client.defaultConnectTimeout=57000 and
-Dsun.net.client.defaultReadTimeout=54321), you will get just message about timeout in your log and no hanging thread with trace like [1].

In my opinion, Glassfish should have some notes in it’s configuration file (domain.xml) regarding such essential options like timeouts for Oracle JDBC Driver and Sun network client.

[1]

 
at: java.net.SocketInputStream.socketRead0(Native Method)                
at: java.net.SocketInputStream.read(SocketInputStream.java:129)        
at: java.io.BufferedInputStream.fill(BufferedInputStream.java:218)     
at: java.io.BufferedInputStream.read1(BufferedInputStream.java:258)                             
at: java.io.BufferedInputStream.read(BufferedInputStream.java:317)  
at: sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687)                            
at: sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632)           
at: sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1064)            
at: java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:373)           
at: com.sun.xml.ws.transport.http.client.HttpClientTransport.readResponseCodeAndMessage(HttpClientTransport.java:209)  
at: com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:162)                   
at: com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:94)        
at: com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:89)        
at: com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)                   
at: com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)                     
at: com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)                       
at: com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)                    
at: com.sun.xml.ws.api.pipe.helper.AbstractTubeImpl.process(AbstractTubeImpl.java:112)                          
at: com.sun.xml.xwss.XWSSClientPipe.process(XWSSClientPipe.java:160)                     
at: com.sun.xml.ws.api.pipe.helper.PipeAdapter.processRequest(PipeAdapter.java:115)  
at: com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:598)                   
at: com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:557)                     
at: com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:542)                       
at: com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:439)                    
at: com.sun.xml.ws.client.Stub.process(Stub.java:222)                             
at: com.sun.xml.ws.client.sei.SEIStub.doProcess(SEIStub.java:135)      
at: com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:109)                             
at: com.sun.xml.ws.client.sei.SyncMethodHandler.invoke(SyncMethodHandler.java:89) 
at: com.sun.xml.ws.client.sei.SEIStub.invoke(SEIStub.java:118)            
…
at: org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:236)                             
at: org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)             
at: org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)         
at: org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)                      
at: com.sun.identity.agents.filter.AmAgentBaseFilter.allowRequestToContinue(AmAgentBaseFilter.java:126)                        
at: com.sun.identity.agents.filter.AmAgentBaseFilter.doFilter(AmAgentBaseFilter.java:75)                          
at: org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)         
at: org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)                      
at: org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:96)                      
at: org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)              
at: org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:246)         
at: org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:214)                      
at: org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:313)                    
at: org.apache.catalina.core.StandardContextValve.invokeInternal(StandardContextValve.java:287)           
at: org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:218)                        
at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)         
at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)         
at: com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:94)  
at: com.sun.enterprise.web.PESessionLockingStandardPipeline.invoke(PESessionLockingStandardPipeline.java:98)              
at: org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:222)      
at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)         
at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)         
at: org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)              
at: org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)                   
at: org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:166)                           
at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:648)         
at: org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:593)         
at: org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:587)              
at: org.apache.catalina.core.ContainerBase.invoke(ContainerBase.java:1096)                   
at: org.apache.coyote.tomcat5.CoyoteAdapter.service(CoyoteAdapter.java:288)               
at: com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.invokeAdapter(DefaultProcessorTask.java:647)              
at: com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.doProcess(DefaultProcessorTask.java:579)                     
at: com.sun.enterprise.web.connector.grizzly.DefaultProcessorTask.process(DefaultProcessorTask.java:831)                          
at: com.sun.enterprise.web.connector.grizzly.DefaultReadTask.executeProcessorTask(DefaultReadTask.java:341)                
at: com.sun.enterprise.web.connector.grizzly.ssl.SSLReadTask.process(SSLReadTask.java:440)                
at: com.sun.enterprise.web.connector.grizzly.ssl.SSLReadTask.doTask(SSLReadTask.java:228)                 
at: com.sun.enterprise.web.connector.grizzly.TaskBase.run(TaskBase.java:265)               
at: com.sun.enterprise.web.connector.grizzly.ssl.SSLWorkerThread.run(SSLWorkerThread.java:106)

Advertisements