When a connection drop occurred, we supplied the times, they pulled the packets and we ran them through Wireshark. Because this error wasn’t occurring frequently, our network team set up packet captures in the background. We were still occasionally seeing this issue in one of our data centers but not the other. The replacement was expedited and it was expected that the connection drop issue would go away. Since that device was end-of-life, it was scheduled for replacement.

Leveraging previous experience with TCP retransmissions and working with our network team, we found that this app server and database were conversing across a networking device that was known to be problematic and was dropping packets. There was one hit (Doc ID 1286376.1) that was a pretty close fit for the error messages we were seeing, but all it pointed to was that a “client connection has experienced a timeout”. Eventually, they contacted our performance engineering team and I engaged with them.įirst stop was the Oracle MOS site. At this point, our DBAs began a months-long foray into “vendors-point-fingers-at-each-other hell”. Support tickets were opened with Oracle and with the vendor of the third-party application (CA).

This problem started with a series of TNS-12535 messages that were seen in the Oracle alert logs for one of our databases:Ĭlient address: (ADDRESS=(PROTOTOL=tcp)(HOST=10.xxx.xxx.222)(PORT=39488) We recently resolved a long-standing issue with TCP retransmissions that were causing connection drops between an application server and one of our databases and I thought this might help others faced with similar issues. Lately, I’ve been making use of packet captures and Wireshark to solve tough issues in the TCP layer. As a long-time performance DBA, I’ve often felt that it is important to know something about troubleshooting the layers that are upstream and downstream of the database in the technology stack.

Using wireshark to recover flag