DNS Performance Troubleshooting

When you are troubleshooting internet performance, there are different parts of the connection should be verified:

 

·        DNS Performance

·        Client Side Performance

·        Server Side Performance

·        Application Performance

·        Network Congestion

 

In this section we are going to investigate DNS Performance.

 

·        When you try to connect online, any request using domain name should first pass through the DNS resolution process to get the mapped IP address

·        To investigate DNS performance using Wireshark, there are couple of steps needed

o   You need to get a capture from testing machine by running wireshark on it or run a SPAN to a sniffing machine to get wider capture

o   Create a profile in Wireshark e.g. DNS Profile by navigating to Edit > Configuration Profiles > '+'

·        This profile can be loaded whenever you want to analyze DNS because it will contain DNS related columns

o   Right click on any column and select Column Preferences

 

 

o   From Appearance > Columns start adding DNS related columns. Mainly

·   DNS Delta Time (dns.time) - this is the time between DNS request and response

·   Domain Name (dns.qry.name) - this is the domain name extract from DNS requests

·   Resolved IP (dns.a) - This is the response IP

o   Now, you need to get indication if the problem is due to slow DNS resolution. For that sort the packets by DNS Delta Time column

 

o   A quick look will indicate that

·   We have DNS responses which are taking more than 1 second which is extremely slow

·   All these responses are sourced from 10.170.7.99 IP (which has to be the DNS server because it the source of response)

·   The source and destination are in the same LAN (10.170.0.0/16 subnet)

·   Yes !! We have problem with DNS resolution

o   Let's find how many responses are slow compared to all responses

·   Find out all DNS responses for internet addresses which are greater than 1 second, use the filter dns.time > 1 and not dns.qry.name contains landomain.com

·   Export the packets using File > Export Specified Packets.

·   Open the new file and go Analysis > Endpoints > Sort by Packets. The number of slow responses from 10.170.7.99 is 124

·        Repeat the same process to get the number of responses from 10.170.7.99 (20620 packets)

 

o   We can see that its not extremely packet (124 / 20620 ) but still we have some slowness which needs to be investigated

 

·        There are many reasons for DNS responses to be slow

o   If the client and server are not on LAN, check the congestion

o   Problem with the DNS server such as high load

o   Problem with the internet if the DNS server is pointed to external forwarder

o   Problem with the external forwarder (you can try another one)

 

TCP Transport Layer # 102 - Sequence and Acknowledgement Numbers

·        Sequence numbers are generated differently by operating systems based on coded algorithms

o   SEQ and ACK number fields are 32-bits which makes a possibility of 2^32 = 4E9

·        When new connection is started, the sender has to generate Initial Sequence Number (ISN)

o   It's important to generate random ISN to avoid hijacking TCP session. How it works?

 

 

·        NMAP can be used to identify how easy is the algorithm to generate ISN

o   ISN will be seen in SYN TCP packet sent from sender ---> receiver & receiver ----> sender

·        Let's examine full session

Packet#1 (3-WAY TCP Handshake   S -----> SYN -----> R)

Packet#2 (3-WAY TCP Handshake    S <----- SYN/ACK <----- R)

Packet#3 (3-WAY TCP Handshake   S -----> ACK -----> R)

Packet#4 (Data Transmission   S -----> R)

 

o   Because this is SSL session the client is sending Client Hello

o   This packet and previous packet were from sender to receiver and have same SEQ=1

·        Because previous sender to receiver segment size was zero bytes  (SYN/ACK packets don't carry application data)

Packet#5 (R -----> ACK -----> S)

 

o   Next packet from receiver has ACK = SEQ (1) + 205 = 206 bytes, i.e. the receiver got the previous 205-bytes segment and waiting for next one

o   This receiver packet and previous receiver packet are having same SEQ=1

·        This is because previous segment from receiver to sender size was zero bytes (Packet#2)

Packet#6 (Data Transmission    R -----> S)

 

o   As part of SSL handshake, the server responds with Server Hello

·        Because Server Hello contains server certificate chains, the application data size is greater than sender MSS value (1460 bytes)

·        Accordingly you will notice that Packets 6 - 8 and 10 - 12 are TCP segments which belong to single application data (labeled as reassembled PDU)

§  To recap, this is the reason for encapsulating application data over transport layer instead of sending it directly over IP layer because application layer don't have mechanisms for error/flow control. Sending such large data as one packet over IP layer will get it dropped because typical Ethernet networks are limited to 1500-bytes without fragmentation. Transport layer manages the segmentation and transportation of this data.

o   The receiver could have sent Server Hello before Previous ACK (Packet#5) because the sender window size in Packet#4 was 66048 bytes

·        The order of sending packets is subject to the operating system

·        The only caveat is to make sure that ACK is sent before retransmission timer expires at sender

o   SEQ is still '1' because previous segment size was zero bytes

o   ACK is still '206' because not data received from sender yet

·        Note that receiver keeps sending ACK flag although previous ACK already sent. This is to avoid sender retransmission timer expiry

o   Segment Size is 93 bytes

 

Packet#7 (Data Transmission    R -----> S)

 

o   As mentioned earlier, this packet is continuation of same application data

o   SEQ = 1 + 93 = 94 bytes

o   ACK = 206 bytes (no segments received from sender)

o   Segment size = 1360-bytes

Packet#8 (Data Transmission    R -----> S)

o   As mentioned earlier, this packet is continuation of same application data

o   SEQ = 94 + 1360 = 1454 bytes

o   ACK = 206 bytes (no segments received from sender)

o   Segment size is 1360 bytes

o   You can see that receiver sent 93 + 1360 + 1360 = 2813 bytes to sender without receiving ACK yet because sender window size in Packet#4 was 66048 bytes

Packet#9 (Data Transmission    S -----> R)

o   This is an ACK packet from sender to receiver before receiver retransmission timer expiry

o   SEQ = 206-bytes

o   ACK = 1454 + 1360 = 2814 bytes

o   Segment size = 0-bytes because SYN/ACK packets don't carry application data

 

TCP Transport Layer # 101

·        TCP protocol runs in the Transport layer

o   Transport layer is responsible to establish/tear-down sessions (called virtual connections)

·        TCP is one type of Transport layer protocols which defines how to establish/tear-down sessions and how to transmit upper-layers

·        TCP uses a combination of headers between sender and receiver to manage each session

o   Application data is carried over these sessions (which is a combination of application headers and raw-data)

·        Why not to send application data over IP Layer directly instead of introducing Transport Layer?

·        Application Data don't have mechanisms to avoid data loss and corruption therefore they are encapsulated over transport layer which enable features like error detection, retransmissions, etc. For example, HTTP protocol which carries HTTP data don't have headers for error detection or recovery. However, because its encapsulated over TCP, lost data segments will be detected and retransmitted

 

 

·        Summary of TCP headers, size of each header and role of each header

 

·        TCP Windowing

o   TCP protocol divides the actual application data into segments

o   The size of each segment is communicated using TCP Option Max Segment Size (MSS) during session handshake

o   Because of TCP reliability each segment needs to be acknowledged with ACK Flag packet

o   While sender is waiting for ACK packet, it will send other TCP segments for faster communication

o   The number of segments allowed to be sent without receiving ACK packets for them is called Window Size

·        Window Size will be selected by operating system

·        Max possible Windows Size is 2^16 = 65,536 bytes

·        This means for MSS 1420 bytes, the max number of segments to be sent without ACK = 65536 / 1420 = 46.1521 segments

·        If no ACK received for X-Allowed packets within timeout, the sender will retransmit

 

Example#1

Host B is sending data to Host A, using a window size equal to one. This means that Host B is expecting an "ACK" for each data segment it sends to Host A. Once the first data segment is sent, Host A receives it and sends an "ACK 2" to Host B. You might be wondering why "ACK 2" and not just "ACK"?

The "ACK 2" is translated by Host B to say: 'I acknowledge (ACK) the packet you just sent me and I am ready to receive the second (2) segment'. So Host B gets the second data segment ready and sends it off to Host A, expecting an "ACK 3" response from Host A so it can send the third data segment for which, as the picture shows, it receives the "ACK 3".

However, if it received an "ACK 2" again, this would mean something went wrong with the previous transmission and Host B will retransmit the lost segment. We will see how this works in the Acknowledgments section later on.

Example#2

 

We have a window size equal to 3, which means that Host B can send 3 data segments to Host A before expecting an "ACK" back. Host B sends the first 3 segments (Send 1, Send 2 and Send 3), Host A receives them all in good condition and then sends the "ACK 4" to Host B. This means that Host A acknowledged the 3 data segments Host B sent and awaits the next data segments which, in this case, would be 4, 5 and 6

 

·        TCP Acknowledgment. How it works?

o   TCP uses positive acknowledgement retransmission method

o   In this method, the sender will transmit segments based on the received windows size

o   Once the sender transmits, it will start a timer to track the ACK response

o   If no ACK response received, sender will transmit the segments again

o   Additionally, if the receiver gets packets not in right sequence, it sends ACK message to sender indicating to retransmit the missing segment