VRF-Aware IKEv2 DMVPN (+ iVRF/fVRF + EIGRP)

R1

……………………………………………………………………………………………………………………………………………

vrf definition dmvpn

 !

 address-family ipv4

 exit-address-family

!

crypto ikev2 proposal prop-01

 encryption aes-cbc-128 aes-cbc-192

 integrity sha256 sha512

 group 14 15

!

crypto ikev2 policy pol-01

 proposal prop-01

!

crypto ikev2 keyring dmvpn-key

 peer ALL

  address 0.0.0.0 0.0.0.0

  pre-shared-key cisco123

!

crypto ikev2 profile prof-01

 match identity remote address 0.0.0.0

 authentication local pre-share

 authentication remote pre-share

 keyring local dmvpn-key

!

crypto ipsec transform-set tset esp-aes 192 esp-sha512-hmac

 mode tunnel

!

crypto ipsec profile dmvpn

 set transform-set tset

 set ikev2-profile prof-01

!

interface Loopback0

 vrf forwarding dmvpn

 ip address 10.150.10.1 255.255.255.255

!

interface Tunnel0

 vrf forwarding dmvpn          !!! …. This is to define iVRF

 ip address 192.168.1.1 255.255.255.0

 no ip redirects

 no ip split-horizon eigrp 102

 ip nhrp map multicast dynamic

 ip nhrp network-id 1

 ip nhrp redirect

 tunnel source GigabitEthernet0/0

 tunnel mode gre multipoint

 tunnel protection ipsec profile dmvpn

!

interface GigabitEthernet0/0

 ip address 10.150.1.1 255.255.255.0

!

router eigrp 1

 !

 address-family ipv4 vrf dmvpn autonomous-system 102

  network 10.150.10.1 0.0.0.0

  network 192.168.1.1 0.0.0.0

 exit-address-family

!

ip route 0.0.0.0 0.0.0.0 10.150.1.2

R1# sh cry ikev2 sa

 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status

1         10.150.1.1/500        10.150.5.1/500        none/dmvpn           READY

      Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 86400/5847 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status

2         10.150.1.1/500        10.150.6.1/500        none/dmvpn           READY

      Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 86400/4183 sec

 IPv6 Crypto IKEv2  SA

R5

……………………………………………………………………………………………………………………………………………

vrf definition dmvpn

 !

 address-family ipv4

 exit-address-family

!

crypto ikev2 proposal prop-01

 encryption aes-cbc-128 aes-cbc-192

 integrity sha256 sha512

 group 14 15

!

crypto ikev2 policy pol-01

 match fvrf dmvpn     !!! ….. Because tunnel is invoked using VRF, this policy should be assigned to same Front VRF

 match address local 10.150.5.1     !!! ….. This source interface should be defined as its in VRF Routing Table instead of Global Routing Table

 proposal prop-01

!

crypto ikev2 keyring dmvpn-key

 peer R1

  address 0.0.0.0 0.0.0.0

  pre-shared-key cisco123

!

crypto ikev2 profile prof-01

 match fvrf dmvpn         !!! ….. Because tunnel is invoked using VRF, this profile should be assigned to same Front VRF

 match identity remote address 0.0.0.0

 authentication local pre-share

 authentication remote pre-share

 keyring local dmvpn-key

!

crypto ipsec transform-set tset esp-aes 192 esp-sha512-hmac

 mode tunnel

!

crypto ipsec profile dmvpn

 set transform-set tset

 set ikev2-profile prof-01

!

interface Loopback0

 vrf forwarding dmvpn

 ip address 10.150.50.1 255.255.255.255

!

interface Tunnel0

 vrf forwarding dmvpn    !!! …. Assign iVRF same as fVRF

 ip address 192.168.1.2 255.255.255.0

 no ip redirects

 ip nhrp map 192.168.1.1 10.150.1.1

 ip nhrp map multicast 10.150.1.1

 ip nhrp network-id 1

 ip nhrp nhs 192.168.1.1

 ip nhrp shortcut

 tunnel source GigabitEthernet0/0

 tunnel mode gre multipoint

 tunnel vrf dmvpn    !!! … Invoke the tunnel using Front VRF used on the source interface

 tunnel protection ipsec profile dmvpn

!

interface GigabitEthernet0/0

 vrf forwarding dmvpn

 ip address 10.150.5.1 255.255.255.0

!

router eigrp 1

 !

 address-family ipv4 vrf dmvpn autonomous-system 102

  network 10.150.50.1 0.0.0.0

  network 192.168.1.2 0.0.0.0

 exit-address-family

!

ip route vrf dmvpn 0.0.0.0 0.0.0.0 10.150.5.2

R5#sh cry ikev2 sa

 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status

2         10.150.5.1/500        10.150.6.1/500        dmvpn/dmvpn          READY

      Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 86400/2475 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status

1         10.150.5.1/500        10.150.1.1/500        dmvpn/dmvpn          READY

      Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 86400/6088 sec

 IPv6 Crypto IKEv2  SA

R6 (This is same as R5 except that F-VRF is different from I-VRF)

……………………………………………………………………………………………………………………………………………

vrf definition fvrf-dmvpn

 !

 address-family ipv4

 exit-address-family

!

vrf definition ivrf-dmvpn

 !

 address-family ipv4

 exit-address-family

!

crypto ikev2 proposal prop-01

 encryption aes-cbc-128 aes-cbc-192

 integrity sha256 sha512

 group 14 15

!

crypto ikev2 policy pol-01

 match fvrf fvrf-dmvpn

 proposal prop-01

!

crypto ikev2 keyring dmvpn-key

 peer R1

  address 0.0.0.0 0.0.0.0

  pre-shared-key cisco123

!

crypto ikev2 profile prof-01

 match fvrf fvrf-dmvpn

 match identity remote address 0.0.0.0

 identity local address 10.150.6.1

 authentication local pre-share

 authentication remote pre-share

 keyring local dmvpn-key

!

crypto ipsec transform-set tset esp-aes 192 esp-sha512-hmac

 mode tunnel

!

crypto ipsec profile dmvpn

 set transform-set tset

 set ikev2-profile prof-01

!

interface Loopback0

 vrf forwarding ivrf-dmvpn

 ip address 10.150.60.1 255.255.255.255

!

interface Tunnel0

 vrf forwarding ivrf-dmvpn

 ip address 192.168.1.3 255.255.255.0

 no ip redirects

 ip nhrp map 192.168.1.1 10.150.1.1

 ip nhrp map multicast 10.150.1.1

 ip nhrp network-id 1

 ip nhrp nhs 192.168.1.1

 ip nhrp shortcut

 tunnel source GigabitEthernet0/0

 tunnel mode gre multipoint

 tunnel vrf fvrf-dmvpn

 tunnel protection ipsec profile dmvpn

!

interface GigabitEthernet0/0

 vrf forwarding fvrf-dmvpn

 ip address 10.150.6.1 255.255.255.0

!

router eigrp 1

 !

 address-family ipv4 vrf ivrf-dmvpn autonomous-system 102

  network 10.150.60.1 0.0.0.0

  network 192.168.1.3 0.0.0.0

 exit-address-family

!

ip route vrf fvrf-dmvpn 0.0.0.0 0.0.0.0 10.150.6.2

R6# sh cry ikev2 sa

 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status

1         10.150.6.1/500        10.150.1.1/500        fvrf-dmvpn/ivrf-dm   READY

      Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 86400/4924 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status

2         10.150.6.1/500        10.150.5.1/500        fvrf-dmvpn/ivrf-dm   READY

      Encr: AES-CBC, keysize: 128, PRF: SHA256, Hash: SHA256, DH Grp:14, Auth sign: PSK, Auth verify: PSK

      Life/Active Time: 86400/2973 sec

 IPv6 Crypto IKEv2  SA

How Diffie Hellman Secret Session Key is Generated?

• DH RFCs predefine p and g values for each DH group
• Example is RFC 5114 which defines

p = B10B8F96 A080E01D DE92DE5E AE5D54EC 52C99FBC FB06A3C6
       9A6A9DCA 52D23B61 6073E286 75A23D18 9838EF1E 2EE652C0
       13ECB4AE A9061123 24975C3C D49B83BF ACCBDD7D 90C4BD70
       98488E9C 219A7372 4EFFD6FA E5644738 FAA31A4F F55BCCC0
       A151AF5F 0DC8B4BD 45BF37DF 365C1A65 E68CFDA7 6D4DA708
       DF1FB2BC 2E4A4371

g = A4D1CBD5 C3FD3412 6765A442 EFB99905 F8104DD2 58AC507F
       D6406CFF 14266D31 266FEA1E 5C41564B 777E690F 5504F213
       160217B4 B01B886A 5E91547F 9E2749F4 D7FBD7D3 B9A92EE1
       909D0D22 63F80A76 A6A24C08 7A091F53 1DBF0A01 69B6A28A
       D662A4D1 8E73AFA3 2D779D59 18D08BC8 858F4DCE F97C2A24
       855E6EEB 22B3B2E5

• How it works?
• Initiator chooses a secret integer a = 4, then sends Responder A = ga mod p
• A = 54 mod 23 = 4
• Responder chooses a secret integer b = 3, then sends Initiator B = gb mod p
• B = 53 mod 23 = 10
• Initiator computes s = Ba mod p
• s = 104 mod 23 = 18
• Responder computes s = Ab mod p
• s = 43 mod 23 = 18
• Initiator and Responder now share a secret (the number 18)
• Note that a and b can be as large as 600 digits

Diffie Hellman Group Selection in IKEv2

• Because the initiator sends its KEi value in the IKE_SA_INIT, it must guess the DH group that the responder will select from its list of supported groups.  If the initiator guesses wrong, the responder will respond with a Notify payload of type INVALID_KE_PAYLOAD indicating the selected group.  In this case, the initiator MUST retry the IKE_SA_INIT with the corrected DH group.

*Feb  7 16:17:56.924: IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 14

*Feb  7 16:17:56.924: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED

*Feb  7 16:17:56.925: IKEv2:(SESSION ID = 1,SA ID = 1):Request queued for computation of DH key

*Feb  7 16:17:56.926: IKEv2:IKEv2 initiator - no config data to send in IKE_SA_INIT exch

*Feb  7 16:17:56.926: IKEv2:(SESSION ID = 1,SA ID = 1):Generating IKE_SA_INIT message

*Feb  7 16:17:56.927: IKEv2:(SESSION ID = 1,SA ID = 1):IKE Proposal: 1, SPI size: 0 (initial negotiation),

Num. transforms: 11

   AES-CBC   AES-CBC   SHA256   SHA384   SHA256   SHA384   DH_GROUP_2048_MODP/Group 14   DH_GROUP_256_ECP/Group 19   DH_GROUP_1536_MODP/Group 5   DH_GROUP_4096_MODP/Group 16   DH_GROUP_3072_MODP/Group 15

*Feb  7 16:17:56.931: IKEv2:(SESSION ID = 1,SA ID = 1):Sending Packet [To 10.150.2.1:500/From 10.150.1.1:500/VRF i0:f0]

Initiator SPI : 97211F401E241BD2 - Responder SPI : 0000000000000000 Message id: 0

IKEv2 IKE_SA_INIT Exchange REQUEST

Payload contents:

 SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)

*Feb  7 16:17:56.937: IKEv2:(SESSION ID = 1,SA ID = 1):Insert SA

*Feb  7 16:17:56.950: IKEv2:(SESSION ID = 1,SA ID = 1):Received Packet [From 10.150.2.1:500/To 10.150.1.1:500/VRF i0:f0]

Initiator SPI : 97211F401E241BD2 - Responder SPI : 0000000000000000 Message id: 0

IKEv2 IKE_SA_INIT Exchange RESPONSE

Payload contents:

 NOTIFY(INVALID_KE_PAYLOAD)

*Feb  7 16:17:56.952: IKEv2:(SESSION ID = 1,SA ID = 1):Processing IKE_SA_INIT message

*Feb  7 16:17:56.952: IKEv2:(SESSION ID = 1,SA ID = 1):Processing invalid ke notification, we sent group 14, peer prefers group 5

*Feb  7 16:17:56.953: IKEv2-ERROR:(SESSION ID = 1,SA ID = 1):

*Feb  7 16:17:56.953: IKEv2:(SESSION ID = 1,SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 5

*Feb  7 16:17:56.954: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED

*Feb  7 16:17:56.954: IKEv2:(SESSION ID = 1,SA ID = 1):Request queued for computation of DH key

*Feb  7 16:17:56.955: IKEv2:IKEv2 initiator - no config data to send in IKE_SA_INIT exch

*Feb  7 16:17:56.956: IKEv2:(SESSION ID = 1,SA ID = 1):Generating IKE_SA_INIT message

PFS in IKEv2

• Unlike IKEv1, the Perfect Forwarding Secrecy (PFS) Diffie-Hellman (DH) group value displays as 'PFS (Y/N): N, DH group: none' during the first tunnel negotiation.
• After a rekey occurs, the correct values appear. This is not a bug even though the behavior is described in Cisco bug ID CSCug67056.
• The reason that IKE_AUTH messages do not contain KEi/KEr or Ni/Nr payloads. Thus, the SA payloads in the IKE_AUTH exchange cannot contain PFS DH group other than NONE.
• The CREATE_CHILD_SA request used in rekeying MAY optionally contain a KE payload for an PFS DH exchange

For rekeying IKE_SA, the structure of CREATE_CHILD_SA is

   Initiator                                                 Responder
   -------------------------------------------------------------------
   HDR, SK {SA, Ni, KEi} -->

                          <-- hdr="" ker="" nbsp="" nr="" p="" sk="">

For rekeying CHILD_SA, the structure of CREATE_CHILD_SA is

   Initiator                                                 Responder
   -------------------------------------------------------------------
   HDR, SK {N(REKEY_SA), SA, Ni, [KEi,]
       TSi, TSr}   -->

                                          <-- br="" hdr="" nbsp="" nr="" r="" sk="">                                                           TSi, TSr}

IKEv1 vs. IKEv2

• IKEv1 and v2 aren't interoperable
• Fragmentation
• In IKEv1, large packets are encrypted then segmented. The segments are encapsulated in UDP packets
• In IKEv2, large packets are segmented then segments are encrypted.
• Delete Notification
• In IKEv1, delete notifications aren't acknowledged. Once delete is sent, SA will be deleted from local SAD
• In IKEv2, delete notifications are acknowledged. The initiator will wait for ACK or re-Xmit timeout before deleting SA from SAD
• This is resolved if DPD is used

AnyConnect VPN DTLS vs TLS

Difference

DTLS is used for delay sensitive applications (voice and video) as its UDP based while TLS is TCP based

DTLS is supported for AnyConnect VPN not in IKEv2

How it works?

• SSL−Tunnel is the TCP tunnel that is first created to the ASA
• When it is fully established, the client will then try to negotiate a UDP DTLS−Tunnel
• During DTLS negotiation, traffic will be passing over TLS tunnel
• When the DTLS−Tunnel is fully established, all data now moves to the DTLS−tunnel and the SSL−tunnel is only used for occasional control channel traffic
• In case of failures in establishing DTLS Tunnel, traffic will continue passing over TLS tunnel
• After establishing DTLS, in the event of failure in DTLS Tunnel, traffic will pass over TLS tunnel until DTLS tunnel is reestablished

How Data is Forwarded?

• For each packet there is a part in AnyConnect client code which decides whether to send the packet over TLS or DTLS
• If the DTLS tunnel is established, the code will decide to forward the packet over DTLS and start encryption
• If the DTLS is dead, the code will decide to forward the packet over TLS and start encryption

• The key point is the performance of DTLS tunnel
• Since DTLS is based on UDP, it is unreliable and there is no flow control to decide its performance
• Performance can be determined using DPD packets
• When DPD is triggered and no response received, AnyConnect client will start forwarding packets over TLS (assuming TLS is up and DTLS is unhealthy)
• Therefore, there is a packet drop period between DTLS failing and DPD triggering/detection. During this time, AnyConnect client will be forwarding packets over DTLS but they will be lost because DTLS is unhealthy
• In case DTLS is established again, AnyConncect client will forward packets over DTLS
• For receiving ASA with healthy DTLS and TLS, it will reply based on the receiving tunnel, i.e. if packets received over TLS, the response will be over TLS even if DTLS is healthy

What about Idle timeout?

• When a DTLS−Tunnel is active, that is the only tunnel where idle timeout matters. Because very little control channel traffic passes over the SSL−Tunnel, it is almost always idle so it is exempt while there is an active DTLS−Tunnel. If something happened to UDP and the DTLS−Tunnel was torn down, then idle timeout would apply to the SSL−Tunnel

Configuration

DTLS is enabled by default but you can enable it or distable using CLI.

It can be enabled/disable per interface terminating AnyConnect VPN

webvpn

 enable if-name tls-only

Also, you can enable/disable DTLS at Group Policy level

webvpn

 dtls port 443

!

group-policy custom_group_policy attributes

 wins-server none

 dns-server value 10.170.7.99 10.170.7.100

 vpn-tunnel-protocol ssl-client ssl-clientless

 split-tunnel-policy tunnelspecified

 split-tunnel-network-list value sslvpn_split_tunnel

 default-domain value shelfdrilling.com

 split-dns none

 split-tunnel-all-dns enable

 webvpn

  anyconnect ssl dtls enable

  anyconnect mtu 1420

  anyconnect profiles value sslvpnfromrdpprofile type user

  customization value ShelfDrilling-Customization

  always-on-vpn profile-setting

The Importance of Understanding MTU value in AnyConnect VPN

Why do we need it?

During encryption, additional overhead will be added to the packets made by new headers and features. This means that the actual size of the unencrypted TCP segment or UDP datagram which holds the application will be reduced because the MTU of the adapter is still same.

For example with Ethernet and MTU of 1500-bytes, the unencrypted TCP segment can't be more than 1460-bytes. With encryption, for Ethernet and MTU of 1500, the unencrypted TCP segment can't be more 1380 (can be different value). The 80-bytes difference are utilized by encryption overhead.

Now the value of unencrypted TCP segment can be more which leads to MTU more than 1500-bytes but this will cause the networking devices to fragment the packet which is bad and should be avoided.

AnyConnect client builds Virtual Adapter (VA) during installation on the clients machine. This VA will receive unencrypted traffic and emulates Ethernet to forward traffic after encryption. The actual traffic then goes over the physical adapter.

Therefore, we need to know what is the MTU value of the VA and what is the max allowed size of unencrypted traffic to avoid fragmentation. Later the applications need to make sure that they don't create segments and datagrams larger else they will be fragmented.

How it works?

AnyConnect VA gets its MTU value from SSL Server (ASA or IOS. We will focus more on ASA). The default value is 1406-bytes. It can be configured as follow:

group-policy custom_group_policy attributes

 webvpn

  anyconnect mtu 1420

Now the actual MTU used by the VA will be selected based on the smaller between physical NIC MTU and VA configured MTU. This is to avoid scenarios where the VA has MTU configured more than physical NIC which will trigger fragmentation.

Next we need to find out the max value of unencrypted payload. Two values will be calculated, one for TLS Tunnel and one for DTLS tunnel. This can be viewed in ASA using the command debug webvpn anyconnect 1

……

……

Iphdr=20 base-mtu=1300 def-mtu=1500 conf-mtu=1420

tcp-mss = 1260

path-mtu = 1260(mss)

TLS Block size = 16, version = 0x301

mtu = 1260(path-mtu) - 0(opts) - 5(ssl) = 1255

mod-mtu = 1255(mtu) & 0xfff0(complement) = 1248

tls-mtu = 1248(mod-mtu) - 8(cstp) - 20(mac) - 1(pad) = 1219

DTLS Block size = 16

mtu = 1300(base-mtu) - 20(ip) - 8(udp) - 13(dtlshdr) - 16(dtlsiv) = 1243

mod-mtu = 1243(mtu) & 0xfff0(complement) = 1232

dtls-mtu = 1232(mod-mtu) - 1(cdtp) - 20(mac) - 1(pad) = 1210

computed tls-mtu=1219 dtls-mtu=1210 conf-mtu=1420

DTLS enabled for intf=2 (CORP)

tls-mtu=1219 dtls-mtu=1210

……

……

Let's examine the debugs

Iphdr=20 base-mtu=1300 def-mtu=1500 conf-mtu=1420

!!!... Iphdr is 20 bytes, Physical NIC MTU is 1300, configured MTU value for AnyConnect VA is 1420. Conclusion, Physical NIC MTU is used for VA.

!!!... Now will start TLS Tunnel calculations

tcp-mss = 1260

path-mtu = 1260(mss)

!!!... Since TLS is TCP based, the TLS payload size is MTU - 40. 40-bytes is 20-bytes IP Header + 20-bytes TCP Header

TLS Block size = 16, version = 0x301

mtu = 1260(path-mtu) - 0(opts) - 5(ssl) = 1255

mod-mtu = 1255(mtu) & 0xfff0(complement) = 1248

tls-mtu = 1248(mod-mtu) - 8(cstp) - 20(mac) - 1(pad) = 1219

!!!... Subtracting headers (5-bytes ssl header, 1-byte padding, 8-bytes Cisco SSL Tunneling Protocol (CSTP) header, 20-bytes MAC), we will get the size of unencrypted payload. This will be communicated back from ASA to AnyConnect client so that applications shouldn't cross this value else fragmentation will be triggered

!!!... Note: ANDing MSS value will complement 0xfff0 is to make sure that MSS value is power of 2

!!!... Now will start DTLS MTU calculations

DTLS Block size = 16

mtu = 1300(base-mtu) - 20(ip) - 8(udp) - 13(dtlshdr) - 16(dtlsiv) = 1243

mod-mtu = 1243(mtu) & 0xfff0(complement) = 1232

dtls-mtu = 1232(mod-mtu) - 1(cdtp) - 20(mac) - 1(pad) = 1210

!!!... Subtracting headers (20-bytes IP header, 8-bytes UDP header, 13-bytes DTLS header, 8- or 16- bytes for encryption [depending DES or AES], 1-byte Cisco DTLS Tunneling Protocol [CDTP] header, 20-bytes MAC, 1-byte pad), we will get the size of unencrypted payload. This will be communicated back from ASA to AnyConnect client so that applications shouldn't cross this value else fragmentation will be triggered

computed tls-mtu=1219 dtls-mtu=1210 conf-mtu=1420

DTLS enabled for intf=2 (CORP)

tls-mtu=1219 dtls-mtu=1210

Message Authentication Code (MAC)
A Message Authentication Code is a one-way hash computed from a message and some secret data.  It is difficult to forge without knowing the secret data.  Its purpose is to detect if the message has been altered.

Cisco AnyConnect VPN Certificate Matching with Windows

Today I faced strange behavior while deploying anyconnect VPN certificate authentication on windows 7/8. It looks to me as a bug but didn't see this listed in Cisco bug toolkit.

When you are using AnyConnect VPN certificate authentication, AnyConnect client will send the client certificate to the VPN server to verify the identity of the user (VPN server will authenticate the user certificate against CA Root certificate + Expiry + Revocation OCSP/CRL).

Anyconnect VPN Client will browse the all certificate stores (by default) on user's machine and will select the user certificate based on the following:

1. If no certificate matching criteria is specified in AnyConnect VPN Profile, AnyConnect applies the following certificate matching rules:

• Key Usage: Digital_Signature
• Extended Key Usage: Client Auth

2. If any criteria matching specifications are made in the profile, neither of these matching rules are applied unless they are specifically listed in the profile.

For reference:

 

Now, I had a client certificate which is having the KU and EKU as follow:

Also, I didn't define certificate matching criteria assuming that default of anyconnect client will pick the certificate based on my KU and EKU. This worked on Android, iOS, MAC OSX, Windows Server 2008, but not on windows 7/8 pro.

The workaround was to define certificate matching criteria as follow: