Cisco AnyConnect Client DNS issues

Since I started using the Cisco AnyConnect VPN Client, I had DNS issues. I could not fully figure out why this happens on my system, but all signs lead to an issue with scoped DNS queries on macOS. Whenever the VPN is established, macOS uses the wrong order to resolve DNS entries. This leads to DNS query timeouts, but under certain circumstances to complete DNS failures, as the system insists on using the unreachable DNS server.

The problem exists for a long time now, at least since macOS Mountain Lion. For a long time I had to live with this annoyance, as the issue persisted with every new version of the AnyConnect client. But now I found a workaround, maybe even a long-term solution. It is possible to tell the macOS resolver to use certain DNS servers for certain domains. Setting this up is rather easy:

[email protected]:~:$ sudo mkdir /etc/resolver
[email protected]:~:$ echo 'nameserver 10.8.0.1' > /etc/resolver/domain.com
[email protected]:~:$ sudo killall -9 mDNSResponder && dscacheutil -flushcache

After restarting the mDNSResponder daemon and flushing the DNS cache, I was all set. I hope this will become a long-term solution for this issue, that plagued me for so long.

HE & DigitalOcean – Blocked IPv6 ports

I recently created several Virtual Machines on my new Hetzner server as part of my migration efforts from DigitalOcean to my self-hosted ESXi hypervisor. I don’t migrate my VMs and services away from DigitalOcean because I am not satisfied with their service, quite the contrary in fact. I am doing it because my own self-hosted hypervisor allows more flexibility for my learning and testing efforts.

While double-checking the VMs for running services and open ports, I noticed that my Hurricane Electric IPv6 address has some filtered ports that I did not notice before:

Nmap from a HE IPv6 address

I looked at the Hurricane Electric FAQ and it explains why these ports are filtered and how to get these ports un-blocked:

Why can I not connect to IRC?
Due to a high and persistent amount of abuse, we’ve had to filter IRC access by default. If you need IRC access, complete the Sage level of the free IPv6 certification and then please send an email to [email protected] explaining your situation. Approvals will be handled on a case-by-case basis.

I can’t send email via IPv6. What’s wrong?
Due to a high and persistent amount of abuse, we had to filter SMTP (tcp/25) connections by default. If you’re not providing email service yourself, you should be able to use port 587 instead to your provider’s email server. If you are providing email services over your tunnel and need port 25 opened, please send an email to [email protected] explaining your situation. NOTE: this filtering does not affect the SMTP-related tests on the IPv6 certification program.

Pretty straight-forward and well-explained. Knowing that these ports are blocked, I wanted to use one of my IPv6-enabled DigitalOcean droplets as this should not filter anything and give me an unaltered list of open ports:

Nmap from a DigitalOcean IPv6 address

Man, I was so wrong! It looks like DigitalOcean is blocking ports as well, though their blocking made much less sense to me. I could not find anything in the help section on their homepage, but was able to find an article in their ‘Support Center’ that is accessible once you log in:

How to send emails with IPv6 enabled
We currently do not allow SMTP traffic over IPv6 as a side effect of how email black lists treat IPv6 addresses.

You can give priority to IPv4 addresses over IPv6 so that you can continue to send out email without disabling IPv6. You would do that by editing the Droplet’s /etc/gai.conf file and removing the comment (#) from the following line:

Default Configuration: #precedence ::ffff:0:0/96 100
Configuration with Priority to IPv4: precedence ::ffff:0:0/96 100

As mentioned in a comment on their “How To Enable IPv6 for DigitalOcean Droplets” site, there seems to be no way of getting rid of these restrictions. Even when you contact their support on this specific issue.

I can understand the blocking of IRC ports and port 25 for the reasons Hurricane Electric is giving in their FAQ. However I cannot understand the blocking of the ports 587, 465, 110, 995 and 143.

Lesson learned: Always use additional external tools — like MXTools — to verify your VM’s open ports, if your source IPv6 address is from DigitalOcean or Hurricane Electric!

HE IPv6 tunnel with Ubnt EdgeMax Lite router

IPv6 gets more and more important. I think 2016 will be the year when it get’s adopted on a large scale and finally will start to be mandatory with every new Internet connection — at least in Dual Stack configuration.

My web and DNS servers are available via IPv6 for a while now. There is not much demand (yet) for it, but it’s available for users who wan’t/can use it. Because IPv6 is currently not that widely spread, I hesitated for a long time to put up the effort to configure a proper IPv6 tunnel at home.

Well, I had some spare time on my hands and stumbled upon some interesting sites on the Internet. So let’s go!

Availability of IPv6

It looks like no company is providing native IPv6 access to the Internet — at least on the consumer level or for small business accounts. According to some online publications, Deutsche Telekom has already started their IPv6 rollout in their networking — including their cellular network in Germany. They have supposedly chosen to use Dual Stack1 for the transition period. I say “supposedly” because I use a different provider at home and my T-Mobile cell phone internet access seems to be IPv4 only, so I cannot verify it.

Dual stack where you can; tunnel where you must

There are several tunnel brokers available on the Internet. Sixxs disqualified itself years ago with an extraordinarily arrogant attitude that annoys the hell out of me, so I chose to use Hurricane Electric. This choice really was a no-brainer, as I also use their DNS services for quite some time now and never had any problems.

Let’s dig a tunnel

Firs things first, register with Hurricane Electric and create a tunnel. The creation wizard will show you all required information — like your IPv4 tunnel endpoints and your IPv6 subnet(s). The Ubiquiti EdgeMax Lite router that I use is a quite versatile and powerful little thing. Log in via SSH and start creating the tunnel:

configure
edit interfaces tunnel tun0 set encapsulation sit
set local-ip <Client IPv4 Address>
set remote-ip <Server IPv4 Address>
set address <Client IPv6 Address>
set description "HE.NET IPv6 Tunnel"
exit 
set protocols static interface-route6 ::/0 next-hop-interface tun0
commit
save

You now should be able to ping the server IPv6 address and your tunnel is established.

Configure interfaces and network address

Now configure your internal network. HE will route a /64 prefix to you, so choose an IP address from your new network and assign it to your internal network (usually the ::1 address). Also configure the router to advertise it’s network on the internal interface so that clients can automatically get an IPv6 IP address:

set interfaces ethernet eth1 address '<IPv6 router IP>'
set interfaces ethernet eth1 ipv6 dup-addr-detect-transmits 1
set interfaces ethernet eth1 ipv6 router-advert cur-hop-limit 64
set interfaces ethernet eth1 ipv6 router-advert link-mtu 0
set interfaces ethernet eth1 ipv6 router-advert managed-flag false
set interfaces ethernet eth1 ipv6 router-advert max-interval 300
set interfaces ethernet eth1 ipv6 router-advert other-config-flag false
set interfaces ethernet eth1 ipv6 router-advert prefix '2001:xx:xxxx:xxxx::/64' autonomous-flag true
set interfaces ethernet eth1 ipv6 router-advert prefix '2001:xx:xxxx:xxxx::/64' on-link-flag true
set interfaces ethernet eth1 ipv6 router-advert prefix '2001:xx:xxxx:xxxx::/64' valid-lifetime 2592000
set interfaces ethernet eth1 ipv6 router-advert reachable-time 0
set interfaces ethernet eth1 ipv6 router-advert retrans-timer 0
set interfaces ethernet eth1 ipv6 router-advert send-advert true

Change Default Ports For HTTP GUI and SSH

For security reasons change the service ports for the router’s web UI and SSH access to something else, as your IPv6 IP is reachable from the Internet now! I think this will be the most important thing to think about during the transition to IPv6 — no “security” by IPv4 NAT anymore. Everything is accessible on the Internet — if not properly shielded by a firewall on the router or the asset itself!

set service ssh port 8022
set service gui https-port 8443

Common Useful Address Groups

Set up some groups to make handling of these special address ranges easier in the future:

set firewall group address-group Private-RFC-Ranges description 'RFC 1918 Private Ranges'
set firewall group address-group Private-RFC-Ranges address 10.0.0.0/8
set firewall group address-group Private-RFC-Ranges address 172.16.0.0/12
set firewall group address-group Private-RFC-Ranges address 192.168.0.0/16
set firewall group ipv6-address-group IPv6-FE80 description 'fe80::/10 (aka Link-Local) Network'
set firewall group ipv6-address-group IPv6-FE80 ipv6-network 'fe80::/10'

Enable Hardware Offloading

The EdgeMax Lite router offers hardware offloading, so let’s use it!

set system offload ipsec enable
set system offload ipv4 forwarding enable
set system offload ipv4 vlan enable
set system offload ipv6 forwarding enable
set system offload ipv6 vlan enable

If you are using DSL, it might also be a good idea to enable this

set system offload ipv4 pppoe enable
set system offload ipv6 pppoe disable
Optional: Configure MSS Clamping on DSL connections

Some sites on the Internet have problems with PMTU discovery, so it is best to clamp the MSS manually. For PPPoE users, this command will ‘fix’ connectivity to remote sites where ICMP is blocked, and PMTU is broken:

set firewall options mss-clamp interface-type all
set firewall options mss-clamp mss 1452
set firewall options mss-clamp6 interface-type all
set firewall options mss-clamp6 mss 1412

Configuring the firewall

Last but not least, define a set of firewall rules to make sure you are safe and sound:

set firewall ipv6-name HE-To-LAN default-action drop
set firewall ipv6-name HE-to-LAN description 'HE to LAN'
set firewall ipv6-name HE-to-LAN rule 1 action accept
set firewall ipv6-name HE-to-LAN rule 1 description 'Drop non-related incoming IPv6'
set firewall ipv6-name HE-to-LAN rule 1 state established enable
set firewall ipv6-name HE-to-LAN rule 1 state related enable
set firewall ipv6-name HE-to-LAN rule 2 action drop
set firewall ipv6-name HE-to-LAN rule 2 state invalid enable

set firewall ipv6-name LAN-to-HE default-action accept
set firewall ipv6-name LAN-to-HE description 'LAN to HE'
set firewall ipv6-name LAN-to-HE rule 1 action accept
set firewall ipv6-name LAN-to-HE rule 1 state established enable
set firewall ipv6-name LAN-to-HE rule 1 state related enable
set firewall ipv6-name LAN-to-HE rule 2 action drop
set firewall ipv6-name LAN-to-HE rule 2 state invalid enable

The first rule-set will drop all incoming IPv6 traffic unless it is related, while the second rule-set will by default accept all traffic. Now we have to assign these rule-sets to the interfaces we have.

Traffic to the router and your IPv6 network

Assign the HE-to-LAN ruleset for forwarded packets on the inbound tunnel interface and and for packets destined for the router itself:

set interfaces tunnel tun0 firewall in ipv6-name HE-to-LAN
set interfaces tunnel tun0 firewall local ipv6-name HE-to-LAN
Traffic from the LAN to the IPv6 internet

This rule-set will allow traffic from the inside to the Internet via IPv6:

set interfaces ethernet eth1 firewall in ipv6-name LAN-to-HE

Congratulations. You are now able to use IPv6! To make sure everything went alright, test your firewall and your general IPv6 connectivity and work towards your 20/20 rating!


1. Internet access providers that use “DS-Lite” are only offering crippled access to the internet in my opinion. This kind of access is “broken by design” as they use a Large Scale NAT (LSN) for IPv4. [↩](#fnref:1 “return to article”)

EDNS UDP packet size

A couple of weeks ago I set up a local BIND in a CentOS 6.5 VM to have an internal DNS server for my VMs to use. After creating several local zone files and successful initial tests, everything worked fine. Some domains had high query times, and SSH logins sometimes took a bit longer than expected, but I did not have enough time to investigate further.

Today I had some free time on my hands and decided to re-visit the issue. The DNS server works fine, but whenever the cache is empty, it takes up to 3098ms for a DNS query. Once the result is in cache, everything works as expected. To get a better overview I started with enabling debug logging in named.conf:

logging {
    channel default_debug {
        file "data/named.run";
        severity debug;
        print-time yes;
        print-severity yes;
        print-category yes;
    };
    category default {
        default_debug;
    };
};

After a restart of the DNS service I tested several domains and found this suspicious log entry:

05-Oct-2014 11:24:28.014 edns-disabled: debug 1: success resolving 'domain.com.dlv.isc.org/DLV' (in 'isc.org'?) after reducing the advertised EDNS UDP packet size to 512 octets

I then remembered something a friend of mine told me a couple of months ago. Hetzner is using several Firewall and anti-DDOS techniques to prevent attacks. One of these techniques is blocking UDP packets greater a specific size (which is not publicly revealed).

I found a knowledgebase article and a way to test if this is a problem with my local DNS server, or the one remote. It looks like the Hetzner is blocking UDP packages which are greater than 1440 or sometimes 2200 bytes, sometimes I get even lower values. Pretty inconsistent results.

To be on the safe side, I edited the named.conf and set the values for “edns-udp-size” and “max-udp-size” to 512 bytes:

edns-udp-size           512;
max-udp-size            512;

This seems to work for me now. I also contacted Hetzner and applied for the UDP whitelist.

Connection lost

Four weeks ago my Ubiquiti EdgeMax EdgeRouter Lite decided it was a good idea to not start PHP anymore. Some people would say ‘Good choice and good riddance’ — on a router appliance a web interface comes in handy once in a while though. A Ubnt community member was able to help me very quickly, though something was very fishy. I just recently upgraded the firmware on the router, so how could the file system got corrupted? Did something strange happen during the upgrade? — Not that I could remember.

One evening two weeks ago then I was debugging a VPN problem I encountered a massive packet loss. First I thought it was the remote server, or my Internet provider. Soon though I had to realize that the first hop, the little EdgeRouter, was losing 70% of all packages sent to it. Also 60% packet loss on the second hop. No web interface available, not SSH connection possible. I decided to reboot the device… it did not come back! My first thought was a memory issue, but I needed more information what was happening on the device. The lights were no help, everything looked normal. I connected a Cisco Serial Console Cable and found some kernel “Oops” during boot:

This really looked weird. I contacted Ubnt support. I was told to press the famous ‘any key’ during boot, before the kernel was loaded, and then run some tests. First of all a memory test:

Octeon ubnt_e100# mtest
Testing 80100000 ... 80ffffff:
Iteration: 6040

It ran for a couple of hours, but showed no issue at all. Support then had the idea to use the ‘EdgeMax Rescue kit’ to re-install the router from scratch.

This unfortunately did not bring a solution either. But it helped to identify the underlying problem. It looks like the filesystem corruption a few weeks back was just a herald of the problems that would I have to face soon:

It looks like the flash memory is totally fucked up. Since it was a little over a year old, and therefore still covered by the EU wide 2 year warranty, I contacted my reseller and filed for RMA and sent the router back to them. I really hope that I will get a new device soon. It is a quite powerful little device, so I really miss it.