Problems with PPTP and pfSense running as a virtual machine on VMware ESXi

Before you throw some stones saying why the hell are you using PPTP knowing it is legacy and easy to hack, think about Windows XP. It has been around for more than 10 years, it is full of malware around for it and there are applications that simply don't run any more on it, but several companies still have it. I was flying to Sydney last month from London and noticed that the airport in Sydney has several terminal running Windows XP. It is not just about what we want to tell our customers what is best or not, it is all down to costs. Some of my clients have firewalls that are a bit dated, some others want to connect to their network with their WIndows client without any need to 3rd party software like OpenVPN. Some I managed to get around using L2TP, but not always.

pfSense has a built in PPTP server which works fine when needed, also pass forward PPTP traffic just fine and it has worked like this for a long time, until I upgraded my clients to ESXi 5, then I started to have all sort of problems. I spent several months with pfSense support trying to get this sorted, not just about PPTP port forwarding or PPTP server itself on pfSense, but also dialing out PPTP connections behind pfSense.

Not a single machine behind pfSense that was running as a virtual machine on ESXi managed to dial out PPTP connections or be able to host PPTP servers, but this took me a whiloe to figure out as I had a mix of ESXi 4, 5 and 5.5 around. Even at my home server pfSense run virtualised, but inside VMware workstation running on top of a Windows server. It runs just fine inside VMware Workstation, so I assumed that it was not a problem with virtualisation. Going forward with pfSense support, we did many tests, captured loads of traffic and did everything that was available to troubleshoot the problem. In the end I was nearly giving up but then found some hints on forums regarding problems with the E1000 network drivers for freebsd and ESXi, plus some people saying the latest build of ESXi 5.5 sorted out some problems with PPTP connections.

I tested the VMXNET3 drivers for freebsd and for my surprise, indeed PPTP worked like a charm once again, but I didn't want to get into replacing the VMware tools manually across all the pfSense instances I have. It was better to update all hosts to the latest version of ESXi 5.5, build 1623387. I just went to https://www.vmware.com/patchmgr/findPatch.portal

The zip file you download can be uploaded to your ESXi host using WinSCP to the datastore, and then you can use putty to SSH into the ESXi host and run the update with this command:

esxcli software vib update -d /tmp/esxiupdate.zip (change to the location where you dumped the file)

You need to reboot the server afterwards, and hopefully pfSense or any other firewall that uses PPTP or even PPTP port forwarding will work fine. It worked for me.

vSphere Client requires Windows XP SP2 or later. vSphere Client cannot be installed on a domain controller.

A couple of days ago I came across something quite annoying after one of my clients had their server putting the last foot into the grave. I converted the physical machine into a VM, arranged another host and moved it over in a hurry, all worked fine and then remotely I tried to install vSphere client on this old converted Windows 2003 Domain Controller so I could manage the host as the ipsec tunnel for some reason was not passing all traffic properly.

Anyway, I get this message during installation of vSphere client 5.5 "vSphere Client requires Windows XP SP2 or later.  vSphere Client cannot be installed on a domain controller." which was really annoying. I could not deploy another VM quickly enough as I had to do some work for this client right quick. Interesting enough the installation program of vSphere client has some switches you can use and browsing through the help I managed to find this one here:

/VSKIP_OS_CHECKS="1"

So, running the installation program with this switch basically make it ignore that the VM is a domain controller and the installation go through just fine. It is not recommended or supported by VMware by for me it was needed and I didn't have much of a choice, maybe the same situation you may get into the future especially when dealing with legacy systems (screw accounts software that is 8 years old)

How to use pfSense with BT Infinity FTTC and other ISPs PPPOE and static range of IP addresses

As more and more areas in the UK are being covered on a daily by FTTC (fiber-to-the-cabinet) providing very good speeds up to 78Mbit doenstream and 20Mbit Upstream with a very attractive price, I'm seeing a challenge with the current set up BT and all others ISPs are having. As you need the OpenReach white VDSL modem and then another router to actually do the routing between your LAN (you can use a VDSL router but again, I require a proper firewall), you always end up with an annoying set up of having 2 boxes, sometimes even 3 if you use a separated firewall appliance.

I always favour practicality and tidiness, so the least devices I can use the better to maintain, manage and of course, less clutter of cables.

One good solution for this is to use pfSense firewall to authenticate the PPPOE connection, route and be the firewall. Also all of this can be done with pfSense installed as a virtual appliance. I have for example at home pfSense running on VMware workstation as a service so the VM start and shutdown with the host. I have a media centre on 24/7 so for me it is perfect. Of course you can install pfSense on a old PC, nettop or even on some thinclient like the NEOware E14 (limited firewall features though)

My problem was that with the PPPOE set up, the ISP assigns automatically the IP address to the firewall and in my case as we have a block of IP addresses /29 I wanted to be able to use all of them. Normally you would do this simply by going to Firewall>Virtual IPs>Add IP Alias. This works fine when you use static IP address on your WAN like when you have a ADSL router and it has one of the public IP addresses on it, and forward everything else ahead so you can simply set up the Virtual IPs on pfSense and voila, all done. For FTTC as we use PPPOE this changes a alot as the WAN interface gets the IP address from the ISP automatically, and if you set the IP address as Virtual IP, nothing happens.

To troubleshoot this I had to check what was happening, are the packets arriving on the WAN interface? Are they being dropped? Are the IPs actually showing up on the WAN interface? I contacted my ISP (zen.co.uk) just to confirm the block of IPs /29 is available, they confirmed and then I had to run some packet analysis on my side to understand why it was still not working.

Running the command ifconfig -a will show all the IP addresses assigned to all interfaces and you will notice that when authanticatin using PPPOE the other virtual IP addresses allocated to the WAN interface simply don't show up. 

After this I decided to check if traffic to the other IP addresses were actually being routed to my firewall by my ISP on the /29 network. I used the Packet capture feature on the same diagnostics tab from pfSense and enabled only ICMP capture, then from outside the network I run a ping against every single IP address I have on the range and by my surprise, I got them on pfSense as you can see: (IPs hidden for security purposes)

14:55:21.737881 IP 92.x.x.x > 88..x.x.x: ICMP echo request, id 35801, seq 28, length 40

14:55:21.737948 IP 92.x.x.x > 88..x.x.x: ICMP echo request, id 35801, seq 28, length 40
14:55:26.545222 IP 92.x.x.x > 88..x.x.x: ICMP echo request, id 34275, seq 29, length 40

So pings are received but simply dropped. The firewall doesn't have the configuration to route this information anywhere as we have seen with ifconfig-a, no IP address besides the one assigned through PPPOE is assigned to any interface.

I tried assigning the Virtual IPs then instead of WAN, to the localhost interface.

For my surprise, after testing this all IP addresses were working just fine and I manage to ping them from outside the network, establish IPSEC tunnels as if they were being assigned to the WAN interface, exactly like that.

It is weird, but it works and everything else works like it should, all IPs added as Virtual IPs.

Tricky to find out why, but at least I got it working. I hope this help for all FTTC ISPs in the UK using PPPOE authentication with several static IPs allocated for you :)