As more and more areas in the UK are being covered on a daily by FTTC (fiber-to-the-cabinet) providing very good speeds up to 78Mbit doenstream and 20Mbit Upstream with a very attractive price, I'm seeing a challenge with the current set up BT and all others ISPs are having. As you need the OpenReach white VDSL modem and then another router to actually do the routing between your LAN (you can use a VDSL router but again, I require a proper firewall), you always end up with an annoying set up of having 2 boxes, sometimes even 3 if you use a separated firewall appliance.
I always favour practicality and tidiness, so the least devices I can use the better to maintain, manage and of course, less clutter of cables.
One good solution for this is to use pfSense firewall to authenticate the PPPOE connection, route and be the firewall. Also all of this can be done with pfSense installed as a virtual appliance. I have for example at home pfSense running on VMware workstation as a service so the VM start and shutdown with the host. I have a media centre on 24/7 so for me it is perfect. Of course you can install pfSense on a old PC, nettop or even on some thinclient like the NEOware E14 (limited firewall features though)
My problem was that with the PPPOE set up, the ISP assigns automatically the IP address to the firewall and in my case as we have a block of IP addresses /29 I wanted to be able to use all of them. Normally you would do this simply by going to Firewall>Virtual IPs>Add IP Alias. This works fine when you use static IP address on your WAN like when you have a ADSL router and it has one of the public IP addresses on it, and forward everything else ahead so you can simply set up the Virtual IPs on pfSense and voila, all done. For FTTC as we use PPPOE this changes a alot as the WAN interface gets the IP address from the ISP automatically, and if you set the IP address as Virtual IP, nothing happens.
To troubleshoot this I had to check what was happening, are the packets arriving on the WAN interface? Are they being dropped? Are the IPs actually showing up on the WAN interface? I contacted my ISP (zen.co.uk) just to confirm the block of IPs /29 is available, they confirmed and then I had to run some packet analysis on my side to understand why it was still not working.
Running the command ifconfig -a will show all the IP addresses assigned to all interfaces and you will notice that when authanticatin using PPPOE the other virtual IP addresses allocated to the WAN interface simply don't show up.
After this I decided to check if traffic to the other IP addresses were actually being routed to my firewall by my ISP on the /29 network. I used the Packet capture feature on the same diagnostics tab from pfSense and enabled only ICMP capture, then from outside the network I run a ping against every single IP address I have on the range and by my surprise, I got them on pfSense as you can see: (IPs hidden for security purposes)
14:55:21.737881 IP 92.x.x.x > 88..x.x.x: ICMP echo request, id 35801, seq 28, length 40
14:55:21.737948 IP 92.x.x.x > 88..x.x.x: ICMP echo request, id 35801, seq 28, length 40
14:55:26.545222 IP 92.x.x.x > 88..x.x.x: ICMP echo request, id 34275, seq 29, length 40
So pings are received but simply dropped. The firewall doesn't have the configuration to route this information anywhere as we have seen with ifconfig-a, no IP address besides the one assigned through PPPOE is assigned to any interface.
I tried assigning the Virtual IPs then instead of WAN, to the localhost interface.
For my surprise, after testing this all IP addresses were working just fine and I manage to ping them from outside the network, establish IPSEC tunnels as if they were being assigned to the WAN interface, exactly like that.
It is weird, but it works and everything else works like it should, all IPs added as Virtual IPs.
Tricky to find out why, but at least I got it working. I hope this help for all FTTC ISPs in the UK using PPPOE authentication with several static IPs allocated for you :)
