Chapter 8. IPsec

Table of Contents

8.1. Preface
8.1.1. Features
8.1.2. Site to Site VPN Explained
8.1.3. Remote Access IPsec VPN
8.1.4. Tunnel Mode
8.1.5. Perfect Forward Secrecy
8.1.6. IPsec Software Clients
8.2. Special Features
8.2.1. Dead Peer Detection
8.2.2. Dynamic DNS Support
8.2.3. NAT Traversal
8.2.4. IPsec Traffic Filtering
8.3. Prerequisites
8.4. Configuring the VPN Tunnel
8.5. Possible Issues
8.5.1. What if your m0n0wall is not the main Internet Firewall?
8.5.2. Additional Questions
8.6. Quick Start for RSA Signature Authentication

This chapter will go over configuring a site to site Virtual Private Network (VPN) links between two m0n0walls, discuss how to configure site to site links with third party IPsec-compliant devices and discuss VPN to remote IPSec client software. Once you have IPSec properly configured you will be able to take advantage of the following capabilities:

The Example VPN Configurations chapter goes over, in detail, how to configure site to site IPsec links with some third party IPsec devices. Although it might seem confusing, in most cases you just need to assure that all of the parameters match on both sides (except of course the definition of who is the remote network). Some routing issues might come up depending on your situation but reading the rest of this chapter should be enough to successfully use IPsec encryption.

If you have gotten m0n0wall working in a site to site IPsec configuration with some third party IPsec device that is not already listed, we would appreciate if you could put together a short write up of how you got it configured, preferably with screenshots where applicable.

8.1. Preface

IPsec (IP security) is an international standard for providing security to IP protocols via encryption and/or authentication, typically employing both. Its use in m0n0wall is for Virtual Private Networks (VPN's). After two or more points securely authenticate each other's identification, access rights, and how to encrypt data (phase 1), they will be able to communicate using encrypted data packets (phase 2). The two points can be on a local network, a wireless network or even on the Internet.

There are two general types of IPsec VPN capabilities in m0n0wall, site to site and remote access. Site to site will connect entire networks while remote access allows mobile users access to secured networks.

8.1.1. Features

The IPsec specification includes many features and services. Below is a list of IPsec features, including features not currently supported by selected m0n0wall versions.

Table 8.1. IPSec Feature List

Feature   1.2 1.3
Site to site   x x
Mobile user to site   x x
Tunnel mode   x x
Transport mode      
Perfect Forward Security (PFS)   x x
Main Mode   x x
Aggressive Mode   x x
Remote gateway hostname/domain support     x
IKEv2 support      
Phase 1 local IP, Domain, FQDN Identifier   x x
Phase 1 local RSA Cert Subject Identifier     x
Phase 1 Authentication Hashes md5, sha1 support   x x
Phase 1 Authentication Hashes tiger192, ripemd160 support      
Phase 1 Authentication Preshared Key support   x x
Phase 1 Authentication RSA / PKI X.509 Certificate support   x x
Phase 1 Authentication DSA Certificate support      
XAUTH Authentication      
Phase 2 Diffie-Hellman Key support 768, 1024, 1536 bit (also Modp)   x x
Phase 2 Diffie-Hellman Key support 2048, 3072, 4096 bit (also Modp)      
Encryption Ciphers DES,3DES, Blowfish, CAST128   x x
Encryption Cipher AES (Rijndael)     x
Encryption Ciphers Twofish, Serpent, IDEA      
NAT-T Traversal     x
Dead Peer Detection     x
IPSec diagnostic logs   x x
Dynamic DNS remote site support     x
IPSec Traffic filtering      
DHCP over IPSec      
L2TP Authentication      
Manual Key support      
Certificate Revocation List      

8.1.2. Site to Site VPN Explained

Site to site VPN's connect two locations with static public IP addresses and allow traffic to be routed between the two networks. This is most commonly used to connect an organization's branch offices back to its main office, so branch users can access network resources in the main office. Prior to VPN's, much more expensive private Wide Area Network (WAN) links like frame relay, point to point T1 lines, etc. were commonly used for this functionality. Some organizations are moving towards VPN links between sites to take advantage of reduced costs.

Site to site VPN's can also be used to link your home network to a friend's home network, to provide access to each other's network resources without opening holes in your firewalls.

While site to site VPN's are a good solution in many cases, private WAN links also have their benefits. IPsec adds processing overhead, and the Internet has far greater latency than a private network, so VPN connections are typically slower (while maybe not throughput-wise, they at least have much higher latency). A point to point T1 typically has latency of around 4-8 ms, while a typical VPN connection will be 30-80+ ms depending on the number of hops on the Internet between the two VPN endpoints.


When deploying VPN's, you should stay with the same ISP for all sites if possible, or at a minimum, stay with ISP's that use the same backbone provider. Geographic proximity usually has no relation to Internet proximity. A server in the same city as you but on a different Internet-backbone provider could be as far away from you in Internet distance (hops) as a server on the other side of the continent. This difference in Internet proximity can make the difference between a VPN with 30 ms latency and one with 80+ ms latency.

8.1.3. Remote Access IPsec VPN

m0n0wall provides two means of remote access VPN, PPTP and IPsec (with OpenVPN available in beta versions only for now). m0n0wall's mobile IPsec functionality has some serious limitations that hinder its practicality for many deployments. m0n0wall version 1.2 does not support NAT-Traversal (NAT-T) for IPsec, which means if any of your client machines are behind NAT, IPsec VPN will not work. This alone eliminates it as a possibility for most environments, since remote users will almost always need access from behind NAT. Many home networks use a NAT router of some sort, as do most hot spot locations, hotel networks, etc.


NAT-T is supported in m0n0wall version 1.3 beta.

One good use of the m0n0wall IPsec client VPN capabilities is to secure all traffic sent by hosts on a wireless network or other untrusted network. This will be described later in this chapter.

FIXME - A second limitation is the lack of any really good, free IPsec VPN clients for Windows. Most of your remote users will likely be Windows laptop users, so this is another major hindrance.

For most situations, PPTP is probably the best remote access VPN option in m0n0wall right now. See the PPTP chapter for more information.

8.1.4. Tunnel Mode

IPsec's Tunnel mode is supported on m0n0wall devices. This mode allows secured communication between entire subnets. When the packet leavs the subnet it will be encrypted, when it gets to the remote IPSec device the packets are decrypted and routed/ sent into the remote network.

The IPsec Specification supports a 2nd mode of operation called Transport mode. Transport mode limits encrypted communication to the 2 devices that are encrypting the information. If this was supported it would only allow secured communication to the m0n0wall device itself and not to its connected networks. Transport mode is not supported.

8.1.5. Perfect Forward Secrecy

This option increases security during authentication by assuring that new keys (which are generated on a regular basis to ensure security) are not based on previous keys. When activated, this means that if someone obtains or discovers 1 encryption key that they cannot use it to discover previous or future keys. This can be disabled to allow faster key negotiation.

8.1.6. IPsec Software Clients

Most operating systems include IPsec clients. Windows 2000 and above includes a free IPsec client but it is also difficult to configure. MacOSX 10.3 and later also includes a free IPsec client but the free configuration tool is for a special version of IPsec called L2TP/IPsec. Free configuration tools exist for both operating systems but commercial solutions, at least for Windows, are more evolved and easier to use than the built-in free version.


m0n0wall does not support L2TP so if your IPsec client software only supports L2TP it will not work with m0n0wall. However, for adventure seekers, there is a how to for using IPsec on a device and L2TP on an internal Windows 200x server to offset the encryption workload: This has not been tested yet with m0n0wall devices.

Below is a list of IPsec software clients.


In some versions of Microsoft Windows, you must deactivate the built-in IPsec client before installing a commercial 3rd party IPsec client. Be sure to read the installation instructions.