Why you should use iptables along with Docker

The command iptables is the historic userland access to Linux networking stack. It allows so set rules for packets filtering and to build firewalls. To route network traffic from hosts to container withing a network bridge, Docker users iptables to insert its rules. This article provides iptables basics to understand what is going on, and solve issues you can have.

Iptables is weird, this is why ufw exists

When ufw leads to more problems than solutions

In my current job, we are using Keycloak for identity management. Keycloak use an internal LDAP to check users' passwords. Keycloak and the LDAP are deployed on the same host as Docker containers along with a HAProxy to handle incoming traffic. To access LDAP, Keycloak use a private domain name that points to the HAProxy which routes the requests to the LDAP.

Without any firewall rule, Keycloak connects to the LDAP.

With ufw enabled, it does not. I see blocked connections from docker bridge toward the host using LDAP port in /var/log/ufw.log. My attempt to setup ufw to allow the trafic failed. I decided to give up and directly use iptables instead.

What is iptables and how to use it

Iptables and ip6tables are userland commands set up, maintain, and inspect the tables of IPv4 and IPv6 packet filter rules in the Linux kernel within a framework called Netfilter. That's the man page. In shorter form: it allows to configure a firewall.

Iptables consists in tables that correspond to what we want to do with ip traffic. The interesting tables when you setup a firewall and work with docker are the filter table and the nat table. Each tables contains rules organized in chains. The filter table contains chains INPUT to allow or deny incoming traffic, OUTPUT to allow or deny outgoing traffic and FORWARD to allow or deny forwarded traffic, there the server act as an intermediate. The important chains in the nat table are PREROUTING POSTROUTING.

You can create subchain of rules within a chain. This is what tools like Docker and ufw use to interact with iptables.

In each chains, each rule's condition is tested. If a packet match the condition, the rule is applied. It can be a non terminating rule (like LOG to append a log line) or a terminating rule to stop testing. ACCEPT accept the packet, DROP reject the packat and RETURN stops current chain or subchain execution.

You can append, insert and delete rules using commands iptables -A, iptables -I or iptables -D. If you want to block any network access to your server, just type:

iptables -I INPUT 1 -j DROP

Thank me later.

Use logging to help you to build your rule set

When you are debugging your firewall rules, it's a good idea to log blocked traffic in /var/log/syslog. To do so, append a LOG action before the DROP.

  iptables -A INPUT -j LOG --log-prefix "Some log prefix "
  

What is Docker doing with iptables

When you create a Docker network and a container within this network, Docker creates two sets of rules:

• PREROUTING and POSTROUTING rules in the nat table
• FORWARD rules in the filter table

sudo iptables -t nat -L -v 
sudo iptables -L -v 

Option -v is important here as it will show you which network interfaces are concerned by the rules. Docker creates one interface for each network. They are named docker0 for default bridge dans are prefixed by br for user created networks.

Here is an example of PREROUTING rule set that Docker creates:

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere
    0     0 RETURN     all  --  br-something any     anywhere             anywhere
  14M  845M DNAT       tcp  --  !br-something any     anywhere             server-name       tcp dpt:http-alt to:10.10.11.2:8080 

And the associated POSTROUTING rule set:

 Chain POSTROUTING (policy ACCEPT 14M packets, 848M bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  any    !docker0  172.17.0.0/16        anywhere
  41M 2445M MASQUERADE  all  --  any    !br-something  10.10.11.0/24        anywhere
    0     0 MASQUERADE  tcp  --  any    any     10.10.11.2           10.10.11.2           tcp dpt:http-alt 

What happen then?

As you can see on this netfilter flowchart, PREROUTING matches are used to make a routing decision. Packets pass by INPUT and OUTPUT rules OR by FORWARD and POSTROUTING rules. It means that when you create a network and containers with published ports in this network, the traffic that goes to the containers is not tested through INPUT and OUTPUT rules in the filter table.

By the way, it's a good idea to precise the IP address when you expose your container ports, as by default it will setup your netfilter rules to accept the traffic from everywhere to any IP address (0.0.0.0).

So, what happened to me?

If you decrypt the nat PREROUTING rules, you'll see that the traffic that is not coming from interface br-something get NATed. Unfortunately, when Keycloak calls the LDAP, the requests are seen as coming from interface br-something, do not trigger DNAT rule and get blocked by INPUT rules from the filter table.

In spite of my efforts, I have not been able to configure ufw to allow this traffic. I directly used iptables instead, and resulting INPUT rule set is straightforward:

iptables -F INPUT
iptables -A INPUT -m conntrack --ctstate ESTABLISHED -j ACCEPT
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp -i br-something --dport 389 -j ACCEPT
iptables -A INPUT -p tcp -i lo --dport 53 -j ACCEPT
iptables -A INPUT -p udp -i lo --dport 53 -j ACCEPT
iptables -A INPUT -j DROP 

It flushes the filter table, accept already established connections, accept ssh and ping and traffic from Docker network interface to port 389 (default LDAP port). Default policy is to accept all, so I won't be kept outside of my server in case of reboot.

Iptables is for boomer anyway, it has been replaced by nft

Now I know iptables and it feels great! However, I also learned that the utility is deprecated and that nft should now be used to interact with netfilter.

I have been thinking that iptables was crytptic, but it was because I did not know nft complicated syntax.

I guess I still have a lot to learn!