Observe the following three iptables commands:

# iptables -A INPUT -s 10.20.0.0/24 -d 10.10.0.0/24 -i eth0 -m policy -p esp --dir in --pol ipsec --reqid 1 -j ACCEPT
# iptables -A INPUT -s 10.20.0.0/24 -d 10.10.0.0/24 -i eth0 -m policy --proto esp --dir in --pol ipsec --reqid 1 -j ACCEPT
# iptables -A INPUT -s 10.20.0.0/24 -d 10.10.0.0/24 -i eth0 -m policy --protocol esp --dir in --pol ipsec --reqid 1 -j ACCEPT

Whereas --proto is a valid synonym for --protocol (as are -p, --proto[c[o[l]]]), if it, namely the exact token --proto, appears in the rule after -m policy, it will be appropriated by the policy extension of iptables. Indeed, the resulting rules as reported by iptables -vL are:

# iptables -vL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     esp  --  eth0   any     10.20.0.0/24         10.10.0.0/24         policy match dir in pol ipsec reqid 1
    0     0 ACCEPT     all  --  eth0   any     10.20.0.0/24         10.10.0.0/24         policy match dir in pol ipsec reqid 1 proto esp
    0     0 ACCEPT     esp  --  eth0   any     10.20.0.0/24         10.10.0.0/24         policy match dir in pol ipsec reqid 1

This token overloading is an unfortunate design conflict, and such subtlety confounded SaltStack's iptables state module, though not its execution module. It's also really confusing unless you know that --proto can be used in two distinct ways in two distinct places.

So, according to the iptables(8) and iptables-extensions(8) man pages, -p, --proto[c[o[l]]] specify "The protocol of the rule or of the packet to check." and --proto as used by the IPSec policy extension matches the encapsulation protocol. Evidently the encapsulation protocol is different from the protocol used for IPSec traffic. I still have more to learn about this, otherwise I would be more helpful here.