2018-12-21
___t_h_e__m_o_s_t__e_m_b_a_r_r_a_s_s_i_n_g__a_d_m_i_n__m_i_s_t_a_k_e_

Today I have to share a rather embarrassing admin story that has come
to a happy end yesterday. I started running a small mailserver for my
favourite human 2 years ago. I have been using an OpenBSD VM with
opensmtpd and dovecot which I have documented in my old blog[0].

Initially I have used digital ocean as my hoster. I have written a
small shell script that periodically try to ping the host and run
netcat on the smtpd port to check that it is still up and running.

However at one point digital ocean hosed my VM image, so I migrated
it from a backup over to vultr.com.

All is fine until the human started mentioning strange issues with
mails.  Some would not get delivered or be delayed for some *days*.
The sympthoms would always be the same: E-Mail is entered in a web
shop's order form, mails would not arrive and no connection attempts
were visible in the logs.

My test script didn't show any problem, the server was always
reachable.  This has changed.

Yesterday a mail I have send would bounce and my script would
complain.  While checking into the issue I have noticed that the
script cecks only with:

nc mail.example.com 25 | head -n1 | grep  grep -q ^220.*OpenSMTPD

So this would check whatever the nameserver resolves it to.  BUT this
server has ipv4 and ipv6 connectivity and corresponding MX entries in
its DNS records. So I have fixed the script to explicitly check for
BOTH address families.

And lo and behold it showed an error. I could not connect to the ipv4
address to reach the smtpd!

I have found out, that vultr.com blocks port 25 unless you ask
support to lift the ban due to spam! This has been in there for OVER
A YEAR without me noticing it. So I asked support to lift the ban
which they did eventually after I declined to provide a photocopy of
my national ID (which is illegal unless explicitly required by law
here). Still no connectivity.

I wondered. Then I remembered having set up gray listing on the host
for v4 while on the digital ocean VM, I have since disabled spamd on
this VM but forgot about the pf filter rule redirecting traffic. So
after fixing that I finally got to receive mail via ipv4.

So my lesson from this: Tests matter, do them properly so you know
when something does not work. Also ipv6 is doing pretty good in the
realms of email I would have expected more problems than the
occasionally missing email. Also don't forget about your firewall
settings. Check everything with a ports scanner.

That's all folks. This is my most embarrassing admin story of the
year.

___References________________________________________________________

[0]: http://pestilenz.org/~ckeen/blog/posts/opensmtpd.html