Zimbra 451 4.3.5 Server Configuration Problem: 3 Easy Fixes

Zimbra 451 4.3.5 "Server configuration problem" is a temporary SMTP rejection that Postfix returns while a mail is being received. The sender is told to try again later, and legitimate mail is delayed. On a busy Zimbra MTA the cause I found was the policy daemon cbpolicyd that could not answer all requests in time. This guide shows the log line, why Postfix prints exactly this text, and the settings in cbpolicyd.conf.in that fixed it. The old version of this post started with a stock illustration; it has been replaced by the real log line.

Contents

• The Zimbra 451 4.3.5 error in zimbra.log
• Why Zimbra 451 happens: the policy service
• Step 1: check that cbpolicyd runs
• How to measure the Zimbra 451 problem
• Step 2: give the server enough resources
• Step 3: raise the cbpolicyd worker limits
• Step 4: restart cbpolicyd and watch the log
• What not to do
• FAQ
• Is Zimbra 451 4.3.5 a configuration mistake?
• Do I lose mail with a 451?
• What else should I read?

The Zimbra 451 4.3.5 error in zimbra.log

The error appears in /var/log/zimbra.log on the MTA, at the RCPT step of the SMTP dialogue (addresses are replaced by examples here):

Jan 22 11:31:36 mta postfix/smtpd[9984]: NOQUEUE: reject: RCPT from unknown[198.54.114.232]: 451 4.3.5 Server configuration problem; from=<sender@example.com> to=<user@example.net> proto=ESMTP helo=<HuuPV>

The Zimbra 451 4.3.5 rejection in zimbra.log: a temporary error at RCPT.

The reply code 451 is a temporary failure, so a correct sending server queues the message and tries again. That is why users report "some mail arrives late" and not "mail is lost".

Why Zimbra 451 happens: the policy service

Postfix has a setting named smtpd_policy_service_default_action. Its default is exactly 451 4.3.5 Server configuration problem, and the Postfix documentation describes it as the default action when an SMTPD policy service request fails (Postfix 3.0 and later). In other words, the Zimbra 451 4.3.5 line means that the Postfix SMTP server asked a policy service about the message and got no usable answer.

Zimbra uses such a policy service: the Administrator Guide lists "Cluebringer", the policy daemon cbpolicyd, which is used to enforce actions such as rate limiting. It is started and stopped with zmcbpolicydctl when it is enabled. If the daemon is stopped, overloaded or too slow, Postfix cannot get its answer within the time limit (the default smtpd_policy_service_timeout is 100 seconds) and falls back to the 451 default action.

Step 1: check that cbpolicyd runs

As the zimbra user on the MTA, look at the state of the daemon first. A stopped service explains Zimbra 451 at once, and you can start it:

su - zimbra
zmcbpolicydctl status
zmcbpolicydctl start

Look at /var/log/zimbra.log around the time of the first 451 line for warnings from the Postfix SMTP server about the policy service. If cbpolicyd is running and the errors come only at busy hours, continue with the next steps: the daemon runs out of workers.

How to measure the Zimbra 451 problem

Do not guess; count it. Run grep -c "451 4.3.5" /var/log/zimbra.log before the change and again after a full business day, and compare the two numbers. To see the time pattern, list the hour of each match with grep "451 4.3.5" /var/log/zimbra.log | awk '{print $3}' | cut -c1-2 | sort | uniq -c. If the count peaks at the busiest hours, the policy daemon is short of workers; if the lines are spread evenly or start at one minute, look for a crash or a restart of cbpolicyd at that time. Keep the numbers in your change record, because they show whether the new Zimbra 451 settings really helped.

Step 2: give the server enough resources

In my case the MTA was a medium to large system, and the sizing that worked was this (my own experience, not an official Zimbra limit):

• RAM: 16 GB or more.
• CPU: 8 cores or more for the Zimbra MTA server.

A policy daemon that starts many worker processes needs both memory and CPU. When the host is starved, even a good configuration answers late and Zimbra 451 returns.

Step 3: raise the cbpolicyd worker limits

The configuration of cbpolicyd is a template, /opt/zimbra/conf/cbpolicyd.conf.in. Look at it first:

cat /opt/zimbra/conf/cbpolicyd.conf.in

The template normally takes its worker numbers from Zimbra attributes (the lines with %%zimbraCBPolicydMinServers%% and so on). In this fix those lines are commented out and fixed values are written below them, so that the pool is larger:

#min_servers=%%zimbraCBPolicydMinServers%%
#min_spare_servers=%%zimbraCBPolicydMinSpareServers%%
#max_spare_servers=%%zimbraCBPolicydMaxSpareServers%%
#max_servers=%%zimbraCBPolicydMaxServers%%
#max_requests=%%zimbraCBPolicydMaxRequests%%
min_servers=8
min_spare_servers=8
max_spare_servers=16
max_servers=64
max_requests=1000

What the values mean for the Zimbra 451 problem: min_servers and min_spare_servers keep 8 workers ready, max_spare_servers limits idle workers to 16, max_servers allows up to 64 workers at peak and max_requests restarts a worker after 1000 requests. These are the settings of the pre-forking server of cbpolicyd; I set them for a server that handles many connections at once.

Step 4: restart cbpolicyd and watch the log

Because cbpolicyd.conf.in is a template, restart the daemon so that the configuration is generated and read again. The Administrator Guide lists zmcbpolicydctl for start, stop and restart:

zmcbpolicydctl restart

Then follow /var/log/zimbra.log during a busy hour and check that the number of Zimbra 451 4.3.5 lines goes down. Keep a copy of the original file, because a Zimbra upgrade can replace the template and your edit has to be applied again.

What not to do

• Do not set the default action to DUNNO to hide the error. Postfix allows this value, and it makes failed policy requests pass, but you also lose the rate limits that cbpolicyd enforces.
• Do not raise max_servers without RAM. 64 workers need memory; on a small server they make the problem worse.
• Do not ignore the sender side. A 451 is temporary, so test with a real retry before you decide that mail is lost.

FAQ

Is Zimbra 451 4.3.5 a configuration mistake?

The text says so, but it is Postfix's standard answer to a failed policy request. The configuration problem is that the policy service could not answer, not a wrong line in the Postfix files.

Do I lose mail with a 451?

Normally not. The sending server keeps the message in its queue and retries, so the mail arrives later when the policy daemon answers again.

What else should I read?

If services are shown in red in the admin console see Zimbra some services are not running, and for rejected connections see Zimbra client host rejected. The reference pages are the Postfix SMTPD policy delegation guide and the Zimbra Administrator Guide.

To fix Zimbra 451 4.3.5: check cbpolicyd, size the MTA, raise the worker limits in cbpolicyd.conf.in, restart the daemon and watch the log.