General

Pick an MTA that supports the options described below, and configure them appropriate to your situation.
Leave room for exceptions as not everyone on the internet will come close to running a complaint MTA. You will always have a critical business partner that just doesn’t care.

When making exceptions, try not to make global exceptions like whitelisting a domain. Be as granular as possible by whitelisting the sender’s specific mail server, or an individual address.
Exceptions for non-compliance to RFCs should not exempt the message from passing thru higher-level security scans such as malware, phishing filters, etc.

Client Security

Require strong passwords (KC)
Use multifactor authentication when appropriate – webmail, etc. (KC)
Use TLS to secure all client access methods (not just web) (NIST)
User awareness

Server Security

Network

Deploy behind firewall/IPS

At least deploy a host firewall and possibly HIDS (NIST)

Deploy in a DMZ, behind a firewall and not NAT’d from the local network

Logging

Ensure logging is configured and forwarded to a centralized logging server

DNS

Multiple DNS Servers

Find supporting information

MX Records

According to RFC5321 section 2.3.5, MX records must point to one or more A/AAAA records, and cannot be CNAME records. The sending MTA will loop the destination’s MX records in order of ascending priority until a connection can be made and the message successfully transferred (RFC5321 section 5)
While SMTP does have a store-and-forward design, servers will typically purge undeliverable email after a certain period of time has passed. Having at least two MX records ensures that email can be delivered to an online server.
Resilience Strategies

At least 2 MX records in different subnets, similar to how DNS recommends servers in difference subnets for DoS protection.
A primary MX and a backup MX which could be another service provider paid to store your mail if you go offline.

Reverse DNS matches Forward DNS (KC)
SPF Record (KC)
DMARC Record
DomainKeys/DKIM (KC)

SMTP

RFC

Simple Mail Transfer Protocol
Anti-Spam Recommendations for SMTP MTAs
*

Prefer (or enforce) TLS transmission (KC)

SMTP Service Extension for Secure SMTP over Transport Layer Security
Will forcing TLS transmission to prevent legit emails? Will it cut down on spam/malware?
Only allow connectivity with certificates which can be validated (i.e. not self-signed or expired.

Enforce relay control (KC)

Anti-Spam Recommendations for SMTP MTAs
Only accept mail for hosted domains/users (Chris)
Enforce authentication controls for relay (i.e. only hosted domain/users can send outbound email) (KC)

Rate Control/Tarpitting

Anti-Spam Recommendations for SMTP MTAs
Use “tarpitting” to increase the resources/time required to send multiple emails to the domain

Verify FROM

Anti-Spam Recommendations for SMTP MTAs
FROM domain MX record should exist

Verify RECEIVED BY Headers

RDNS lookups – reject if (KC)

RBL lookups – reject/flag (KC)
SPF lookups – flag/reject
DKIM/DomainKeys – flag/reject
Content Filtering (Chris)

Attachment blocking (Chris) – block executables and common malware vectors

Use both file extension and file headers to determine file type – foils renamed files
Executables – EXE, BIN, COM (others)
Scripts – BAT CMD PS1
Screen Savers – SCR
Flash/Shockwave
Windows Help Files – HLP and newer version?
Compiled HTML – MHT, MHTML files
Advanced level – stripping based on regex’s for commonly used schemes to hide dangerous extensions from users (but this really wont be needed if you are doing the basics)

Malware scanning (KC)

Strip known malware (matching sigs) (Chris)
Scan within archive/zip files (Chris) – multilevel
Mark encrypted attachments as suspect? (Chris) – worth the discussion as malware vector but also common for security
AdvancedConsidering deploying unknown files in a sandbox

POP3/IMAP

Find RFC/Documentation backup

Enforce the use of TLS (KC)
Strong Passwords (KC)

Require strong passwords which have not been used in a known breach.

Backup/Recovery

Perform a routine configuration backup – at least before each time you change the configuration. Store the configuration off-server for security.
Perform a routine data backup of all mailbox contents.

Patching – routinely patch the server (NIST)
Prevent password guessing (NIST)

Account lockouts, etc.
Consider fail2ban on linux machines (Chris)

After-reception attack prevention

Scan mailboxes on a routine basis and remove malware. This will provide malware protection for messages received before the attachment or sender was determined to be dangerous.

Sources

NIST https://nvlpubs.nist.gov/nistpubs/Legacy/SP/nistspecialpublication800-45ver2.pdf
KC https://www.alienvault.com/blogs/security-essentials/basic-best-practices-for-configuring-email-servers (by Kim Crawley)