Skip to main content
🚀 Major Release

Rspamd 4.1.0

Major Release with Load-Aware Upstreams, MX Check Rework, Dynamic Composites, and Hardening

🔄 Changed

  • MX check rework: mx_check replaces its single domain-keyed cache with a three-layer Redis design (d:<domain> / m:<mxhost> / i:<ip> under <key_prefix>:), so two domains sharing an MX host (every G-Suite/M365 tenant, every ESP customer) reuse the m- and i-layer entries and hit cache with zero new DNS or TCP work. Outcome symbols are finer-grained and MX_NONE now replaces the old MX_NXDOMAIN/MX_MISSING; the probe is split into clean connect-only and full SMTP-banner shapes with multi-line greeting support. Operators should review their mx_check scores and any rules referencing the old symbol names (#6055, #6032)
  • Fuzzy SRV discovery by default: The stock rspamd.com fuzzy rule now uses service=fuzzy+rspamd.com SRV-based discovery instead of a hardcoded round-robin host list, so backends and ports are managed entirely in DNS. The legacy fuzzy1/fuzzy2 hostnames keep resolving to every live backend, so installs that pinned the old string are unaffected

✅ Added

  • Custom metadata for /checkv3: A headers sub-object in the multipart metadata part is injected into the task request headers (so task:get_request_header() works for custom fields), and the full metadata object is exposed to Lua via task:get_metadata() / task:get_metadata_field(key). Both paths travel in the multipart body, free of the 80KB HTTP header limit that v2 hits; rspamc gains a repeatable --metadata-header KEY=VALUE option (#6074)
  • Dynamic composites: A reserved dynamic key in the composites { ... } block attaches a hot-reloadable map of composites (file, URL list, or signed map) using the same vocabulary as static composites. Reloads register new names with the symcache, materialise removed names as stubs, and swap an atomic generation so in-flight tasks keep their snapshot (#6064)
  • Bulk and regexp symbol lookups: task:has_symbol() / task:get_symbol() accept a {S1, ..., Sn} table form, and new task:has_symbol_regexp(re) / task:get_symbol_regexp(re) match fired symbol names against an rspamd_regexp userdata
  • Phase-specific TCP timeouts: rspamd_tcp.new gains connect_timeout / read_timeout / write_timeout for independent per-phase budgets and an on_error(err, conn) callback that fires once for pre-connect failures (DNS, socket, connect refused/timeout, SSL handshake). Legacy single-timeout callers keep their existing deducted-budget contract unchanged (#6034)
  • Stalled-scan diagnostics: On task timeout the log now includes a grouped summary of still-pending async events (DNS / Redis / Lua HTTP / fuzzy / TCP) annotated with the stalling rule and operation, plus the list of symbols that started but never finished — replacing the opaque "forced processing" line with an immediate picture of what hung the scan (#5990)
  • Load-aware upstream selection: Upstreams now support Power of Two Choices selection, a per-upstream latency EWMA with configurable half-life, and linear slow start on revive to avoid thundering-herd flap loops. RSPAMD_UPSTREAM_RANDOM callers are transparently upgraded to P2C (#6013)
  • Per-target SRV upstreams: Each SRV reply target now materialises its own struct upstream with its own error budget, weight, latency EWMA, and address list, so SRV weights are honoured and a single failing target no longer drains the whole cluster's budget (#6030)
  • Deferred DNS resolution: A transient DNS failure at config time no longer drops an upstream and cascades into module init failures. Unresolvable hostnames are kept pending and retried by the async lazy-resolve machinery (with exponential backoff), so the daemon starts and recovers automatically without a restart (#6008, #6000)
  • MX check classification and trust maps: mx_check partitions resolved MX-target IPs into PUBLIC / LOCAL / BOGON (only PUBLIC is probed), adds bad_mxs (glob on hostnames) and bad_ips (radix on IPs) punishment maps with optional per-entry weight multipliers, per-source checks across envelope / reply-to / MIME-from, and check_authorized / check_local run-scope toggles (#6039, #6032)
  • External-service anchor symbols: Every external service is scheduled under a stable <RULE>_CHECK callback symbol so it is a predictable dependency target regardless of how the scan-result symbols are named, generalising the existing VADE_CHECK / CLOUDMARK_CHECK pattern
  • Chain-aware URL redirector cache: url_redirector gains a per-hop Redis cache with intermediate-hop injection, so shared intermediate links are reused across chains and resolved hops are made visible to downstream modules; the walk self-heals on partial cache misses and surfaces previously-silent lock/stale conditions (#6014)
  • Stealth fingerprint profiles: url_redirector replaces its flat default_ua list with five coherent browser profiles (Chrome, Edge, Firefox, Safari), each bundling a User-Agent with the exact header set, values, and order that browser sends (including sec-ch-ua client hints where appropriate). One profile is picked per task and reused across every hop (#6053)
  • Glob redirector hosts: redirector_hosts_map now accepts glob patterns (e.g. *.bit.ly, *.t.co); bare hostnames still match exactly, so existing maps are unaffected (#6056)
  • GET for selected redirector URLs: url_redirector can issue GET (rather than HEAD) for an operator-defined list of URLs matched by regexp (#6043)
  • Insertion-ordered HTTP headers: An opt-in RSPAMD_HTTP_FLAG_ORDERED_HEADERS emits headers in insertion order instead of hash order; lua_http accepts a list form ({{'name','value'}, ...}) to preserve order, used by the redirector stealth profiles
  • Richer Elasticsearch logging: The elastic module now logs Reply-To user/domain, received IPs, per-URL and CTA URL metadata via a new collect_urls block, and the forcing module name from task:has_pre_result() (#6018)
  • ClickHouse column presets: Named extra_columns presets, including an initial outbound preset, can be selected without hand-rolling the schema (#5983)
  • New selectors: fuzzy_digest, fuzzy_shingles, authenticated, and received_count (#5981)
  • HTML5 tag definitions: 32 modern HTML5 tags (sectioning, media, text-level, interactive, forms, web components), notably video/audio/source/track/picture/svg, so their URLs and structure are visible to the parser; new tag IDs are appended so existing IDs stay stable
  • DMARC report throttling: rspamadm dmarc_report gains -w/--batch-wait to pause between batches (and stagger report generation), avoiding overload of the SMTP server and a weak resolver (#5985)
  • autolearnstats sorting: --sort-by <col> and --group options for the rspamadm autolearnstats table output (#6050)
  • Feedback report parsers: New lua_feedback_parsers lualib for parsing DSN (delivery status) and ARF (abuse) reports (#5982)
  • Structured custom-Lua loader: lua_extras provides a two-phase loader for custom selectors, maps, and regexps under lua.local.d/{maps,selectors,regexps}/, resolving cross-kind dependencies (maps → selectors → regexps) so a selector can consume a map registered by an earlier kind (#6020)
  • eXpurgate scanner: New lua_scanners engine for the eXpurgate anti-spam service (#5755)
  • Per-worker memory dumps: New rspamadm control memstat command reports per-worker RSS, per-callsite mempool counters, Lua heap usage, and structured per-arena jemalloc stats, with --short, --sort, and per-section toggle flags (#6016)
  • Container-friendly baseline config: The baseline pidfile and logging type/filename are now env-overridable (RSPAMD_PIDFILE, RSPAMD_LOG_TYPE, RSPAMD_LOG_FILE); an empty pidfile disables it (useful as PID 1), and RSPAMD_LOG_TYPE=console logs to stdout. Stock installs render the previous defaults bit-for-bit (#6067)
  • Auto-loaded fasttext model: When no fasttext_model is configured, the shipped $SHAREDIR/languages/fasttext_model.ftz is loaded if present, so images bundling the model can drop the explicit override; stock installs without the file behave exactly as before (#6067)
  • Fixed-point float formatting: fpconv gains %.Nf fixed-point formatting with correct rounding and carry handling (#6061)

🔧 Fixed

  • Composite visibility from postfilters: Composites depending only on ordinary filter-stage rules were wrongly deferred to the post-composites stage because nearly every virtual/callback symbol carries NOSTAT; a composite is now deferred only when a dependency actually resolves to the postfilter stage, restoring task:get_groups()/get_symbols() visibility from postfilters
  • Ratelimit multi-bucket tracking: A selector rule with several buckets (e.g. 200/1h plus 30/1m) keyed every bucket on the selector value alone, so only the last bucket was tracked and the other limits were silently ignored; each bucket now gets a distinct Redis key (#6076, #6059)
  • Verbatim href preserved: url:get_raw() returned the partially percent-decoded scratch buffer for HTML URLs; url->raw now points at a mempool-owned copy of the verbatim trimmed href (#5986)
  • Mailto canonicalisation: A bare email in text/HTML and the same address inside an explicit mailto: URL were extracted as two separate emails; both injection sites now canonicalise to the slash-less mailto: form (RFC 6068) so dedup collapses them
  • Long userinfo URLs preserved: A blanket length REJECT silently dropped URLs whose userinfo exceeded 2KB — exactly the https://legit.com @evil.com/... userinfo-obfuscation phishing pattern; the cap is raised to 16KiB and the URL is flagged obscured as soon as userinfo crosses 64 bytes
  • Naked-domain false positives: url_suspect now requires a TLD of at least 3 chars for word_dot naked-domain matches, so prose like "pale blue dot so insignificant" no longer normalises to blue.so; explicit-protocol patterns still match two-char TLDs
  • Deterministic ARC header order: lua_mime.modify_headers honoured its order list but serialised in hash order; ARC sets are now emitted in the conventional ARC-Seal / ARC-Message-Signature / ARC-Authentication-Results layout, which some validators (e.g. O365) require (#6052, #6045)
  • DKIM permfail mapping: An invalid DKIM record now maps to dkim=permerror in Authentication-Results instead of falling through to dkim=none (#6028, #5957)
  • EAI Message-ID handling: The INVALID_MSGID rule now honours mime_utf8, and the enable_mime_utf8 option spelling is registered as an alias so it actually takes effect, fixing false positives on valid SMTPUTF8 Message-IDs (RFC 6532) (#6011, #6007)
  • Fuzzy storage peer pipe: Partial peer-pipe writes were retried from byte 0, shoving garbage into the pipe; writes now resume at the sent offset, and pending requests are drained on worker shutdown instead of leaking
  • Fuzzy dynamic-ban leak: Re-applying an already-present ban prefix orphaned a freshly allocated ban struct in the long-lived mempool on every refresh; the prefix is now looked up first and mutated in place on a hit
  • Fuzzy persistent-TCP leak: Per-frame cmd_session state (extensions buffer, key refs) leaked on every frame after the first on a persistent TCP connection; per-command state caching is dropped so each frame starts clean (#6001)
  • Allowed fuzzy clients on TCP: Allowed clients are no longer blocked on the TCP path (#5992)
  • Neural ANN survival: A trained ANN is now preserved across symbol-list drift and symcache-driven profile rotation; the profile digest is stabilised under disable_symbols_input (keyed on the providers config rather than the unrelated symbol catalogue), and training is retargeted to the newest profile so inference no longer goes dark for weeks until a fresh model trains (#6041)
  • Inline /checkv3 settings: Inline metadata.settings on /checkv3 were stashed directly on the task and skipped the apply pipeline; they now run through the same settings.lua apply path as v2, so action thresholds, symbols, subject, variables, and header edits take effect (#5999)
  • /checkv3 request headers: Arbitrary client HTTP headers are now registered so task:get_request_header() works under v3, restoring v2 behaviour (#5998)
  • Token-bucket recovery: A flapping upstream's token bucket drained monotonically toward zero and never recovered; lazy time-based refill restores tokens over wall time so a recovered upstream re-enters selection
  • get_random termination: rspamd_upstream_get_random looped forever when the only alive candidate matched the except argument; the empty and single-survivor cases are now front-gated
  • Lua TCP leak: A connection that read without writing leaked; the leak is closed (#6048)
  • Redis master under Sentinel: Standalone rspamadm tools never resolved the current Redis master, round-robining writes that failed READONLY on replicas; a new one-shot lua_redis.prepare_redis_setup resolves the master (and loads scripts) for tools like dmarc_report (#6015, #6009)
  • Elastic overflow guard: The row-limit overflow guard called a non-existent lua_util.newdeque(); it now resets the buffer via the local Queue class
  • DMARC report on PUC Lua: The connect timestamp is floored before os.date, which PUC-Rio Lua rejects for non-integer floats (only seen on the Fedora build)
  • Greylist timeout separation: The greylisting period (5 min) was being picked up by lua_redis as the Redis connection timeout; a separate redis_timeout (default 1.0s) is introduced and propagated into nested redis{} blocks (#5977)
  • rspamadm vault output: rspamadm vault list produced empty output for large vaults because the payload passed through a format-string logger; it is now written to stdout directly via io.write (#6006, #6005)
  • Regexp capture groups: An empty capture group in the middle of a match no longer discards all following groups (PCRE1 and PCRE2); results are truncated at the last non-empty group instead (#5974, #5973)
  • x-binaryenc charset: ICU conversion is skipped for the synthetic x-binaryenc charset across all detection paths, silencing the spurious "cannot open converter" warning and correctly marking the part as raw binary (#5984)
  • Timeout sanity check: configtest now warns when task_timeout is less than a symcache symbol timeout, including per-worker overrides (#5978)
  • libc++ builds: Use string_view::data() for pointer access where libc++ returns a wrapped iterator from begin(), fixing builds with libc++ 22 on FreeBSD (#5969)
  • In-place header rewrites: mime_headers/mime_encoding now recompute lengths after in-place strip/trim rewrites, so stale trailing bytes are no longer pulled into the Message-ID or normalised charset name, and large-buffer offsets use goffset to avoid 32-bit truncation
  • rfc2047 decode: A failed base64 encoded-word no longer leaves uninitialised bytes in the decoded header value; the token length is reset to the saved offset on the failure path

🛡️ Security

  • S/MIME NULL deref: An S/MIME signed message wrapping a zero-length pkcs7-data OCTET STRING crashed the parser on the first byte check; the empty inner recursion is skipped and a defensive guard is added against NULL/empty buffers
  • S/MIME recursion bound (DoS): Deeply nested application/pkcs7-mime layers re-entered the parser without incrementing the nesting counter, recursing to a depth bounded only by message size and exhausting the worker stack; the S/MIME re-entry is now accounted against max_nested and the CMS/PKCS7/BIO objects are freed on the error path
  • MIME parser guards: Additional defensive guards against NULL dereference (Content-Type iteration, malformed PKCS7 signed-data) and a leak of the recursive parser context on the early error-return path, plus a corrected begin-base64 UUE prefix offset
  • Nested-query URL DoS: A crafted message with a few levels of percent-escaped nested query URLs exhausted the multipattern hyperscan scratch pool and aborted the worker via assertion; the scratch budget is enlarged, the query-nesting cap is restored to a fixed functional limit, and scratch exhaustion is made non-fatal (#6066)
  • Archive parser hardening: RAR (v4/v5), ZIP, and 7-zip parsers are hardened against malformed attachments — re-validated filename lengths, guarded section/digest/bit reads, clamped extra-field advances, a fixed EOCD minimum, widened offset arithmetic against 32-bit wrap, and corrected 7-zip varint decoding (uninitialised value and an undefined-shift fix)
  • Image linking guard: Content-Id image linking is guarded against a NULL decoded header
  • CSS out-of-bounds read: The CSS ident-escape scanner could read one byte past a tightly-sized style-attribute buffer when a token ended in backslash plus a hex digit; the increment is now bounds-gated
  • Header lookahead over-read: rspamd_string_find_eoh peeked p[1] guarded only by p < end, reading one byte past the buffer on input ending in \r\r; it now checks p + 1 < end
  • SPF over-read: A TXT record of exactly spf2. advanced past one unvalidated byte and read past the string end; the parser now advances only past the validated prefix with short-circuiting checks
  • DNS label overrun: rdns_parse_labels never verified that label data fits within the packet, so a reply declaring more bytes than remained made the second-pass memcpy read past the (exactly-sized, on TCP) buffer; both plain and compressed labels are now validated, with an off-by-one fix in offset decompression
  • HTML entity overflow: In-place HTML entity decoding assumed the replacement is never longer than the source, which fails for short names expanding to multiple codepoints (nGt, nLt, nvap); the replacement is now bounds-checked against the remaining buffer
  • Empty-host URL over-read: rspamd_url_maybe_regenerate_from_ip could read host[-1] on an all-dots or zero-length-after-decode host; the trailing-dot loop condition is reordered and host length is re-checked after decoding
  • Fuzzy network input hardening: Three defensive fixes on user-controlled UDP/TCP paths — reset msg_namelen before every recvmsg to avoid parsing stale stack bytes, validate the reconstructed 14-bit TCP frame length before use, and clamp n_extra_flags before the fixed-size reply memcpy
  • libucl upstream fixes: Ported security fixes from libucl — msgpack negative fixint and unaligned-access fixes, a key-length validation, a nesting-depth limit, parser bounds checks, and a NULL guard in schema validators

This major release reworks load-aware upstream selection (Power of Two Choices, latency EWMA, slow start, per-target SRV, deferred DNS) and the `mx_check` module (three-layer cache, finer outcome symbols, IP-class classification, trust maps), adds hot-reloadable dynamic composites, richer `/checkv3` metadata, phase-specific TCP timeouts, stalled-scan diagnostics, and numerous new features across logging, selectors, and tooling. It also lands an extensive round of memory-safety and DoS hardening across the MIME, archive, URL, DNS, HTML, SPF, and fuzzy-storage paths. Recommended upgrade for all users; operators of `mx_check` should review symbol names and scores, and Sentinel/SRV deployments will benefit from the upstream resilience improvements.