Skip to main content
🐛 Bug Fix Release

Rspamd 4.1.5

Patch Release Gating File and Shared-Memory Message Sources, with Fuzzy Sender Facts and Font-Aware PDF Text Extraction

✅ Added

  • Fuzzy sender facts shared with storages: Rules that share extensions with their storage now attach a sender-facts extension to check commands sent over encrypted connections: a single 32-bit word carrying the SPF, DKIM and DMARC results, the PTR state — including the generic/dynamic hostname verdict that hfilter now publishes for other modules to reuse instead of duplicating its pattern set — a bucketed recipient count, and whether the sending client used TLS. Authentication results separate "this address is the source" from "this address relayed or forged", which is exactly the distinction an IP reputation model built from fuzzy telemetry has been missing. Every field describes the sender and never the recipient or the local policy: recipient counts are bucketed rather than exact, and the client's own verdict, action and score are deliberately absent, since they would create a feedback loop with the reputation list. The values come from the checks themselves rather than from symbol names, so the SPF and DMARC modules now publish their policy results in the task mempool, and FUZZY_CALLBACK gains ordering dependencies so the results exist by the time commands are built (#6179)
  • Font-aware PDF simple font text decoding: A string in a PDF content stream holds character codes, not characters — byte 0xe9 is Oslash in StandardEncoding, e-acute in WinAnsiEncoding and Egrave in MacRomanEncoding — so without the /Encoding of the font that drew it the text cannot be decoded, and running charset detection over raw codes is guesswork. A new libmime unit owns the three Annex D base encodings and per-font /Differences overrides, plus a builder that turns a page worth of text runs into a single UTF-8 buffer with no per-run allocation and unescapes literal and hex strings in one pass instead of a chain of Lua gsubs. The extractor now keeps the font selected by each Tf operator, resolves every font of a page to an encoding, and replays the whole page through one decoder; quality filtering is scored per page, where the statistics mean something, rather than per text operator (#6175)
  • PDF /ToUnicode CMaps and composite fonts: A composite (Type0) font addresses glyphs by index, so its character codes mean nothing on their own and no encoding table can recover the text — the only thing that can is the /ToUnicode CMap the producer embedded, which is now parsed: codespace ranges give the code widths, bfchar maps single codes and bfrange maps runs, with UTF-16BE destinations that may expand to several characters or a surrogate pair. Every table is capped, because a CMap comes from the message. A font's CMap takes precedence over /Encoding — its codes are glyph indices, so an encoding table would only invent letters — codes the CMap does not map produce nothing rather than leaking a glyph index into the text, and the UTF-16 heuristic now applies only when no CMap is authoritative, since composite fonts emit exactly the zero-interleaved byte pattern it looks for and it used to hijack the very strings a CMap is meant to decode (#6175)
  • Structured fuzzy results and message size in the Elastic schema: The elastic plugin replaces the flat fuzzy_hashes list with per-match objects from task:get_fuzzy_results() — rule, symbol, found and queried hashes, type, probability, flag and exactness — with a fuzzy_nested option to toggle the nested type, and adds the message size to the schema mapping so its type is declared rather than inferred per index (#6154, #6136)

🔄 Changed

  • GPT condition callback returns only a boolean (incompatible): The condition option was previously expected to return both the decision and the content for the LLM request, a contract that was non-obvious and only discoverable through debugging. The callback now returns only true/false plus an optional reason, and request content generation has moved into gpt_check itself, making custom conditions much simpler to write. Deployments whose condition returned request content must update it to return the boolean decision alone (#6159, #5857)
  • Explicit allow_file_and_shm_inputs defaults shipped: The normal, controller and proxy worker configurations now set allow_file_and_shm_inputs = true explicitly, with a comment explaining what the option permits, that it applies to TCP listeners only (unix sockets always allow these inputs), and that a listener with it enabled must not be exposed to untrusted networks — authentication and encryption do not authorize the capability. No shipped default value changes in this release, so existing deployments keep working; the default becomes false in the next major release, so operators relying on file or shared-memory inputs over TCP should set the option deliberately now. Commented max_connections examples are also added to the controller and proxy (#6162)

🔧 Fixed

  • Per-key fuzzy forbidden_ids replaces the default set: The v2 reply path checked the global default_forbidden_ids first and suppressed matching flags unconditionally, letting a per-key set only add further restrictions. That inverted the semantics: a key carrying its own forbidden_ids — including an explicitly empty one, which means "nothing is forbidden for this key" — is supposed to replace the global default, which exists only for keys with no ACL of their own. In production this meant a storage-wide forbidden_ids = [8] suppressed flag 8 for every client, including keys whose own list was empty, so that feed was not served at all. The per-key set alone is now checked whenever the session is encrypted and the key has one, with the global list as the fallback
  • Per-key ACL applied to delayed v1 replies: The v1 reply path gated the per-key check on a positive match probability, but the DELAY branch zeroes the probability while leaving the flag intact, so a delayed reply fell through to the global default and a key silently lost its own ACL — the same inversion as in the v2 path, limited to encrypted delayed replies. The probability check is dropped; outside DELAY nothing changes, since a zero probability means a miss with no flag to filter
  • Unknown fuzzy wire extensions are skipped, not fatal: The extensions TLV was not self-describing — an entry's length was implied by its type — so a storage could not skip a type it did not know and rejected the whole command, meaning the client got no fuzzy answer at all rather than merely losing the extra field, and every new extension type would have broken every storage predating it. Type bytes from 0x80 up are now always followed by a length byte and can be skipped when unknown, the sender-facts extension is reserved in that range, and the parser was consolidated so a malformed packet from the network can no longer reach an assertion that aborts the daemon (#6179)
  • No fuzzy extensions for own senders or plain-text rules: The sender address extension was shipped whenever a rule did not set no_share, so deployments running fuzzy checks on outbound or authenticated submission traffic sent their own users' IP addresses to public fuzzy servers. Extensions are now omitted entirely for authenticated tasks and sources in local networks — such addresses are noise in a reputation model and exactly the values that had to be filtered downstream anyway — and, as the code already claimed but did not enforce, for rules that talk to their storage in plain text, where the data would travel in the clear (#6179)
  • static_embed word extraction bounded: The static_embed neural provider embedded every word of a message, so a multi-megabyte text part became a million-entry table plus a mean-pool over every token — about 200 ms per megabyte inside a synchronous rule, observed as multi-second event-loop stalls with "slow synchronous rule" warnings — and the subject handling compounded it by prepending words one at a time, shifting the whole table per word. Word extraction now honours max_words as a total cap that the provider passes through (the knob fasttext_embed already had), with subject words seeding the table first and body words appending up to the cap. The cap changes long-text vectors, so training and inference must agree on it (#6187)
  • Neural fusion survives a missing provider: The fusion loop iterated the collected feature vectors with ipairs, which stops at the first hole in the table — one failed or unregistered provider in an early slot silently discarded every later provider's collected block, and downstream fusion consumers went dark with no diagnostic at all. The loop now covers the whole provider slot range including the trailing metatokens slot, so a missing provider yields a partial fused vector that dimension checks can actually see and report (#6186)
  • Phantom slow-rule warnings from the symcache: A rule's start time was stamped only on profiled tasks, but the elapsed time was measured whenever the task was profiled or the rule had ever been flagged slow. Once a rule had been flagged slow once, every subsequent non-profiled task measured it against a start of zero — the task start — so any DNS-bound or deadline-hitting scan logged a multi-second "slow synchronous rule" against a callback that actually ran in sub-millisecond time; one production scanner saw around 1200 phantom 5–8 s warnings per day. The stamp is now taken under the same condition as the measurement (#6188)
  • HFILTER_HELO_IP_A compares HELO addresses against the connecting IP: The symbol is documented as "Helo A IP != hostname IP", but the comparison sat in a branch reached only when both the A and AAAA lookups had failed — where the resolved-address list is necessarily empty — so it fired if and only if the HELO host resolved to nothing, a condition already covered by HFILTER_HELO_NORES_A_OR_MX, and never on the mismatch it describes. Because the two lookups complete in arbitrary order, the fix joins on an explicit completion counter and judges once both callbacks have returned, keeping the MX fallback on the both-failed path and ignoring transient lookup failures rather than treating them as a mismatch. Functional coverage is added for a plugin that previously had none (#6176, #6165)
  • Schemeless email hrefs no longer fire URL_USER_PASSWORD: When guessing the missing scheme of an HTML href, only a leading @ selected the mailto: prefix, so <a href="user.name@example.com"> — where the first non-alphanumeric character is the dot in the local part — kept the http:// prefix and produced http://user.name@example.com with a userinfo component, firing URL_USER_PASSWORD (2.0 points) on legitimate messages; nodemailer emits exactly such quote-attribution links. Characters legal both in a hostname and in an email local part (., -, _, +) are now treated as ambiguous and scanning continues until a genuinely structural character decides, so javascript:/data: hrefs, relative paths and bare hostnames all keep their previous handling
  • get_urls_filtered() no longer drops flagless URLs: When called without an include list, the function sets its mask to "all flags", but the include test was a plain bitwise and against that mask — which a URL carrying no flags at all still fails, and URLs extracted from HTML parts carry no flags. The no-argument form, which asks for everything, silently returned nothing for HTML-only messages. The include filter now applies only when a list was actually given: rspamadm mime urls shows HTML URLs again, and URI_COUNT_ODD no longer runs its parity check on an undercount for exactly the multipart/alternative messages it targets (#6182, #5891)
  • MIME rewriters unwind every boundary level: add_text_footer(), remove_attachments() and multipattern_text_replace() closed a fixed number of boundaries when the part walk returned from a nested part to a shallower one, rather than however many levels were actually left behind. As soon as a message nested deeper than the hardcoded count, the boundary stack desynchronised: closing delimiters landed in the wrong places, turning attachments into an unreachable epilogue in the footer case and leaking literal --boundary-- text into part bodies in the others — reproducible through rspamadm mime strip on a triply-nested message. All three now unwind to the boundary being re-entered, which also subsumes an earlier multipart/related special case, with unit tests over flat and one- to three-level nesting that fail on the unpatched code (#6173, #6178, #6170)
  • TJ positioning gaps kept out of the PDF character code stream: The numbers in a TJ array are positioning adjustments, not character codes, but a large negative one was turned into a space and concatenated with the strings around it. For a simple font that only made an ASCII space arrive by the wrong route; for a composite font it inserted one byte into a stream of two-byte codes, so every code after the gap was read across a boundary. The pieces of a text operator now travel separately and the gap is appended structurally, where it cannot disturb the framing (#6175)
  • Inline PDF /Resources dictionaries honoured: Only the indirect form of a page's /Resources was unwrapped; a /Resources written out as an inline dictionary — the common case — was replaced by an empty table, so everything a page declares, fonts above all, was invisible to extraction (#6175)
  • Font-blind ligature substitution removed: A fixed rewrite turned seven character codes into ligatures regardless of the font in force, though a character code has no meaning without one: two of the codes are fi/fl only in StandardEncoding, two only in MacRomanEncoding, and three in neither anywhere, while in WinAnsiEncoding — the most common encoding of all — every one of the seven is an ordinary letter. German text lost each ß to an "fl" and each registered sign became "fi" before charset detection ever saw the bytes. Ligatures now come back correctly through the font-aware decoder's per-encoding slots (#6175)
  • CMap codes carry their byte width: Mappings were stored under the numeric code alone, so a one-byte <41> and a two-byte <0041> collided even though a CMap may declare both — the byte count is what places a code in a codespace, which the input framing already honoured but the tables then ignored, so a one-byte mapping could answer for a two-byte code or be overwritten by one. Both singles and ranges are now keyed by width as well (#6175)
  • CMaps inheriting from another CMap are refused: A usecmap reference was treated as a marker and thrown away, so a CMap built as overrides on top of a base returned only the overrides — every code the base was meant to answer for silently vanished, and the surviving ones made the result look like real text. The base is a CMap resource that is not in the file and cannot be resolved, so such a CMap is now refused and the part left undecoded, where the computed-part guard can see it for what it is (#6175)
  • Computed text parts kept raw and CED charset names resolved through the alias table: PDF extraction injects synthetic text parts whose bytes may be valid in no charset at all; running charset re-detection over them produced non-IANA names such as "BIG5-CP950" that ICU rejects. Computed and extracted text parts are now kept raw, so undecodable glyph data stays out of tokenisation and language detection, while CED-detected charsets for regular parts are routed through the alias table (mapping "big5-cp950" to "big5") so legitimate Big5 content still converts (#6175)
  • EUC-TW misnomer aliases restricted to CED output: Folding CED's invented "CNS" and "EUC" labels into euc-tw in the global substitution table also resolved the charset a message declares — and a message declaring charset=euc is far more likely to carry sloppy EUC-JP or EUC-KR than EUC-TW, so data that used to be left raw decoded cleanly and said something else. Both labels now resolve only where CED output is turned into a charset name, and nowhere else (#6175)
  • Hyperscan cache usable with a Redis backend: The TTL-refreshing GETEX is classified by Redis as a write command, but the cache issued it without marking it as one, so with a split master/replica setup every single load failed with "READONLY You can't write against a read only replica" and all maps and multipatterns silently ran on the PCRE/ACISM fallback — easy to miss, because the EXISTS check used to skip recompilation is a genuine read that replicas serve happily, keeping startup fast while no worker ever received a database. GETEX is now used only when reads and writes go to the same servers; otherwise reads hit the replicas and a separate EXPIRE against the master refreshes the TTL. The fix also handles nil replies properly via a published rspamd_redis.null sentinel with lua_redis.is_null() (previously a missing key crashed inside the error-formatting code), sanitises non-positive TTLs that would have deleted the very database just read, reports why a hot-swap did not happen instead of a misleading file-not-found warning, loads databases asynchronously on worker startup, and compiles regexp maps read after the fork in the process that read them — previously a large pattern map served over HTTP without a cached copy stayed on PCRE for good, because its compilation queue entry lived in a process that never compiled it (#6161)
  • Installed hyperscan databases are not fetched twice: A map enters the compilation queue via its read callback and is announced by a notification once the database exists, and both paths loaded it, so every map update cost double the traffic — megabytes for a map of a few thousand patterns. The load is now skipped when the helper already has a database; this cannot keep a stale one, since a helper's patterns never change after creation and every load verifies the content digest
  • ARC_ALLOW_TRUSTED scored at most once: The symbol could be inserted multiple times for a single message, multiplying its score; it is now added at most once (#6152)
  • History reset clears every row: Two separate bugs left rows behind. The legacy in-memory ring buffer zeroed only the rows up to the current position and never reset the cursor, so once the buffer had wrapped, the tail kept completed entries that the read path still displayed; all rows are now cleared and the cursor reset. The Redis backend used LTRIM <key> 0 0, which retains list index 0 — the newest entry, since rows are prepended — so the most recent row always survived a reset; the list is now deleted outright, with a functional test covering the whole flow (#6185, #2910)
  • WebUI searches symbol option values again: The Tabulator global search built its haystack from symbol name and description only, dropping the option values rendered in the symbols cell — a 4.1.4 regression that broke searching for values such as asn:205640. Options are included in the haystack again, restoring the old FooTable behaviour for both the history and scan tables (#6169, #6168)
  • Logger format string and argument mismatches: Eight logger calls across lualib and the Lua plugins supplied a different number of arguments than their format string consumed. Most only produced extra-argument noise, but three misreported or lost information: the clustering plugin's Redis success message printed a literal "unknown error", the neural module printed a rule name where a value belonged and dropped the value itself, and lua_stat dropped the actual Redis error from a metadata conversion message (#6177, #6174)

🛡️ Security

  • File, Path and Shm message sources gated by transport: The File/Path/Shm/Shm-Offset/Shm-Length headers, their v3 metadata equivalents and the proxy's File query argument make rspamd open a path or map a shared memory object that the client names, and they were accepted on any transport — so any client able to reach a scanner, controller or proxy TCP port could have an arbitrary file readable by the rspamd user parsed and its content reported back. A new per-worker allow_file_and_shm_inputs boolean now gates the capability, derived solely from the transport the connection was accepted on: unix sockets always allow these inputs, since access is already controlled by the socket's filesystem permissions, while nothing the client sends takes part in the decision and no other credential — controller password, enable_password, secure_ip match, encrypted connection — unlocks it. Gating is applied before the named object is opened, mapped or statted, and two bypasses are closed: query arguments that the proxy turned into trusted upstream headers, and the fuzzy controller endpoints that build their own task. The inputs themselves are hardened — paths validated before any syscall, files snapshot with bounded reads instead of parsing a shared mapping the caller can truncate mid-parse, shared memory mapped as a page-aligned window rather than the whole object — the proxy strips client-supplied Shm headers as reserved hop-by-hop headers, the scanner's max_tasks now counts accepted body-pending connections, and the controller and proxy gain max_connections/max_connections_per_source admission limits enforced before any expensive allocation. encrypted_only also no longer exempts peers via a locality test that matched IPv6 link-local and site-local addresses, which had let any host on the segment skip the encryption requirement. The option defaults to true in this release with a prominent startup warning per TCP listener, and becomes false in the next major release (#6162)
  • Controller authentication failures rate-limited per source: Verifying a controller password runs a deliberately expensive key-derivation function (~25 ms for the catena hash that rspamadm pw emits by default) synchronously in the controller's event loop, and the password cache keys on the plaintext that succeeded — so a stream of distinct wrong passwords fell through to the KDF every time, and roughly 40 attempts per second were enough to saturate a single-process controller and stall every other request it serves. Failed attempts now feed a per-source leaky bucket checked before any password work, so a source over budget gets a 429 in well under a millisecond; a successful login clears the bucket, trusted sources (secure_ip, unix sockets) are never throttled, and throttled requests add no further penalty, so a shared source cannot be pushed into a permanent lockout. Configurable via max_auth_failures (default 10, 0 disables) and auth_failure_window (default 60 s); note the key is the real TCP peer, so behind a trusted reverse proxy all clients share one bucket. The redundant second KDF run when password and enable_password are configured to the identical hash is also skipped, halving the cost of every wrong password (#6162)

This release closes a long-standing gap in the scan protocol — the File, Path and Shm message sources were accepted from any TCP client, letting anyone who could reach a scanner, controller or proxy port have arbitrary readable files parsed — behind a new transport-derived `allow_file_and_shm_inputs` option, and rate-limits the controller's expensive password verification per source. Fuzzy checks gain a privacy-conscious sender-facts extension over encrypted rules, riding on a wire format made forward compatible so new extension types no longer break older storages, while per-key fuzzy ACLs once again replace the global default as documented. PDF text extraction becomes font-aware, decoding through each font's /Encoding and /ToUnicode CMap instead of guessing over raw character codes, and the hyperscan cache finally works against replicated Redis. Recommended upgrade for all users; operators exposing scanner, controller or proxy ports beyond localhost should review the new `allow_file_and_shm_inputs` and connection-limit options now, ahead of the default flip in the next major release.