Patch Release with checkv3 Content Negotiation, Neural Extensibility, CSS Hidden-Text Detection, and 4.1.0 Regression Fixes
/checkv3 reply now honours the client's Accept and Accept-Encoding headers at the single reply_v3 chokepoint shared by the scan worker, rspamd_proxy, and the controller. application/json / application/msgpack yield a plain v2-style single-body reply, message/rfc822 a multipart/mixed envelope, and multipart/form-data (also the default for an absent or wildcard Accept) a true form-data envelope; an Accept listing only unsupported types returns 406 Not Acceptable. Compression is negotiated separately — zstd when offered, identity otherwise — and Vary: Accept, Accept-Encoding is always advertised. rspamc now requests multipart/form-data and accepts any multipart/ subtype (#6083)position:absolute|fixed with a large negative left/top), image-replacement text-indent, clip / clip-path collapsing the element to zero area (rect(0,0,0,0), inset(100%), circle(0)), visibility:collapse, and tiny font sizes (≤ 3px). These are now modelled as a hidden display so the effect correctly propagates to descendants, replacing a fragile substring heuristic that only bumped a feature counter and never actually hid the textlua_archive (unpack / unzip / untar) gains an optional opts table that caps extraction while reading, so a tiny hostile archive can no longer expand to gigabytes and OOM the worker: max_output (total uncompressed bytes), max_file_size (per-member cap, truncated at the cap), max_files (member count), and max_ratio (per-member uncompressed/compressed ratio). The ratio uses real compressed bytes consumed, checked per chunk above a 64 KiB floor, so a bomb is stopped early even with no size cap set. All limits are opt-in (0/absent = unlimited), preserving the previous behaviour, and a second boolean return flags truncation so a capped extraction is never mistaken for a complete onerspamd_text class expose ENT-style byte statistics for content analysis — entropy(), byte_mean(), byte_deviation(mean), serial_correlation(), and monte_carlo_pi(). Each takes an optional 1-based (start, len) range (consistent with the rest of the slicing API), runs O(n) over a single shared histogram pass, and produces deterministic, bit-reproducible results (#6106)0xEDB88320) is now available as rspamd_text:crc32([start[, len]]), rspamd_util.crc32(input[, start, len]), and a streaming rspamd_cryptobox_hash.create_specific("crc32"), bit-exact with zlib crc32() and YARA's hash.crc32 (e.g. "123456789" → 0xCBF43926). The text/util forms return a Lua integer over a zero-copy 1-based slice with no buffer copyRSPAMD_MULTIPATTERN_SOM (rspamd_trie.flags.som) explicitly requests start-of-match offsets, overriding the incompatible single-match flag. The offset convention is now documented — 1-based pattern id, 0-based byte offsets, start inclusive and end exclusive (so end - start is the match length) — and the regex fallback used when hyperscan is unavailable now reports the true start/end from the PCRE search instead of a bogus end - strlen(pattern) for variable-length matches (#6105)register_architecture(name, builder) / get_architecture(name) join the existing register_provider; the built-in symbol, embedding, and conv1d architectures are now registered through it, create_ann() dispatches on rule.architecture and falls back to the historical auto-selection when it is unset (so existing configs are unaffected), and an unknown architecture now fails loudly with a hint that the providing module may not be loadedfasttext_embed provider gains output_mode = "sequence", emitting the first max_words word vectors flattened word-major and zero-padded to max_words * channels, so custom architectures can learn their own pooling instead of receiving a pre-pooled vector. An optional word_selection = "sif" fills the sequence with the most distinctive (highest SIF weight) words from anywhere in the message rather than the leading ones; the default stays prefixattn_pool KANN operator performs multi-head dot-product attention pooling over a zero-padded sequence of word vectors with learned query vectors, masking all-zero padding positions out of the softmax. Exposed as rspamd_kann.layer.attn_pool(node, n_words[, n_heads]), it converges on a word-order-invariant needle-in-haystack task that a flat dense net cannot solve, and the operator is appended to preserve model serialization compatibilityrspamd_kann.transform.slice(node, axis, start, end) (0-based, end exclusive) and rspamd_kann.transform.concat(axis, node1, node2, ...) expose the already-serialized kad_slice / kad_concat_array graph operators to Lua, making split/bypass/merge topologies — e.g. routing parts of a fused input vector through different sub-networks for attention pooling with late fusion — buildable from Luarspamd_dns_server_init, crashing the worker with SIGSEGV at startup. Resolver nameservers are consumed synchronously and can never be promoted asynchronously, so they are no longer deferred, with a NULL guard added as defense in depth (#6096)task:set_pre_result('quarantine', 'reason')) suppressed the command entirely and the message was accepted instead of quarantined. The supplied message now becomes the quarantine reason, matching the reject and tempfail branches; this affects both METRIC_ACTION_QUARANTINE and reject converted via quarantine_on_reject (#6088)worker:spawn_process that returned nil (or any non-string value) logged an error but wrote nothing to the result pipe, so the parent waited forever for a reply while the child blocked on the post-reply ack — deadlocking both processes and anything serialised behind them, notably the neural training lock. Invalid return values are now reported to the parent as a regular error reply, so on_complete fires and the caller can recoverfasttext_embed, text_hash) with the historical learning_rate = 0.01 could drive the net into tanh saturation and silently save a degenerate single-class model that classified every message the same way until the next retrain — roughly one weight-init in three on a real corpus. Dense-feature rules now use the funnel (embedding) architecture, whose layernorm fixes the conditioning; the learning-rate default is resolved by input type (0.001 for dense embeddings, 0.01 for symbol vectors, an explicit config value still wins); and a quality gate rejects a constant/single-class model so training retries on the next cycle with a different initNEURAL_SPAM / NEURAL_HAM stopped firing while the controller logged a version mismatch forever — because a stale, empty, high-version profile entry shadowed the live lower-version blob and, with no TTL, never expired. New versions are now seeded monotonically from the profile actually trained from, selection falls back to the next compatible profile with a live blob (with a throttled warning) when the chosen one is missing, and profile/blob entries get a TTL refreshed each cycle so an actively used ANN never expires out from under its entry (#6089)127.0.0.1 in /etc/hosts), but it scored MX_BOGON_ONLY (+8.0) — the strongest "not spam infrastructure" signal treated as the strongest spam signal, penalising fully DMARC-aligned self-hosted mail. 127.0.0.0/8 and ::1/128 move from the bogon set to the local set, so such mail now emits MX_LOCAL_ONLY (3.0) (#6101)random_monitored RBL checks built random labels from an alphabet including - and _, producing names like _Q8...0- or -7d0... that are not valid DNS labels (RFC 952/1123) and that authoritative DNSBL servers such as spfbl.net reject with SERVFAIL. The alphabet is restricted to alphanumerics, which always forms a valid label regardless of position while keeping ample entropy (#6103)orig_boundary is now used in protocol.c, rspamd_proxy.c, and rspamdclient.c/checkv3, so a plain rspamc --protocol-v3 — which defaults to the controller port 11334 on localhost — returned 404. The endpoint is now registered and routed to the existing scan handler, branching on CMD_CHECK_V3 in both directions to mirror the proxy, with an auth posture matching /check and /checkv2 (#6083)saved_priority initialised to -1 split the first item of each pre/postfilter/idempotent phase off from its priority group and counted it individually, inflating the computed maximum symbols-cache timeout. It is now seeded from the first item's priority so items at the same priority group correctly (#6094, #6092)= (e.g. base64 padding in the path), but the query-embedded scan treated everything before the first = as a parameter key and discarded such URLs. The prefix is now treated as a key only when it has no URL structure characters (: or /); otherwise the whole parameter is scanned@ signs and backslashes were multiplied into many spurious URLs. Long legitimate conversations can accumulate such links, so the spurious expansion could drop mail and, with autotrain enabled, train the system badly; those URLs are no longer multiplied (#6065)max-width:0;max-height:0;overflow:hidden, opacity:0, or height:0 was missed — max-* and overflow were unparsed, opacity was parsed but discarded, and a copy-paste bug applied height:0 to the block width. These properties are now parsed, a zero-dimension-with-overflow:hidden, low-opacity, or hidden-ancestor block is treated as invisible and propagated to descendants, and a child display value can no longer resurrect content hidden by an ancestorASN1_STRING (and thus ASN1_OCTET_STRING) opaque, so direct access to its length/data fields no longer compiled; ASN1_STRING_length() / ASN1_STRING_get0_data() (available since OpenSSL 1.1.0 and LibreSSL 2.7) are used instead, and the legacy OpenSSL init calls are moved under the pre-1.1.0 guard to fix no-deprecated builds (#6087)mime_string's iterator is intrinsically const, so it is marked as such and the const_iterator reference is dropped, fixing a compile error where doctest 2.5.0 receives a const reference and cannot use a mutable iterator (#6072)This patch release fixes a critical 4.1.0 regression that crashed workers at startup when a DNS nameserver failed to resolve, restores milter QUARANTINE handling broken since 3.10.0, and hardens the async Lua coroutine pool against use-after-free on completion races. It also adds `/checkv3` content negotiation via `Accept`/`Accept-Encoding`, opt-in archive extraction limits against zip bombs, broader CSS hidden-text detection, and an extensibility layer for neural feature providers and ANN architectures (including multi-head attention pooling), alongside several neural training-stability fixes. Recommended upgrade for all users, and especially for anyone running 4.1.0.