Major Release with SMTPUTF8 Address Checks, Separate URL Mismatch Scoring, Browser-Accurate HTML, CSS and URL Parsing, and Hardened MIME, Archive, SPF, DKIM and ARC Handling
SIZE_MAX, and the worker aborted. One unauthenticated request was enough. Any other line of this kind was joined to the next field name, so X\r\nFoo: bar became XFoo: bar. The parser now refuses a header line without :, as upstream http-parser and nginx doDeliver-To header or query argument hit an assertion that aborted the worker. Any client could therefore keep the workers crashing. An empty value now keeps the previous value, and the brace stripping checks the length before readinghosts = "localhost") and the default token bucket rotation. The worker then either crashed with SIGSEGV or stopped calling accept(). In the second case the listen backlog filled up and MTA connections stayed in CLOSE_WAIT. Because heartbeats_loss_max defaults to 0, the worker could stay in that state indefinitely. Releases since 4.0.0 are affected, including 4.2.3. 3.14 is not affected. A failed milter message is now completed the same way as a scanned one:no_ssl_verify. The key was a host name, and a Host header can set it. A request could therefore go over a connection to a different address, and forbid_local was checked against an address other than the one actually used. Both are now part of the pool key. Other pool fixes:close in any Connection header ends the connection;Key header also replaced the peer key of the message without releasing the old one. On a client, this meant that a Key header in the plaintext reply headers could replace the key used to decrypt the reply. A Key header is now ignored once the message has a peer keyforbid_local in lua_http covers all local and special-purpose addresses: The option refused loopback, link-local and the networks in local_addrs. However, 0.0.0.0 and :: were allowed, and a connection to either reaches the local host, where the controller trusts 127.0.0.1 and ::1. The option now also refuses:0.0.0.0/8, the shared space 100.64.0.0/10, 192.0.0.0/24 and 198.18.0.0/15;X-Rspamd-Rcpt-Esmtp-Args: The recipient index was converted with strtol into an int without a range check. A negative index read and wrote before the start of the per-recipient array, and a large index grew the array to billions of slots. Only digits are accepted now. The index is bounded by max_recipients, or by 1024 when there is no configuration. The same bound applies to rcpt_esmtp_args in the v3 protocol. Milter passes ESMTP keywords without a value, such as SMTPUTF8 or REQUIRETLS, as bare keywords in these headers. The worker used to drop them and now keeps themk=rsa or an Ed25519 key under k=ecdsa256. Such a key now fails verification. The helpers also leave no errors on the OpenSSL queue, and the Ed25519 helper checks that the key has a raw public keydmarc=pass with a header.from equal to the From domain. Quotes, angle brackets, a trailing dot, IDN and case are normalised before the comparison;d= that the C code verifieddmarc=pass that softened a DMARC reject. Results are now read from the raw value, and Authentication-Results reused for a seal are copied unchanged. A trusted forwarder that writes the sender unescaped into a comment could also let that text open a quote or a comment, which then hid the forwarder's own dmarc=fail. A DMARC result is now treated as ambiguous, and is not trusted, in two cases. The first is when splitting the header on every ; finds more DMARC results than the split that respects quotes and comments. The second is when the header was too long to read). A sender could therefore end the comment and add a result of their own to the Authentication-Results and ARC-Authentication-Results headers that Rspamd writes, for example dmarc=pass header.from=bank.example. Comments are now escaped, \ is escaped in quoted values, and control characters become ?. The reader honours quoted-pairs in comments, unquotes quoted values, and records the method of each element; of an empty tag-spec (;;) as part of the next tag name, while arc.lua splits the header on ;. A seal that ended in ;;d=trusted.example was verified with the key of the first d=, while arc.lua took the sealer to be trusted.example. Empty tag-specs, and tag-specs without =, are now skipped--b, an empty line, --b) is empty. The parser treated such a part as an error that ended parsing of the whole message. No part after it was parsed or scanned, at any nesting level, and nothing was flagged. Such a part is now skipped. The leftovers pass also continues after a garbage segment[]{};,, and in that case also encode %, so the joined form can be decoded without ambiguity. Several plugins used to join multiple values into one comma-separated option: asn, reputation, rbl, dmarc, content rules and MULTIPLE_UNIQUE_HEADERS. They now insert each value as a separate option. Rules and external tools that parse these options, or the joined headers, may need changes (#6344)-ip4:1.2.3.4 +ip4:1.2.3.0/24 -all let 1.2.3.4 pass, and so did -all ip4:1.2.3.4. The order of the record is now kept. An include was copied in with the qualifiers of its own terms, and the include's own qualifier was dropped. So -include:x passed whatever x passes, and a fail inside x became a fail of the outer record. A match inside an include now takes the include's qualifier when the included record passes. Otherwise, evaluation continues after the include, as RFC 7208 specifies. A redirect is applied after all mechanisms, wherever it appears in the record. This changes SPF results for records that relied on the old order. In Lua, elements that only end an include carry a jump fieldl=0 was treated as if no l= were present. It means that no body is signed, so such a signature is no longer valid for a message that has a body;k= declares: an RSA key, or an EC key on a named P-256, P-384 or P-521 curve;xp= is no longer read as p=d= domains are lowercased: Domain names are case-insensitive, but an ASCII d= tag was stored in whatever case the signer used. Results, symbol options and case-sensitive matchers such as glob maps all saw that case. A signer could therefore evade a glob-based lookup just by changing the case of d=. The parsed domain is now lowercased. Non-ASCII domains were already folded by IDNA conversion. Verification is not affected, because it hashes the raw header. Maps and rules that list mixed-case d= values need lowercase entries (#6350)INVALID_FROM_8BIT and INVALID_RCPT_8BIT (6.0) fire only for addresses that contain 8-bit characters, which matches their names. An all-ASCII address can still break draft-ietf-mailmaint-smtputf8-syntax, with an A-label atom in the local part or a control character. Such addresses get the new INVALID_FROM_ASCII_ADDR and INVALID_RCPT_ASCII_ADDR symbols, scored 1.0. These addresses are valid under RFC 5322, and the draft still lists its A-label rule as an open issue, so this is weak evidence on its own (#6345)multipart/digest, the default is an embedded message. The default applies only to data that is not binary. A child is not text if control characters appear in more than 1/32 of the first 4 KB of its decoded data. Executables and archives are therefore no longer read as garbage text. A child without headers now starts after the empty line that ends the headers, so lua_magic sees its data from the first byte. Short-text fuzzy hashes of headerless children change. Each child of a digest without Content-Type counts as two parts against the parts limitget_files(n) and get_files_full(n) count files, not names: Rules read only a limited number of archive names. CHECK_ARCHIVES reads the first 10, and mime_types, multimap and the scanners read 1000. A zip lists the other names of a file (Unicode Path) right after the file, and lists the names of its local headers after its directory. Names could therefore crowd out files: nine harmless names of a first file pushed a second payload.exe out of CHECK_ARCHIVES. The functions now read the first n files of the directory and the first n files of the local headers. Each file is followed by its other names, flagged as aliases (RSPAMD_ARCHIVE_FILE_SAME_ENTRY, alias in get_files_full). An entry has at most 9 such names, so reads stay bounded. Some rules count the files of an archive, such as SINGLE_FILE_ARCHIVE_WITH_EXE and the single-file checks of mime_types. These rules skip aliases, while extension checks still see themarchive.unpack is bounded by default: All limits of archive.unpack were off unless set explicitly. Without options, output is now limited to 1 GiB and to one million headers. Zero still means no limit. Only filters that libarchive implements itself are used. "All" filters used to include filters that run an external program on the data (grzip, lrzip, lzop). A numeric key in files was converted to a string in place while the table was being walked, which breaks lua_next. This is also fixedinflate() in the rspamd_util Lua module returns errors instead of logging them: The function logged an error for every stream it could not read. Such streams come from the files being parsed, and a single PDF can contain thousands of them if the sender wants it to. The error is now returned as a second value, for the caller to handleurl:get_redirected() and url:get_phished() are now distinct: get_redirected was an alias of get_phished, and both returned the linked URL in every case. A redirect target was therefore reported as the phished URL, and the text of a link as a redirect target. Each function now answers according to the flags of the URL. A redirect that is set later replaces the shown URL, so such a URL has no phished URLtask:get_urls() leaves images out by default, so an image with the same address as a link hid that link from rules. The flag now stays only while every occurrence of the URL is an image source, and such links are returned by get_urls()task:get_date{format = 'message'} returns 0 when the Date header is missing or invalid. The time('message') selector returned 1970 for this value. The spamassassin plugin's check_for_shifted_date read it as a date far in the past, so rules like DATE_IN_PAST_96_XX matched mail without any Date header. Both now return nothing, as SpamAssassin does. More dates are now rejected as invalid (see the date parser fixes below), so this case occurs more oftenET inside a string, BT at the start of a stream, 32 KB blocks, and, without Hyperscan, CRLF line ends. Other changes:Td, Tm) produce a space or a line break depending on the distance, so words placed one by one are no longer glued together;flags.invisible_text.BAD_DKIM_SIGNER multimap rule: The d= of every checked DKIM signature, taken from DKIM_TRACE, is matched against the bad_dkim_signers glob map served from maps.rspamd.com. Failing signatures also match, because a forged signature that claims a listed domain only hurts its sender. Glob entries cover signers that rotate subdomains (#6350)HTML_TINY_PIXEL_TRACKER and the HTML_TINY_PIXEL_PROBE composite: Bounce and address probes typically have no content, and their only images are tiny external tracking pixels, with no links at all. HTML_TINY_PIXEL_TRACKER detects this. It uses the images_tiny_external feature, which the HTML parser already collected but no rule used. Details:DATE_FAR_FUTURE_NO_TZ: A Date header without a timezone is read as UTC+14, the earliest instant any sender could have meant. This is done by appending +1400 and passing the result to the regular date parser. The symbol fires when that instant is still more than 48 hours in the future. The detection is conservative: the weekday is not cross-checked (#6359)MIME_UNREADABLE_ARCHIVE (1.0) fires when a part starts as an archive of a known format but its file list cannot be read. The old code branch for this case was never reached. MIME_TRUNCATED_ARCHIVE (0.0) fires in two cases. The first is an archive with more names than are checked: 1000, or the per-message names limit. The second is a document package with more parts than are checked. An empty 7z file is not counted as unreadable, and neither is a volume of a split archive. Several name checks were easy to bypass:::$DATA are dropped, so evil.exe is exe;x.docx was not checked at all. It is now skipped only when its content really is that kind of document;MIME_BAD_EXTENSION;EXE_IN_ARCHIVE looks at the first 1000 names instead of the first 10;TEXT_WORDS_TRUNCATED: A long text part is represented by its first words and a sample of the rest. When it is long enough, none of the rest is included. A message also has a word budget shared by all its parts. Nothing indicated that either limit had been applied, so statistics, fuzzy hashes and word rules treated a message built around these limits like any other. The part now carries a flag for each case, and get_stats() of a text part reports words_sampled and words_truncated. The TEXT_WORDS_TRUNCATED symbol (no weight) reports both. The limits themselves are unchangedPDF_NO_OBJECTS: the file has PDF markers but no objects;PDF_DECODE_LIMIT: a stream has more filters, or more decoded data, than is processed;PDF_UNDECODED_STREAMS: a filter failed on some streams;PDF_PARSE_ERRORS: some dictionaries cannot be read, or some objects raised an error;PDF_TEXT_TRUNCATED: the file has more text than is extracted.OOXML_INCOMPLETE: An Office document has more parts than are processed (file:truncated), or parts that could not be parsed (file:failed=N). No weight by defaultSVG_INCOMPLETE: An SVG image has text, links or embedded documents over the limits (file:text, file:urls, file:payloads), or an embedded image that was not read (file:nested=reason). No weight by defaulthidden attribute, or by a rule with a single compound selector goes to the invisible text. Common implied end tags are honoured.archive.unpack options: With ignore_case, the names in files select members regardless of the case of their ASCII letters. This matches how applications match the parts of an OOXML package. filters = false refuses an archive wrapped in a compression filter. The OOXML handler sets it, because a package is always a plain ziprspamadm statistics_dump dump --user: The repeatable --user option dumps only the statistics of the given users from a per-user classifier. The user prefix is built from the classifier prefix pattern (%s, %l, %r/%u), the same way the Redis backend builds it. Membership is checked with SISMEMBER instead of fetching the whole <SYMBOL>_keys set. The emitted _keys lines list only the selected prefixes, so the output can be restored without registering other users (#6343)keepalive set for a backend, the proxy did not close a connection's socket, because the connection might be in the pool. The pool only dropped its reference. Sockets that were never pooled, or that left the pool, therefore stayed open. In this mode connections now own their sockets. A pooled connection taken for a mirror whose message cannot be copied is releasedrspamd_http_parse_date and rspamd_pubkey_from_base32 treat a length of 0 as meaning a NUL-terminated string. Header values are not NUL-terminated, so an empty Date, Last-Modified or Key value made them run strlen past the end of the header storage. An empty date is now treated as no date, and an empty key is not decoded/checkv3 metadata is not bound by the HTTP header size limit and can repeat a name hundreds of thousands of times. This kept a worker busy for tens of seconds before the task timeout even startedowner= and group= tokens from the client made it look up users and groups. The path is now taken as it isidna_convert_utf8() with a pool argument: The Lua method used its userdata as the memory pool, while the userdata only holds a pointer to the pool. The converted name was allocated in that invalid pool, which corrupted the heap and crashed the worker shortly afterwards. The bug needs a domain with 8-bit characters and a pool argument. The ARC module passes a pool when adjust_dmarc is used with a map of signersadd_cmd on a released connection and a request without a command no longer crash, and connect_sync honours no_pool;12, and database 1 with password 2, therefore shared connections. Each field is now hashed together with its length;max_conns connections to one server are in use at once. A server that did not answer used to get a new connection for every requestssl, ssl_ca, ssl_cert, ssl_key and sni settings and connected in plain text, which a TLS-only Redis rejects. This affected SCRIPT LOAD requests, the synchronous script loader, rspamadm statistics_dump and the fuzzy storage Redis backend. They now use these settings (#6347)connect() failed with anything other than EINPROGRESS, and nothing created it again later. Every truncated UDP reply was then delivered as is for the lifetime of the process. SPF therefore failed for any TXT record set larger than the UDP limit. This happens when the resolver is not yet reachable at startup. Examples are "No route to host" from an eBPF service load balancer such as Cilium, or ENETUNREACH while the network is still coming up. The channel is now kept, and the next truncated reply reconnects it (#6355)poll_timeout * 20 before the next try, which is 100 minutes with the default interval. A map whose first fetch failed because DNS or the source was not ready yet stayed empty for that long. Failures are now classified:Retry-After are retried when the server asks, within 2 s and 20 times the poll interval. A 429 without Retry-After counts as permanent.Expires and the other fail with 429 Retry-After: 30. The next attempt then came 3599 seconds later. A failure now decides the retry time on its own. Retry-After: 0 was treated as a missing header, which turned a 429 into a permanent failure. Zero, and dates in the past, now retry at the 2 s minimum. Three failure paths never reached the scheduler and leaked the periodic timer, so the map was not polled again until restart:a:%{l}.example with the local part host/0 therefore passed any address. a//64 gave IPv4 a prefix length of zero, a/65536 wrapped to zero, and garbage after a length was ignored. Such terms are no longer used. Mechanism names are compared in full (+allfoo was read as +all), and so is the version (v=spf10 was accepted). ip4=1.2.3.4 and include=domain are mistakes found in records of legitimate senders, and they are still read as the sender meant themip6=: a/24//64 was not read correctly. After the IPv4 length, one of the two slashes was consumed and the other became a bad length. The term was dropped, and the domain fell back to its all. The second slash is now required and consumed, and a/24/64 is not a valid term. ip6= never worked for an address, because the parser found the first colon of the term, which is part of the address, before the =ptr now use the address that the record is checked for. With external_relay, this is not the address of the connection. A flattened record has at most 8192 elements and is cached for a week at most. A negative TTL is no longer accepted as a TTLrspamadm dnstool spf-flatten follows the record order: The tool stopped collecting networks at the first all. With the new ordering, that is the all of the first include. The tool also wrote out the all of each include. It now skips elements that only end an includev=DMARC1; x1=a; p=reject therefore published no policy, and spaces around = had the same effect. With two tags of the same name, the result depended on their order, or on a table walk when they differed in case. Unknown and malformed tags are now skipped, and the first tag of a name wins. pct must be a whole number from 0 to 100. nan, inf and negative values used to be accepted. rua is limited to 1024 bytes and must not contain control characters, because it is stored in a key for every message of the domain. Fields of aggregate reports are escapedd= may hold the A-label form of the same domain, and the two forms were compared as written. The DMARC module now uses the A-label form of the From domain everywhere: in alignment, in the policy lookup and in reports (#6346)rspamadm dmarc_report renamed keys to *_processing but ignored the result, so overlapping runs could send the same report twice. Keys are now claimed with RENAMENX, which refuses instead of overwriting, and a run skips any date or report that another run has claimed. Claimed keys are read from the write server. A lagging replica could return them empty, and the data would then be deleted without being sent. The reporting Redis script hard-coded a two-day expiry. It now uses reporting.keys_expire (#6348)lua_mime.modify_headers: Headers added through lua_mime.modify_headers were not included in DKIM signatures. When a header that the message already had was modified for the first time, only its first instance was copied into the modified view that DKIM signing reads. A header added next to two existing ones was therefore signed as if there were only two, which broke the signature. Every instance is now kept (#6336, #6312)\v and \f as whitespace, so these characters could be added to a signed message. Only space and tab are whitespace now. The signature tag was matched as any b= at the end of a tag name. As a result, xb= was left out of the hash and b = was not, which made a valid signature fail. The tag is now found by its whole name. The scan also read one byte past the header, which is past the end of the message when the signature header is the last oneDMARC=pass or dmarc/1=pass therefore gave no DMARC result. Elements are now read as RFC 8601 method specs (method, version, result), and property names are lowercased. The method is taken after leading comments, which RFC 8601 allows, so (forwarder result) dmarc=pass header.from=... is read correctly. An empty element (;;) no longer hides the results that follow iti= tag: The header for an instance was the first header that contained i=N; as a substring, so text in another tag or in a comment could select the wrong header. The tag list is now parsed properly. The ARC-Seal header list is sized from i= (3 * i - 1 headers), but it was built before i= was checked, so a seal without i= reserved space for 2^32 headers. i= is now validated first. Instance numbers of 0, or above the ARC limit, are refused, so a value above UINT_MAX is no longer truncated into a valid instanceb4-- part reopens the --b4-- line of its parent. Each such part scanned the rest of the message. The step budget that bounded this could be used up by thousands of cheap parts placed in front. After that, recovery was skipped and the content behind them was not scanned. Candidates are now looked up by hash in an index that is built on demand, and the result equals that of an unlimited scan. 9000 such parts with 90000 candidates take 0.2 s instead of 14.7 s.eml child raised the nesting counter and never lowered it again. After 64 sibling containers, the next container was refused as too deeply nested and the rest of the message was dropped. Content placed after enough harmless containers was therefore never scanned. Reaching the nesting limit also ended parsing of the whole message. Now only the content past the limit is skipped, and the message is still flaggedapplication/pkcs7-mime signed-data part was parsed with the boundary candidates of the outer message. A multipart inside it therefore had no parts, and its text and URLs were not scanned. It is now parsed in a runtime of its own. The decoded ASN.1 structures are freed first, so nested S/MIME no longer keeps them at every levelContent-Transfer-Encoding was applied to children that had none. RFC 2045 allows only 7bit, 8bit and binary on a multipart. With base64 on the multipart, readable children were decoded into garbage. With quoted-printable, decoding could open an HTML comment around a link. Only 7bit and 8bit are now passed to children. Base64 is passed only to a child whose whole data is base64GIF89a at the start made it an image, and %PDF-1.x in its first KB made it a PDF. <?xml, BEGIN:VCALENDAR or BEGIN:VCARD turned text extraction off. Such a part now stays text when two conditions hold. Its data must not be binary, meaning control characters in at most 1/32 of the first 64 KB. It must also not be an attachment named as the detected format. The content modules still read the part, so PDF_* and SVG_* symbols still fire. Text formats (XML, SVG, CSV, iCalendar, vCard) stay text whatever bytes they contain. Binary data that clients still show as text now has its links read as well. Examples are a binary text/plain part, or a headerless child that starts with binary bytes. Exceptions are attachments, parts named as the detected format, and images that the HTML shows by Content-ID. R_BAD_CTE_7BIT ignores the 8-bit bytes of a document that a content module has readhttps://evil.com was missed;URI_COUNT_ODD checks.unicodefeff and unicodefffe are read as UTF-16LE and UTF-16BE. Because clients disagree, links in the other byte order are read as well. Padding can therefore no longer select a byte order that hides a link most clients show. A link found only in the other byte order is added. A link found in both keeps the name that the other reading shows for it, and is not counted twiceBROKEN_CONTENT_TYPE on 5 ham messages, and none after the fix. A message or embedded message without Content-Type is read as text, whatever it starts withname*=, and each RFC 2231 piece up to 999. The first parameter that matches a slot takes it. Values that only name a charset, or that end the pieces, are also kept for the resolver. After 1024 junk parameters, filename*=utf-8''invoice+AC4-exe; filename*=utf-7'' is therefore invoice.exe, as in Thunderbirdname*= or a piece, it is resolved as Thunderbird does: name*= if it decodes, otherwise the pieces joined from 0 up to the first missing one, otherwise the plain value. Pieces used to be joined with any other value of the name, so filename*1=w; filename="a.exe" was read as a.exew. Other rules now follow the readers:% followed by two hex digits, so a+b.exe stays a+b.exe instead of becoming a b.exe;foo=;) stopped the Content-Disposition parser, so attachment; foo=; filename="invoice.exe" had no filename. The parser now continues past it. Resolving RFC 2231 names zeroed an array of 1000 pointers for each name, and a header can hold up to 1024 names. Short chains of values now use a short list instead. Lowercasing a value that gets shorter keeps the new length;;(()) was never closed, and the rest of the value was dropped as a comment;for addresses in Received headers;=2=20). This corrupted mail that Rspamd rewrites: footers, the 8bit to quoted-printable downgrade in lua_smtp, and metadata_exporter output. In CR mode, a soft break written right before a literal LF was decoded as a soft CRLF, so the LF was lost. No soft break is written there nowiso-8859-16, l10 or ISO_8859-16 therefore failed to convert. Every alias now uses the internal table;INVALID_DATE;+0060 was read as one hour;UTC is accepted as a zone, and comments may contain quoted pairs such as \);049 was read as 2049. Zero-padded four-digit years such as 0102 are read the same way, since mailers that print tm_year with %04d write themnreceived_bad_time: On error, the date parser returned (uint64_t)-1, and that value went straight into the Received timestamp. A bad date now gives 0, the same as a missing date. 0 is true in Lua, so in the meta features that neural networks use, a dateless hop added its distance from 1970 to the sum of hop delays. Such hops are now skipped. The token keeps its meaning and position. The lua_meta version and the neural Redis prefix are therefore unchanged, and trained networks are kept. About 7% of tokens change in a sample of about 14000 messagesrspamd_gmtime stored days, seconds and years as unsigned values, so any time before 2000-03-01 wrapped around: 0 came out as year 438499967. rspamd_tm_to_time put dates in non-leap years between 1902 and 1967 one day late. It also treated century years not divisible by 400 as leap years. The callers format fuzzy reply times and HTTP dates<p><font color=white>one<p>two, both lines are white. These elements are tracked until their own end tag, and a table or a cell clears them; does. A three-digit colour without # follows the legacy rules, so fff is #0f0f0fget_html_urls, and a repeated link is not parsed again;svg element that is open at the limit still makes its links read xlink:href;href, src, style and similar attributes, and a link after 128 junk attributes was never checked. Interpreted attributes are now always kept, and a repeated one is ignored, as browsers do. Only other attributes count against the limitsscript was decoded into a script element, so the following content was hidden as raw text. For browsers it is an unknown element, and its content is shown. In <a href="good" href="evil">, the destination was taken as good. Names are no longer decoded. After an attribute without a value, the next attribute name lost its first character: a /style gave /style, and a b style gave bstyle. A new name now starts the way it does in browsershref is followed only for links and action only for forms. In <a action="good" href="evil">, only good was extracted;src is the destination of iframe, frame, embed, audio, video, source and track elements. Browsers load it and ignore any href on these elements;formaction on submitting buttons and inputs is a link;xlink:href of an SVG <a> without href is followed, extracted, rewritten and listed;src of a link whose href is empty<base>, every link that was not absolute was appended to the base. <a href="//evil.com/x"> under <base href="https://bank.example/"> was therefore taken as bank.example, out of reach of the phishing check. Resolution now follows the browser rules:/	/evil.com, \\evil.com and HTTPS:\\evil.com all lead to evil.com>. The URL is still the part before the quote, because href=3D"... from undecoded quoted-printable is a relative link. The exception is a quote inside the authority followed by @: href=https://www.paypal.com"@evil.com/ is now taken as evil.com. URL rewriting replaces the whole value. The rewrite policy and task:get_html_urls() receive the whole value too, which may differ from the text that task:get_urls() returns. Tags with broken markup that browsers accept are now rewritten and listed. Examples are <a/href=...>, a stray quote and a missing closing quotecolor, bgcolor and hidden attributes took precedence over author CSS. Any author declaration of the same property now replaces them. visibility:hidden was compiled into a hidden display that no descendant could undo. It is now a separate inherited property, so a descendant with visibility:visible is showndisplay. opacity: 0; display: block !important was therefore read as visible text. These ways of hiding now have their own flag, which no display value resets. A stylesheet rule that sets display on a descendant no longer makes the content of a display:none ancestor visible. Unknown elements now take class, id and universal rules, because browsers render them as inline elements> gave > plus U+20D2. References to C1 controls are now read as Windows-1252, so ’ is a right single quote. An unfinished entity at the end of the text was copied twice<Preview> pads the preview text with U+200E and U+200F, among other invisible characters. Every padding unit counted as two hidden characters, so short previews crossed the HIDDEN_TEXT threshold. These marks are now ignored (#6374)<img width="0" height="0"> trackers are detected. Sizes come from inline styles and stylesheets through the CSS parser. They follow the cascade, !important, and minimum and maximum bounds, and physical units are converted to pixels. Relative units and functions such as calc() mark the size as unknown. Relative image sources are resolved against the document base, as browsers resolve them. cid: and data: are matched case-insensitively*/;@media all, screen or only screen are applied;!important stylesheet declaration now wins over normal inline styles and over more specific rules. The flag used to be set on a copy, and for the wrong case. Specificity was a single sum, with 100 per id and 10 per class, so eleven classes outranked one id. Ids, classes and types are now compared in that order1. and 1e+ are not numbers, while +.5 is, and long runs of digits are no longer cut;-;url creates a URL token, so a ; inside urlx(...) does not end a declaration;calc((1px) + (2px)) is one component.hsl() without saturation turned every grey into black, so white-on-white text written this way was not detected. A hex colour with a digit that is not hex is not a colour, and the four-digit form with alpha is now accepted. color and bgcolor attributes are read in any case, with spaces around the value and without #@ signs, the host starts after the last one in the authority. A backslash ends the authority, so http://a.com\@b.com/ is a.com;%2F, %3F) in the authority stay encoded, so http://a.com%2F@b.com/ is b.com;http://0177.0.0.1/ was reported as 177.0.0.1 and was not marked as obscured, while browsers go to 127.0.0.1. Octal, hex and short forms are now read as the URL standard specifies. Every form other than plain dotted decimal is marked as obscured, and an out-of-range number is not an address. When an IPv6 host is rebuilt, it keeps its closing bracket and the slash of an empty pathClickhttps://);www. without a scheme.sip:sip:sip:..., each scheme scanned the rest of the token for @, so a 200 KB line took seconds. Candidates now share the result of that lookahead. A file:// at the very end of the text made the scanner read past the end. When max_urls is reached, the task now sets the urls_limit_reached variable, so rules can see that the URL set is incomplete. The limit is logged once per messageurl:to_http() percent-encodes control characters and spaces again. Previously, %0d%0a in a path came out as a raw line break. The function also no longer leaks the buffer for URLs with userinfo;url.create with text that contains no URL returned the text itself as a URL;set_redirected raised an error when given such a string;task:inject_url dereferenced a null pointer when its third argument was userdata other than a MIME part;callto scheme was set to the length of teltask:get_urls() sampling: Over max_lua_urls, each URL was kept with a probability based on the whole set, including URLs that the call filters out. Adding email addresses to a message could therefore make its links disappear from rules. The URLs that pass the filter are now counted first, and exactly the limit is drawn from them. The draw uses a per-process seed, so a sender cannot predict which URLs are kept. The plain array form of protocols was ignored, sort = false still sorted, and the comparator returned a number, which table.sort does not accepterror_expire (10 minutes);lock_wait (3 seconds) and then resolves the URL itself;URL_REDIRECTOR_LIMIT symbol reports how many redirector URLs exceeded max_urls and were not followed;Location is read as browsers read it: backslashes count as slashes, and tabs and line ends are removed. Two Location headers no longer leave the chain unfinished, and a Location written into a symbol option is cut at 128 bytes0xffffffff without zip64 records, as written by zip -fz to a pipe;cat a.zip b.zip). Both archives are listed: the last one, which Python and unzip list, and the first one, which 7-Zip and WinRAR list. The first archive's entries are walked as 7-Zip walks them;payload.exe\0.txt as payload.exe.invoice.pdf could extract as invoice.exe unnoticed;invoice\xc0\xaeexe is invoice.exe;evil.exe was listed as il.exeMIME_UNREADABLE_ARCHIVEg_string_hash or a zero seed, so it was easy to create colliding names. A zip with 16384 such names took a second to list, and the time quadrupled with each doubling. A random per-process seed is now usedMSCF or @echo off anywhere in a program made it a cabinet or a batch file. The MS-DOS header pointer is now followed;[Content_Types].xml and a directory of its kind. Two xl/... members no longer make any zip a spreadsheet;ole typex.js or x.hta, is opened by its name. A part named as a different kind than was detected is therefore now scanned. A document whose detected kind differs from its declared type is read by the handler for the detected kind, for example a PDF declared as a spreadsheet. x.jpeg detected as jpg no longer counts as named as another kind, which used to send every such image to the scannersendobj, so files with spaces there had no objects. Whitespace in the header, a missing endobj, and markers inside streams of known length are now handled. Other fixes:null, hex strings with whitespace, references with other than single spaces, comments, empty names or padding;/#4AS is /JS, and string escapes follow the PDF format;/OpenAction, /A, /AA) are processed, and launch actions are detected, including those with only a /Win dictionary;/Encrypt is found whatever follows the key, and in any trailer/N declares, and the deadline is checked for each decoded stream. 98 such streams used to run for 6 seconds against a 2-second limit;%REF%, and in octal escapes over 255 no longer stop processing of later objects. Each object is processed under its own protected call/Differences array stored as a separate object is honoured, and it no longer counts codes past 255rspamd_util, a z stands for four zero bytes, but the output was sized for five characters per four bytes. Data with many zeroes therefore failed to decode. The decoder also accepts an optional output limit/ are accepted, and part names are built only from validated segments;href takes precedence over xlink:href. Tabs and line ends are removed before a scheme is read, so java	script: is detected. Style sheets of any size are scanned in pieces, whereas 64 KB of padding used to hide an import. Presentation attributes with URLs are scanned too. Text or links over a limit are left out, and the rest of the document is still read, with a flag. An SVG sent with a generic content type is handled by its name, and nested images are read up to three levels deepATTACH, CONFERENCE, COMMENT, CONTACT and X- properties are now checked for URLs, because applications put meeting links there. A line without a colon was read as the start of the next property's name, and that property's URLs were missed. A megabyte of such lines took 23 seconds to process. Such lines are now skipped. Every line used to be kept as a table for the lifetime of the task. Now the first megabyte is read, 10000 properties are kept, and truncated is set when there is moreR_MIXED_CHARSET is the average badness over the words of a part, so plain words diluted a mixed one: pаypal among eight plain words did not fire. The sum for mixed-script words is now divided by at most 24 words. Letters with diacritics still count as a share of the part, so a long English text with names like Müller no longer fires. R_MIXED_CHARSET_URL converts xn-- hosts to their displayed form before checking themlua_mime.extract_text_limited runs in linear time: The text sent to external services was trimmed with a pattern that took quadratic time on long runs of spaces. No text at all was returned when the first line exceeded the byte limit. Lines are now cut to the limit and trimmed without a patternmax_html_len is written to its own field: The option was registered at the offset of max_word_len. Setting it changed the word length limit and zeroed words_decay, while the HTML limit stayed at its defaulttextpart:filter_words(): The full mode aborted on a part that contains a URL, because the URL's placeholder word has no normalised text. A custom tokenizer no longer moves backwards in the text when two exceptions overlaptask:set_recipients() and task:set_from(): set_recipients with the rewrite flag freed the wrong list and leaked the replaced one. set_from overwrote the original envelope sender instead of keeping it next to the rewritten one, and leaked the previous rewriteclassifier { backend = "redis"; ... } are now supported. Glob characters in prefixes are escaped in SCAN MATCH, so a user name containing *, ? or [ no longer matches tokens of other users (#6343){abcd, bc, cd}, cd was not found in abcd;ZSTD_isError, and the output buffer leaked on errors. A truncated frame made the output grow forever. Truncated input is now an errorrspamd_strtol and related functions treated an empty string as 0 and did pointer arithmetic on NULL. An HTML attribute without a value reached them this way. The hex overflow check used the decimal cutoff, so long hex numbers could wrap around. An opacity with no number in it is no longer treated as 0max_tag_len and tag_strip_policy (#6300)unknown worker type when the configuration was read from the source tree, as the functional tests do. The section is now skipped in such builds. An unknown worker type now reports its name in the errorL·L is accepted, not only l·l. Builds with older ICU give the same verdict as newer ones (#6345)X-Mailer: iPhone Mail (24B91016). The regexp allowed at most four digits after the build letter, so FORGED_X_MAILER (4.5) fired on every message from such devices (#6353, #6349)U8_NEXT takes int32_t offsets. backward-cpp now builds against the libbfd of binutils 2.39 and laterl=0 over a body, and keys under 1024 bits;d= is lowercased;Recommended upgrade for all users.