We have released Rspamd 1.8.1 today.

There are couple of the incompatibilities introduced, hence, please check the migration notes. These incompatibilities could affect almost any Rspamd installation with custom configuration, so please read the upgrade notes carefully.

The most important features and fixes

• Critical fixes in DMARC module

  DMARC module could insert temporary failure symbol instead of strong rejection in certain cases. This has been fixed in 1.8.1. There are many other bugs fixed in this module since the whole check logic has been rewritten in accordance to RFC.

• Fuzzy check

  Due to bug introduced in 1.8.0, there algorithm used to deterministically skip words in large text parts was not deterministic. It means that the exact words pipelines produced by different Rspamd instances might be different. It could affect if your words_limit was reached (default: words_decay = 200 words). Hence, for large text parts it was expected to have misses in fuzzy and in Bayes classification. Whilst bayes missing should not be significant, the fuzzy misses might be very severe and they might break fuzzy detection for large text parts.

  In 1.8.1, we have fixed this issue and, since we have already broken the compatibility with 1.7.9, we have decided to increase words_decay to 600. Please ensure that you don’t override this parameter anywhere (e.g. in local.d/options.inc, override.d/options.inc or any other override or local file) or your compatibility with Rspamd fuzzy storage would be lost for messages with more than words_decay threshold words.

• Various language detection issues have been fixed
• Experimental clustering plugin
• Important fixes for the dynamic ratelimits
• Fix options insertion
• Plug memory leak in redis pool
• Add check_violation feature to DKIM/ARC signing to avoid signing messages when there is an existing invalid DKIM signature
• Add only unique elements to Clickhouse url arrays
• Allow g+: and g-: composite atoms to include symbols with positive score and negative score
• Allow dkim domains check in surbl module
• Allow maps with HTTP auth using standard URL syntax
• Allow to disable actions by users settings by setting them to null
• Extend whitelisting options:
  • Allow per element whitelist/blacklist only behaviour
  • Introduce three types of lists:
    • Blacklists (bl)
    • Whitelists (wl)
    • Bidirectional lists (both)
• Use heuristical verdict instead of the plain action in plugins to detect if a message is clearly spam, junk, ham or uncertain when there is not enough confidence despite of the action being set
• Various Web Interface improvements and fixes driven by Alexander Moisseev

Full list of the meaningful changes

• [CritFix] Fix options insertion
• [CritFix] Fix words decay one more time (affects long messages)
• [CritFix] Increase default words_decay
• [CritFix] Plug memory leak in redis pool
• [Feature] Add check_violation feature to DKIM/ARC signing
• [Feature] Add only unique elements to Clickhouse url arrays
• [Feature] Allow g+: and g-: composite atoms
• [Feature] Allow dkim domains check in surbl
• [Feature] Allow maps with HTTP auth
• [Feature] Allow to disable actions by users settings
• [Feature] Extend whitelisting options
• [Feature] Store url object in images
• [Feature] Use verdict instead of the plain action in plugins
• [Fix] Allow to call fstring append with NULL string
• [Fix] DCC - luacheck
• [Fix] Do not load torch on each rspamadm invocation
• [Fix] Fix boundaries detection and rework stop words algorithm
• [Fix] Fix dependencies for DNS_SIGNED symbol
• [Fix] Fix errors when dealing with dynamic rates/bursts in Ratelimit
• [Fix] Fix groups mess
• [Fix] Fix groups mess
• [Fix] Fix parsing address with comments
• [Fix] Fix resolving in DMARC reports
• [Fix] Fix various issues with parsing of the received headers
• [Fix] Fix watchers issue in lua_tcp when doing no resolving
• [Fix] Plug memory leak in language detector (affects reloads)
• [Fix] Remove one letter stop words
• [Fix] Slashing: backport chunk logic from libucl
• [Fix] Stop libevent from using cached time in rspamadm
• [Fix] Try to fix watchers chaining
• [Fix] Various fixes in redis sync interface
• [Fix] ip_score - respect check_authed and check_local settings from config
• [Project] Rework passthrough actions
• [Project] Clustering module
• [Rework] Always create result for a task
• [Rework] Completely rewrite DMARC checks logic
• [Rework] Rework and fix whitelist plugin
• [WebUI] Add symbols sorting buttons
• [WebUI] Change symbols order without updating history
• [WebUI] Colorize symbols
• [WebUI] Do not display password form when secure_ip is set
• [WebUI] Fix symbol description tooltips display
• [WebUI] History: add sorting by symbol score value

We have released Rspamd 1.8.0 today.

There are couple of the incompatibilities introduced, hence, please check the migration notes, especially if you use clickhouse module or users settings.

The most important features and fixes

• New selectors framework

  This framework allows to combine and process different data extracted from messages and use that in different plugins, such as multimap, reputation or ratelimits. It is also possible to use data extracted in Rspamd regular expressions.

• Coroutines API support in Lua

  Now you can write code in a usual imperative manner but you still will not block any other tasks. Each potentially blocking operation creates a yielding-point. In turn, this means the code is suspended until the operation is done (just like blocking) and resumes only when there is some result. Meanwhile, other tasks are processed as usual.

• Clickhouse optimization

  Rspamd now uses a flat table to optimize ClickHouse SQL requests. In fact, joins are not recommended by the ClickHouse developers as multiple joins have proven to be slow. Hence, Rspamd has moved all data to a single table. Schema migration is done automatically, however, please read the migration notes in case of any doubts. Old data is not migrated nor deleted automatically.

  There is now optional data retention support in the ClickHouse module. You can set retention policies for the data stored in Clickhouse to conform different regulations (e.g. GDPR).

• Unicode processing improvements

  Rspamd now normalizes all unicode data using NFKC schema prior to processing. This helps to prevent “glyph” attacks used by some spammers nowadays. Unicode conversion has also been improved to continue on bad symbols instead of giving up and working with raw data.

• Language detection improvements

  We have reworked the language detector to use stop-words and rely on unicode glyphs more extensively. As the result of this work, the speed of language detection has been increased significantly (by 10 times in some cases). The preciseness of the detection has also been improved.

• Fixed various bugs in sesssions handling

  We have located and fixed various hidden issues caused by async rules chaining. It might cause inconsistencies in the dependencies processing, crashes in rare cases and other “bad things”.

• Various Web Interface improvements and fixes

  There are multiple improvements and fixes in the Web Interface. In particular, the issues with cluster support and aggregation have been addressed.

• New mailing lists for the project

  • Rspamd-Users: general purpose mailing list that will replace this group
  • Rspamd-Announce: read only list with low traffic that is intended for project announcements only, for example, new versions, or vulnerabilities disclosures

  Unfortunately, there is no automatic conversion from the Google groups to the new mailing lists, hence, you need to subscribe to those lists manually. We are sorry about the potential inconveniences caused by this transition.

  To subscribe to a list, click the list name at https://lists.rspamd.com/mailman/listinfo . The page that is displayed should contain all of the necessary subscription instructions for that list. You can always find all possible support channels here.

Full list of the meaningful changes

• [Feature] Add arguments schemas to processors and extractors
• [Feature] Add functional selectors library
• [Feature] Add generic selector to reputation module
• [Feature] Add more ratelimits: by digest, by attachments data, by filenames
• [Feature] Add preliminary stop words detection support
• [Feature] Add pure Lua debugm function
• [Feature] Add schema validation for Redis settings
• [Feature] Add selectors combine function
• [Feature] Add some recursion protection to lua logger
• [Feature] Add support for Lua API tracing
• [Feature] Allow to apply schema to arguments
• [Feature] Allow to get dkim signing data directly from HTTP headers
• [Feature] Allow to reuse existing authentication results
• [Feature] Cache selectors results in re runtime
• [Feature] Implement new text tokenizer based on libicu
• [Feature] Integrate selectors framework to multimap
• [Feature] Relax FORGED_RECIPIENTS
• [Feature] Support (almost) all html entities
• [Feature] Support adding and deletion of recipients in the milter block
• [Feature] Support gathering HTTP body from fragments in lua_http
• [Feature] Support multi flag in regexp and glob maps
• [Feature] Support selectors in ratelimit module
• [Feature] Support selectors in settings
• [Feature] Use khash in HTML parser
• [Feature] Use pure Lua debugm function
• [Fix] Add fail-safety for destroying sessions
• [Fix] Allow to add result-less fake DNS records
• [Fix] Another try to fix race conditions on config unload
• [Fix] Call Lua callback on DNS timeouts
• [Fix] Deprecate task:inc_dns_req as it is redundant
• [Fix] Do not allow events deletions on cleanup
• [Fix] Do not try to process skipped messages
• [Fix] Fix HTTP requests with no body
• [Fix] Fix another cleanup race condition
• [Fix] Fix bug in processing of pcre regexps
• [Fix] Fix byte array allocation in the pool
• [Fix] Fix crashes on task cleanup
• [Fix] Fix dynamic buckets in ratelimits
• [Fix] Fix endless loop when waiting for Rspamd to stop
• [Fix] Fix lua_util.str_split in case of delimiters set
• [Fix] Fix more issues with watching of async events
• [Fix] Fix stop words detection and loading logic
• [Fix] Fix various corner cases for language detection
• [Fix] Fix watchers in lua_tcp
• [Fix] Fix words decay algorithm
• [Fix] Implement watchers replacement to handle nested calls
• [Fix] Save faked code into fake dns record
• [Fix] Show the proper frame when using lua_util.debugm
• [Fix] Use fake dns records in tests
• [Fix] Use unicode replacements for HTML entities
• [Fix] fixed “cannot find dependency on symbol 1” issue when using replaced symbols in spamassassin rules
• [Fix] partition_id is not available in old versions of CH
• [Project] Add implicit conversion logic to selectors
• [Project] Add initial support for selectors in regexps
• [Project] Add method concept
• [Project] Further changes in unicode operations
• [Project] Implement Clickhouse migrations
• [Project] Implement implicit conversions to userdata
• [Project] Implement insert method
• [Project] Implement selectors registration for regular expressions
• [Project] Implement selectors support in re_cache
• [Project] Improve language detector: cleanup unused files, categorize
• [Project] Migrate CH data to a fat table
• [Project] Rework selectors logic
• [Project] Start Clickhouse utilities library
• [Project] Start unicode rework
• [Project] coroutine threaded model for API calls: thread pool
• [Rework] Move phishtank to a DNS based service
• [Rework] Rework Clickhouse plugin to use the new API
• [Rework] Rework language detector
• [Rework] Rework utf content processing in text parts
• [WebUI] Add progress bar for AJAX requests
• [WebUI] Avoid errors table reinitialization
• [WebUI] Avoid history table reinitialization
• [WebUI] Avoid throughput summary table reinitialization
• [WebUI] Destroy summary table on disconnect
• [WebUI] Fix “auth” request URL
• [WebUI] Fix disabling and hiding controls on page reload
• [WebUI] Fix maps loading from neighbours
• [WebUI] Fix symbols sorting by score
• [WebUI] Fix tables destroying
• [WebUI] Fix throughput data consolidation
• [WebUI] Fix upload buttons disabling
• [WebUI] Notify user on module loading failure
• [WebUI] Update FooTable 3.1.4 -> 3.1.6

We have released Rspamd 1.7.9 today. There are no incompatible changes introduced with this version to our best knowledge.

The most important features and fixes

• Ratelimits are reworked and now work as intended (and documented)
• Clickhouse module supports data retention policies
• Reworked C modules to avoid global contexts (simplifies leaks detection on reload)
• Reputation plugin now supports SPF records reputation
• WebUI code is now even more conformant to the modern JS standards
• Maps are now distributed remotely with local file safety fallback to allow faster maps update without waiting for a new release
• Antivirus module checks attachments only (as decoded content) in attachments_only mode to improve AV performance by hiding the mime content from them

Full list of the meaningful changes

• [CritFix] Fix caseless comparison of equal length strings
• [Feature] Add HTTP basic auth support to elastic and clickhouse plugins
• [Feature] Add SPF selector to reputation
• [Feature] Add support of the fallback backends for HTTP maps
• [Feature] Allow to print full mime structure when extracting mime data
• [Feature] Allow to split symbols in reputation plugin
• [Feature] Check attachments only on AV scanners in attachments_only mode
• [Feature] Disable all SSL checks if ssl_no_verify flag is set
• [Feature] Implement parsing of scoped IPv6 addresses
• [Feature] Improve rspamc counters output
• [Fix] Add sanity checks when expanding SPF macros
• [Fix] Allow to parse SA rules with no spaces around =~ (dirty hack)
• [Fix] Avoid one extra byte writing
• [Fix] Deal with direct hash table
• [Fix] Detect empty text part as text, not HTML
• [Fix] Do not reduce map watch timeout for mixed http/file maps
• [Fix] Fix HTML part detection heuristic
• [Fix] Fix double free in redirectors cleanup
• [Fix] Fix legacy history handling in the controller
• [Fix] Fix messages insertion
• [Fix] Fix sending string method
• [Fix] Fix statconver command line arguments
• [Fix] Fixed argument checking for being null
• [Fix] Fixed issues reported by luacheck
• [Fix] Freeze updates queue when do actual storage update
• [Fix] HTTP map hash is per-backend and not per-map
• [Fix] Plug memory leak in fuzzy updates
• [Fix] Prefer ‘MTA-Name’ when producing authentication results
• [Fix] Replace bad unicode sequences instead of stopping on them
• [Fix] Set classifier version on learning
• [Project] Reworked ratelimits
• [Project] Apply topological sorting for symbols in Rspamd
• [Project] Remove global contexts from C modules
• [Project] Move performance critical hash tables to khash
• [WebUI] Avoid unused indexes
• [WebUI] Do not execute on_success callback
• [WebUI] Fix history reset for “All SERVERS” (#2346)
• [WebUI] Fix query URL for selected server
• [WebUI] Fix symbols display in legacy history,
• [WebUI] Hide symbols order selector for legacy history
• [WebUI] Refactor query functions into one
• [WebUI] Remove previously-attached event handlers
• [WebUI] Save symbols to the selected server
• [WebUI] Unify arguments of query functions
• [WebUI] Use common query functions to get graph data
• [WebUI] Use common query functions to save symbols

We have released Rspamd 1.7.8 today. There are no incompatible changes introduced with this version to our best knowledge.

The most important features and fixes

• Rspamd mime tool can now show you fuzzy hashes extracted from text
• Fuzzy hashes are now updated when being hitted to prevent expiration of the important hashes
• Fuzzy updates queue is now deduplicated that allows to reduce amount of Redis update requests by 10 times in some cases
• HTTP maps are now cached on disk to provide preload on startup
• WebUI code is now more conformant to the modern JS standards (special thanks to Alexander Moisseev)

Full list of the meaningful changes

• [Feature] Add more extended statistics about fuzzy updates
• [Feature] Add more non-conformant Received headers support
• [Feature] Add preliminary function to get fuzzy hashes from text in Lua
• [Feature] Allow to configure AV module rejection message
• [Feature] Implement fuzzy hashes extraction in mime tool
• [Feature] Improve WHITE_ON_WHITE rule
• [Feature] Improve integer -> string conversion
• [Feature] Reuse maps in multimap module more aggressively
• [Fix] Avoid race condition in skip map as pool lifetime is not enough
• [Fix] Eliminate all specific C plugins pools
• [Fix] Fix DKIM check rule if DNS is unavailable
• [Fix] Fix build where ucontext is defined in ucontext.h
• [Fix] Fix crash in base url handling
• [Fix] Fix descriptors leak in sqlite3 locking code
• [Fix] Fix messages quarantine
• [Fix] Fix padded numbers printing
• [Fix] Fix race condition on maps reinit
• [Fix] Fix regexp functions when no data is passed
• [Fix] Fix specific urls extraction
• [Fix] Fix styles propagation
• [Fix] Improve resetting of the limit buckets
• [Fix] Initialize sqlite3 properly
• [Fix] Work with broken resolvers in resolv.conf
• [Project] Implement HTTP maps caching
• [Project] Refresh fuzzy hashes when matched
• [Project] Add logic to deduplicate fuzzy updates queue
• [WebUI] Add missed declarations
• [WebUI] Avoid using “undefined” property
• [WebUI] Do not accept passwords containing control characters
• [WebUI] Do not redeclare variables
• [WebUI] Enable strict mode,
• [WebUI] Fix variable assignment
• [WebUI] Initialize variables at declaration
• [WebUI] Remove duplicated path from RequireJS config
• [WebUI] Remove unused block
• [WebUI] Remove unused variable
• [WebUI] Remove unused variables
• [WebUI] Use self-explanatory notation
• [WebUI] Use type-safe equality operators

We have released Rspamd 1.7.7 today. There are no incompatible changes introduced with this version to our best knowledge.

The most important features and fixes

• Add rspamadm mime tool to do various email operations:
  • extract text/HTML content
  • extract statistical tokens
  • exctact URLs
• Fixed encryption mode in Rspamd proxy
• Fixed various crashes in maps during reload
• Preload maps data before starting of the worker processes when possible
• Better HTML styles processing: add ZeroFont exploit filtering rules
• Fix ED25519 DKIM signatures as described by the latest RFC draft
• Added crash reporting system via libunwind

Full list of the meaningful changes

• [CritFix] Check NM part of pubkey to match it with rotating keypairs
• [CritFix] Do not overwrite PID of the main process
• [CritFix] Fix maps after reload
• [CritFix] Fix maps race conditions on reload
• [CritFix] Fix shmem leak in encrypting proxy mode
• [Feature] Add a concept of ignored symbols to avoid race conditions
• [Feature] Add ability to print bayes tokens in rspamadm mime
• [Feature] Add method to get statistical tokens in Lua API
• [Feature] Add preliminary mime stat command
• [Feature] Add rspamadm mime tool
• [Feature] Add urls extraction tool
• [Feature] Address ZeroFont exploit
• [Feature] Allow rspamadm mime to process multiple files
• [Feature] Allow to extract words in rspamadm mime
• [Feature] Allow to print mime part data
• [Feature] Allow to show HTML structure on extraction
• [Feature] Distinguish IP failures from connection failures
• [Feature] Improve output for mime command
• [Feature] Improve styles propagation
• [Feature] Main process crash will now cleanup all children
• [Feature] Preload file and static maps in main process
• [Feature] Print stack trace on crash
• [Feature] Process font size in HTML parser
• [Feature] Propagate content length of invisible tags
• [Feature] Read ordinary file maps in chunks to be more safe on rewrites
• [Feature] Support base tag in HTML
• [Feature] Support more size suffixes when parsing HTML styles
• [Feature] Support opacity style
• [Fix] Another fix for nested composites
• [Fix] Fill nm id in keypairs cache code
• [Fix] Fix colors alpha channel handling
• [Fix] Fix destruction logic
• [Fix] Fix double free
• [Fix] Fix maps preload logic
• [Fix] Fix nested composites process
• [Fix] Fix proxying of Exim connections
• [Fix] Fix reload crash
• [Fix] Fix rspamadm -l command
• [Fix] Update ed25519 signing schema
• [WebUI] Stop using “const” declaration
• [WebUI] Update RequireJS to 2.3.5