Rspamd 2.3 has been released

04 Feb 2020

We have released Rspamd 2.3 today.

This release has various improvements, pdf parsing support and new SPF plugin. Numerous of the bug fixes, including some critical ones have also been applied during this release cycle.

Here is the list of the most important changes.

Lua content library

This library is a part of content scanning project that enables Rspamd to process content formats that are commonly met in email. In this release, we have added some preliminary PDF files support.

Rspamd supports the following features so far:

  • URLs extraction from PDF files
  • JavaScript extraction from actions
  • Using of fuzzy hashes storage to store suspicious js extracted from PDF files
  • Couple of rules related to bad PDF files (e.g. PDF_ENCRYPTED)

This library is written in pure Lua, using LPEG and Hyperscan under the hood to provide memory safety and high processing speed.

SPF plugin revamped

SPF plugin has been one of the oldest plugins in Rspamd. Unfortunately due to the fact that it has been written in C, it was very hard to add new functions to this plugin nor maintain it in the consistent state. In this release, SPF plugin has been rewritten in Lua providing configuration compatible mode with some of new features, such as support of the external relay handling

Slow rules protection

Rspamd will now try to check async timers and events when it notices that some rule takes too much time. This will allow to break dead rules to hold processing for more than task timeout time. In future, we plan to extend task timeout relation for all internal rule timers.

Other features

Here is a list of other important but not categorised features:

  • Allow milter code to deal with multiple headers
  • Antivirus : Add Avast support
  • Dkim_signing : Allow to sign via milter_headers
  • Send quit command to Redis
  • Speed up is_ascii function
  • Improve memory pool allocation routines and add memory debugging

Important bug fixes

  • Critical fix: fix html entities decoding. This bug could cause incorrect URLs being extracted from messages.
  • Critical fix: fix re cache when mix of pcre and hyperscan is used. This bug could cause random failures when scanning regular expressions that could not be evaluated nor approximated to hyperscan.
  • Fix arc seal validation
  • Fix base tag processing according to stupid HTML renderer behaviour
  • Fix dealing with \0 in ucl strings and JSON
  • Fix gpg parts misdetection
  • Fix ignored symbols exporting (e.g. to ClickHouse)
  • Fix processing of numeric url’s
  • Fix processing of the closed tcp connections
  • Fix mixed charset rule for some languages (e.g. Czech)
  • Fix mixing of the IPv4 and IPv6 addresses in Radix maps
  • Fix soft hypen processing
  • Fix O(N^2) algorithm when comparing recipients
  • Various stability improvements when dealing with large or specially crafted messages

Full list of the meaningful changes

  • [Conf] SPF is no longer a C module
  • [Conf] Update spamtrap map path example
  • [CritFix] Fix html entities decoding
  • [CritFix] Fix re cache when mix of pcre and hyperscan is used
  • [Feature] Allow milter code to deal with multiple headers
  • [Feature] Antivirus: Add avast support
  • [Feature] Dkim_signing: Allow to sign via milter_headers
  • [Feature] Implement content hashes
  • [Feature] Lua_text: Add regexp split iterator method
  • [Feature] Lua_text: Implement flattening of the input tables
  • [Feature] Send quit command to Redis
  • [Feature] Speed up is_ascii function
  • [Feature] Spf: Add external_relay option
  • [Fix] Avoid double escaping
  • [Fix] Fix O(N^2) algorithm
  • [Fix] Fix arc seal validation
  • [Fix] Fix base tag processing according to stupid HTML renderer behaviour
  • [Fix] Fix dealing with \0 in ucl strings and JSON
  • [Fix] Fix gpg parts misdetection
  • [Fix] Fix ignored symbols exporting
  • [Fix] Fix processing of numeric url’s
  • [Fix] Fix processing of the closed tcp connections
  • [Fix] Fix regexp type check for pcre2
  • [Fix] Fix urls encode function
  • [Fix] Fix urls shifting when doing decode to include separators
  • [Fix] Fix white on white rule and add is_leaf flag
  • [Fix] Further fixes in charset detection
  • [Fix] Ignore diacritics in chartable module for specific languages
  • [Fix] Limit size of symbols options by max_opts_len option
  • [Fix] More fixes in html tag content calculations
  • [Fix] Plug memory leak in fuzzy storage
  • [Fix] Process high priority settings even if settings/id has been specified
  • [Fix] Select a different upstream on last retransmit
  • [Fix] Treat soft hyphen as zero width space
  • [Fix] Try harder to watch the lifetime of the key_stat
  • [Fix] Use ipv6-mapped-ipv4 addresses in radix trie
  • [Project] Add logic to break execution when processing symbols*
  • [Project] Add methods to set specific content for mime parts from Lua
  • [Project] Lua_content: support PDF files
  • [Project] Move dns_tool to using of the rspamd_spf from FFI module
  • [Project] Preliminary SPF plugin in Lua
  • [Project] Show debug stat for memory pool
  • [Project] Some rework about specific data that is now tagged
  • [Project] Start reworking of the mempool structure
  • [Rework] Allow to add userdata as symbols options
  • [Rework] Change mime part specifics handling
  • [Rework] Move LRU SPF cache from spf plugin
  • [Rework] Rework HTML tags content attachment
  • [Rework] Rework options hash structure
  • [Rework] Start lua_content library
  • [Rework] Stop using of uthash for http headers
  • [Rework] Use faster hashing approach for memory pools variables
  • [Rules] Add PDF related rules