pmacct Changelog

What's new in pmacct 1.5.2

Sep 8, 2015
  • Introduced support for a RabbitMQ broker to be used for queueing and data exchange between Core Process and plugins. This is in alternative to the home-grown circular queue implementation. plugin_pipe_amqp directive, along with all other plugin_pipe_amqp_* directives, can be set globally or apply on a per plugin basis (ie. it is a valid scenario, if multiple plugins are instantiated, that some make use of home-grown queueing, while others use RabbitMQ based queueing).
  • Introducting support for Maximind GeoIP v2 (libmaxminddb) library: if pmacct is compiled with --enable-geoipv2, this defines full pathname to a Maxmind GeoIP database v2 (libmaxminddb) Only the binary database format is supported (ie. it is not possible to load distinct CSVs for IPv4 and IPv6 addresses).
  • Introduced infrastructure for sFlow counters and support specifically for generic, ethernet and vlan counters. Counters are exported in JSON format to files, specified via sfacctd_counter_file. The supplied filename can contain as variable the sFlow agent IP address.
  • Introduced a new thread_stack config directive to allow to modify the thread stack size. Natanael Copa reported that some libc implementations, ie. musl libc, may set a stack size that is too small by default.
  • Introduced networks_file_no_lpm feature: it applies when aggregation method includes src_net and/or dst_net and nfacctd_net (or equivalents) and/or nfacctd_as_new (or equivalents) are set to longest (or fallback): an IP prefix defined as part of the supplied networks_file wins always, even if it's not longest.
  • tee plugin: added support for (non-)transparent IPv6 replication [further QA required]
  • plugin_common.c, sql_common.c: added log message to estimate base cache memory usage.
  • print, AMQP, MongoDB plugins; sfacctd, BGP, BMP daemons: introducing timestamps_since_epoch to write timestamps in 'since Epoch' format.
  • nfacctd: flow bytes counter can now be sourced via element ID #352 (layer2OctetDeltaCount) in addition to element ID's already supported. Thanks to Jonathan Thorpe for his support.
  • Introducing proc_priority: redefines the process scheduling priority, equivalent to using the 'nice' tool. Each daemon process, ie. core, plugins, etc., can define a different priority.
  • fix, BMP daemon: improved preliminar checks in bmp_log_msg() and added missing SIGHUP signal handling to reload bmp_daemon_msglog_file files.
  • fix, bgp_logdump.c: under certain configuration conditions call to both write_and_free_json() and write_and_free_json_amqp() was leading to SEGV. Thanks to Yuriy Lachin for reporting the issue.
  • fix, BGP daemon: improved BGP dump output: more accurate timestamping of dump_init, dump_close events. dump_close now mentions amount of entries and tables dumped. Thanks to Yuriy Lachin for brainstorming around this.
  • fix, cfg.c: raised amount of allowed config lines from 256 to 8K.
  • fix, print/AMQP/MongoDB plugins: SEGV observed when IPFIX vlen variables were stored in the pending_queries_queue structure (ie. as a result of a time mismatch among the IPFIX exporter and the collector box).
  • fix, vlen primitives: when 'raw' semantics was selected, print_hex() was returning wrong hex string length (one char short). As a consequence occasionally some extra dirty chars were seen at the end of the converted string.
  • fix, vlen primitives: memory leak verified in print/AMQP/MongoDB plugins.
  • fix, print, MongoDB & AMQP plugins: dirty values printed as part of the 'proto' under certain conditions. Thanks to Rene Stoutjesdijk for his support resolving the issue.
  • fix, amqp_common.c: amqp_exchange_declare() call changed so to address the change of rabbitmq-c API for support of auto_delete & internal for exchange.declare. Backward compatibility with rabbitmq-c

New in pmacct 1.5.1 (Feb 23, 2015)

  • BMP daemon: BMP, BGP Monitoring Protocol, can be used to monitor BGP sessions. The current implementation is base on the draft-ietf-grow-bmp-07 IETF draft. The daemon currently supports BMP events and stats only, ie. initiation, termination, peer up, peer down and stats reports messages. Route Monitoring is future (upcoming) work but routes can be currently sourced via the BGP daemon thread (best path only or ADD-PATH), making the two daemons complementary. The daemon enables to write BMP messages to files or AMQP queues, real-time (msglog) or at regular time intervals (dump) and is a separate thread in the NetFlow (nfacctd) or sFlow (sfacctd) collectors.
  • tmp_net_own_field directive is introduced to record both individual source and destination IP addresses and their IP prefix (nets) as part of the same aggregation method. While this should become default behaviour, a knob for backward-compatibility is made available for all 1.5 until the next major release.
  • Introduced nfacctd_stitching and equivalents (ie. sfacctd_stitching): when set to true, given an aggregation method, two new non-key fields are added to the aggregate upon purging data to the backend: timestamp_min is the timestamp of the first element contributing to a certain aggregate and timestamp_max is the timestamp of the last element. In case the export protocol provides time references, ie. NetFlow/IPFIX, these are used; if not the current time (hence time of arrival to the collector) is used instead.
  • Introduced amqp_routing_key_rr feature to perform round-robin load- balancing over a set of routing keys. This is in addition to existing, and more involved, functionality of tag-based load-balancing.
  • Introduced amqp_multi_values feature: this is same feature in concept as sql_multi_values (see docs). The value is the amount of elements to pack in each JSON array.
  • Introduced amqp_vhost and companion (ie. bgp_daemon_msglog_amqp_vhost) configuration directives to define the AMQP/RabbitMQ server virtual host.
  • BGP daemon: bgp_daemon_id now allows to define the BGP Router-ID disjoint from the bgp_daemon_ip definition. Thanks to Bela Toros for his patch.
  • tee plugin: introduced tee_ipprec feature to color replicated packets, both in transparent and non-transparent modes. Useful, especially when in transparent mode and replicating to hosts in different subnets, to verify which packets are coming from the replicator.
  • tee plugin: plugin-kernel send buffer size is now configurable via a new config directive tee_pipe_size. Improved logging of send() failures.
  • nfacctd: introduced support for IPFIX sampling/renormalization using element IDs: #302 (selectorId), #305 (samplingPacketInterval) and #306 (samplingPacketSpace). Many thanks to Rene Stoutjesdijk for his support.
  • nfacctd: added also support for VLAN ID for NetFlow v9/IPFIX via element type #243 (it was already supported via elements #58 and #59). Support was also added for 802.1p/CoS via element #244.
  • nfacctd: added native support for NetFlow v9/IPFIX IE #252 and #253 as part of existing primitives in_iface and out_iface (additional check).
  • pre_tag_map: introduced 'cvlan primitive. In NetFlow v9 and IPFIX this is compared against IE #245. The primitive also supports map indexing.
  • Introduced pre_tag_label_filter to filter on the 'label' primitive in a similar way how the existing pre_tag_filter feature works against the 'tag' primitive. Null label values (ie. unlabelled data) can be matched using the 'null' keyword. Negations are allowed by pre-pending a minus sign to the label value.
  • IMT plugin: introduced '-i' command-line option to pmacct client tool: it shows last time (in seconds) statistis were cleared via 'pmacct -e'.
  • print, MongoDB & AMQP plugins: sql_startup_delay feature ported to these plugins.
  • sql_num_hosts: the feature has been improved to support IPv6 addresses. Pre-requisite is definition of INET6_ATON() function in the RDBMS, which is the case for MySQL >= 5.6.3. In SQLite such function has to be defined manually.
  • nfacctd: improved NF_evaluate_flow_type() euristics to reckon NetFlow/ IPFIX event (NAT, Firewall, etc.) vs traffic (flows) records.
  • fix, GeoIP: spit log notification (warning) in case GeoIP_open() returns null pointer.
  • fix, IMT plugin: pmacct client -M and -N queries were failing to report results on exact matches. Affected: 1.5.0. Thanks to Xavier Vitard for reporting the issue.
  • fix, pkt_handlers.c: missing else in NF_src_host_handler() was causing IPv6 prefix being copied instead of IPv6 address against NetFlow v9 recs containing both info.
  • fix, uacctd: informational log message now shows the correct group the daemon is bound to. Thanks to Marco Marzetti for reporting the issue.
  • fix, nfv9_template.c: missing byte conversion while decoding templates was causing SEGV under certain conditions. Thanks to Sergio Bellini for reporting the issue.

New in pmacct 1.5.0 (Feb 23, 2015)

  • Introduced bgp_daemon_msglog_file config directive to enable streamed logging of BGP messages/events. Each log entry features a time reference, BGP peer IP address, event type and a sequence number (to order events when time reference is not granular enough). BGP UPDATE messages also contain full prefix and BGP attributes information. Example given in QUICKSTART file, chapter XIIf.
  • Introduced dump of BGP tables at regular time intervals. The filename, which can include variables, is set by bgp_table_dump_file directive. The output format, currently only JSON, can be set in future via the bgp_table_dump_output directive. The time interval between dumps can be set via the bgp_table_dump_refresh_time directive. Example given in QUICKSTART file, chapter XIIf.
  • Introduced support for internally variable-length primitives (likely candidates are strings). Introduced also the 'label' primitive which is a variable-length string equivalent of tag and tag2 primitives. Its value are set via a 'set_label' statement in a pre_tag_map (see examples/ pretag.map.example). If, ie. as a result of JEQ's in a pre_tag_map, multiple 'set_label' are applied, then default operation is append labels and separate by a comma.
  • pmacct project has been assigned PEN #43874. nfprobe plugin: tag, tag2, label primitives are now encoded in IPFIX making use of the pmacct PEN.
  • Ported preprocess feature to print, MongoDB and AMQP plugins. Preprocess allows to process aggregates (via a comma-separated list of conditionals and checks) while purging data to the backend thus resulting in a powerful selection tier. minp, minb, minf, minbpp, minppf checks have been currently ported. As a result of the porting a new set of config directives are added, ie. print_preprocess and print_preprocess_type.
  • print, MongoDB & AMQP plugins: if data (start/base) time is greater than commit time then place in pending queue and after purging event re-insert in cache. Concept ported from SQL plugins.
  • MySQL, PostgreSQL plugins: sql_locking_style now supports keyword "none" to disable locking. This method can help in certain cases, for example when grants over the whole database (requirement for "table" locking in MySQL) is not available.
  • util.c: open_logfile() now calls mkdir_multilevel() to allow building intermediate directory levels, if not existing. This brings all log files in line with capabilities of print_output_file directive.
  • Introduced [u|pm]acctd_flow_tcp_lifetime to defines how long a TCP flow could remain inactive. This is in addition to [u|pm]acctd_flow_lifetime that allows to define the same for generic, ie. non-TCP, flows. Thanks to Stathis Gkotsis for his support.
  • Introducing nfacctd_account_options: if set to true account for NetFlow/ IPFIX option records as well as flow ones. pre_tag_map offers sample_type value of 'option' now to split option data records from flow ones.
  • nfprobe plugin: support for custom-defined primitives has been introduced in line with other plugins. With such feature it will be possible to augment NetFlow v9/IPFIX records with custom fields (in IPFIX also PENs are supported).
  • Built a minimal API, for internal use only, around AMQP. Goal is to make re-use of the same AMQP structures for different purposes (logging, BGP daemon dumps, AMQP plugin, etc.).
  • fix, BGP daemon: introduced bgp_peer_info_delete() to delete/free BGP info after a BGP peer disconnects.
  • fix, print, AMQP, memory plguins: when selecting JSON output, jansson library json_decref() is used in place of free() to free up memory allocated by JSON objects. Using free() was originating memory leaks.
  • fix, AMQP plugin: in line with other plugins QN (query number or in case of AMQP messagess number) in log messages now reflects the real number of messages sent to the RabbitMQ message exchange and not just all messages in the queue. Thanks to Gabriel Snook for reporting the issue.
  • fix, IMT plugin: memory leak due to missed calls to free_extra_allocs() in case all extras.off_* were null. Thanks to Tim Jackson for his support resolving the issue.
  • fix, pmacctd: if reading from a pcap_savefile, introduce a short usleep() after each buffer worth of data so to give time plugins to process/cache it.
  • fix, SQL plugins: SQL handler types now include primitives registry index
  • fix, print, AMQP & MongoDB plugins: added free() for empty_pcust allocs
  • fix, plugin hooks: improved checks to prevent the last buffer on a pipe to plugins (plugin_pipe_size) could go partly out of bounds.
  • fix, nfacctd: improved handling of IPFIX vlen records.
  • fix, nfprobe: SEGV if custom primitives are defined but array structure is not allocated.
  • fix, nfprobe: wrong length was calculated in IPv6 templates for fields with PEN != 0.
  • fix, plugin_common.c: declared struct pkt_data in P_cache_insert_pending to be pointed by prim_ptrs. primptrs_set_all_from_chained_cache() is now safe if prim_ptrs is null.
  • fix, nfprobe: tackled the case of coexisting 1) PEN and non-PEN custom primitives and 2) variable and fixed custom primitives.
  • fix, plugin_common.c: declared struct pkt_data in P_cache_insert_pending to be pointed by prim_ptrs. primptrs_set_all_from_chained_cache() is now safe if prim_ptrs is null.
  • fix, lofging: selected configuration file is now logged. cfg_file is passed through realpath() in order to always log the absolute path.
  • fix, print, MongoDB & AMQP plugins: pm_setproctitle() invoked upon forking writer processes in alignment with SQL plugins.
  • fix, pmacct client: it's now possible to query and wildcard on primitives internally allocated over what_to_count_2 registry.

New in pmacct 1.5.0 RC 3 (Jun 27, 2014)

  • BGP daemon: support for BGP ADD-PATH capability draft-ietf-idr-add-paths has been introduced, useful to advertise known paths when BGP multi-path is enabled in a network. The correct BGP info is linked to traffic data using BGP next-hop (or IP next-hop if use_ip_next_hop is set to true) as selector among the paths available.
  • pre_tag_map: de-globalized the feature so that, while Pre-Tagging is evaluated in the Core Process, each plugin can be defined a own/local pre_tag_map.
  • maps_row_len: directive introduced to define the maximum length of map (ie. pre_tag_map) rows. The default value is suitable for most scenarios, though tuning it could be required either to save on memory or to allow for longer entries (ie. filters).
  • Introduced use_ip_next_hop config directive: when IP prefix aggregation (ie. nfacctd_net) is set to 'netflow', 'sflow' or 'fallback' populate 'peer_dst_ip' field from NetFlow/sFlow IP next hop field if BGP next-hop is not available.
  • AMQP plugin: implemented persistent messaging via amqp_persistent_msg configuration directive so to protect against RabbitMQ restarts. Feature is courtesy by Nick Douma.
  • pmacct in-memory plugin client: -T option now supports how many entries to show via ',[]' argument syntax.
  • nfprobe plugin: take BGP next-hop from a defined networks_file. This is in addition to existing feature to take BGP next-hop from a BGP feed.
  • Set of *_proc_name configuration directives renamed to core_proc_name. Value of core_proc_name is now applied to logging functions and process title.
  • Re-implemented reverse BGP lookup based primitives, src_as_path src_med src_std_comm src_ext_comm and src_local_pref, in print, MongoDB and AMQP plugins. Primitives have also been re-documented.
  • pre_tag_map: set_tag and set_tag2 can now be auto-increasing values, ie. "set_tag=1++": "1" being the selected floor value at startup and "++" instructs to increase the tag value at every pre_tag_map iteration. Many thanks to Brent Van Dussen and Gabriel Snook for their support.
  • Added support for NetFlow v9/IPFIX source/destination IPv4/IPv6 prefixes encoded as flow types: #44, #45, #169 and #170.
  • [sql|print|mongo|amqp]_history and sql_trigger_time can now be specified also in seconds, ie. as '300' or '300s' alternatively to '5m'. This is to ease syncronization of these values against refresh time to the backend, ie. sql_refresh_time.
  • Added post_tag2 configuration directive to set tag2 similarly to what post_tag does.
  • SQL plugins: agent_id, agent_id2 fields renamed to tag, tag2. Issued SQL table schema #9 for agent_id backward compatibility. Renaming agent_id2 to tag2 is going to be disruptive to existing deployments instead. UPGRADE doc updated.
  • print, MongoDB, AMQP plugins: added [print|mongo|amqp]_max_writers set of configuration directives to port from SQL plugins the idea of max number of concurrent writer processes the plugin is allowed to start.
  • util.c: comments can now start with a '#' symbol in addition to existing '!'.
  • fix, BGP daemon: removed a non-contextual BGP message length check. Same check is already done in the part handling payload reassembly.
  • fix, BGP daemon: MP_REACH_NLRI not assumed to be anymore at the end of a route announcement.
  • fix, MySQL plugin: added linking of pmacct code against -lstdc++ and -lrt if MySQL plugin is enabled, pre-requisite for MySQL 5.6. Many thanks to Stefano Birmani for reporting the issue.
  • fix, sql_common.c: memory leak affecting AS-PATH and BGP communities. Version 1.5.0rc2 affected. Thanks to Brent Van Dussen for his support solving the issue.
  • fix, MongoDB plugin: timestamp_start, timestamp_end moved from timestamp type, reserved for internal use, to date.
  • fix, print, memory, MongoDB, AMQP plugins: if no AS_PATH information is available an empty string, ie. "", is placed as value (instead of former "^$"). Similar stream-lining was done for communities. Many thanks to Brent Van Dussen and Elisa Jasinska for reporting the issue.
  • fix, AMQP, MongoDB plugins: increased default refresh time to 60 secs, up from 10 and in line with SQL plugins value.
  • fix, nfprobe plugin: IPv6 source/destination masks passed as IE #29 and #30 and not anymore as their IPv4 counterparts.
  • fix, pmacct.c: clibuf variable now malloc'd at runtime so to not impact the data segment.
  • fix, log.c: removed sbrk() calls when logging to Syslog.
  • fix, pmacctd: If compiling against PF_RING, check and compile against libnuma and librt which are new requirement since version 5.6.2. Thanks to Joan Juvanteny for reporting the issue.
  • fix, net_aggr.c: 'prev' array to keep track of hierarchies of networks was being re-initialized by some compilers. Thanks to Joan Juvanteny for reporting the issue.
  • fix, MongoDB, JSON outputs: dst_host_country primitive was not properly shown. Patch is courtesy by Stig Thormodsrud.
  • fix, pre_tag_map: a memory leak was found when reloading rules containing 'filter' keywords. Thanks to Matt Jenkins for his support resolving the issue.
  • fix, server.c: countered a timing issue to ensure EOF is sent after data. Issue was originated by conjunction of non-blocking socket and multiple CPU cores. Thanks to Juan Camilo Cardona and Joel Ouellette Jr for their support.
  • fix, acct.c: added length check to hash_crc32() of custom primitives as selective pmacct IMT client queries, ie. -M and -N, were failing to match entries. Thanks to Joel Ouellette Jr for his support.
  • fix, nfacctd: NetFlow v9/IPFIX sampling correlation has been improved by placing system scoped sampling options in a separate table. Such table is queried if no matching sampler ID is found for a given . Sampling-related fields (ie. sampler ID, interval, etc.) are now all supported if 1, 2 or 4 bytes long.
  • fix, nfacctd: improved handling of the NAT64 case for NSEL. Thanks to Gregoire Leroy for his support.
  • fix, nfacctd, sfacctd and BGP daemon: if IPv6 is enabled, IPv4 mapped is supported and can't obtain an IPv6 socket to listen to, retry with a IPv4 one.

New in pmacct 0.11.6 (Apr 8, 2009)

  • Introduced support for tag ranges into the 'pre_tag_filter' configuration directive (ie. '10-20' matches traffic tagged in the range 10..20). This works both in addition to and in combination with negations.
  • Tcpdump-style filters, ie. 'aggregate_filter', now support indexing within a packet, ie. 'ether[12:2]', to allow a more flexible separation of the traffic.
  • Introduced support for descriptions in networks definition files pointed by the 'networks_file' configuration directive. Thanks to Karl O. Pinc for contributing the patch.
  • fix, pmacctd: libpcap DLT_LINUX_SLL type is not defined in older versions of the library. It was preventing successful compilation of pmacct on OpenBSD. This has been fixed by defining internally to pmacct all DLT types in use. Thanks to Karl O. Pinc for his support.
  • fix, IPv6 networks_file, load_networks6(): wrong masks were applied to IPv6 networks due to dirty temporary buffers for storing IPv6 addresses and masks. Short '::' IPv6 format is currently not supported. Thanks to Robert Blechinger for flagging the issue.
  • fix, pretag.c: Pre-Tagging infrastructure was SEGV'ing after having been instructed to reload via a SIGHUP signal. Patch is courtesy of Denis Cavrois and the Acipia development team.
  • fix, sfacctd, nfacctd: Assign16() was not handling correctly 2-bytes EtherType values (ie. 0x86dd, 0x8847) in 802.1Q tags. As a result 'aggregate_filter' was not able to correctly match IPv6-related filters. Thanks to Axel Apitz for reporting the issue.
  • fix, xflow_status.c: a cosmetic bug was displaying sequence numbers without applying previous increment. This definitely will help troubleshooting and debugging.
  • fix, sfacctd, sfv245_check_status(): AF of the sFlow agent is now explicitely defined: when IPv6 is enabled the remote peer address can be reported as IPv4-mapped IPv6 address. This was causing warning messages to report the wrong sFlow agent IP address. Thanks to Axel Apitz for reporting the issue.
  • fix, IMT plugin was crashing upon receipt of a classification table request (WANT_CLASS_TABLE) when stream classification was actually disabled.
  • fix, pmacct.c: classifier index was not brought back to zero by the pmacct client. This was preventing the client to show correct stream classification when it was feeded with multiple queries. The fix is courtesy of Fabio Cairo.
  • fix, MySQL plugin: upon enabling of the 'nfacctd_sql_log' directive, 'stamp_updated' field was incorrectly reported as '0000-00-00 00:00:00' due to wrong field formatting. Thanks to Brett D'Arcy for reporting and patching the issue.
  • Initial effort to clean the code up by strcpy() calls. Thanks to Karl O. Pinc for taking such initiative.