pgBadger Changelog

What's new in pgBadger 9.2

Jul 28, 2017
  • New features:
  • Add report of checkpoint distance and estimate.
  • Add support of AWS Redshift keywords to SQL code beautifier.
  • Add autodetection of log format in remote mode to allow remote parsing of pgbouncer log file together with PostgreSQL log file.
  • Bug fixes and features enhancements:
  • Fix reports with histogram that was not showing data upper than the last range.
  • Fix parsing of journalctl without the the log line number pattern ([%l-n]). Thanks to Christian Schmitt for the report.
  • Add report of checkpoint distance and estimate. Thanks to jjsantam for the feature request.
  • Append more information on what is done by script to update CSS and javascript files, tools/updt_embedded_rsc.pl.
  • Do not warn when all log files are empty and exit with code 0.
  • Fix build_log_line_prefix_regex() that does not include %n as a lookup in %regex_map. Thanks to ghosthound for the patch.
  • Change error level of "FATAL: cannot use CSV" to WARNING. Thanks to kong1man for the report.
  • Fix use of uninitialized value warning. Thanks to Payal for the report.
  • Add permission denied to error normalization
  • Update pgbadger to latest commit 5bdc018 of pgFormatter.
  • Add support for AWS Redshift keywords. Thanks to cavanaug for the feature request.
  • Fix missing query in temporary file report when the query was canceled. Thanks to Fabrizio de Royes Mello for the report.
  • Normalize query with binded parameters, replaced with a ?.
  • Sanity check to avoid end time before start time. Thanks to Christophe Courtois for the patch.
  • Fix a lot of mystyped words and do some grammatical fixes. Use 'pgBadger' where it refers to the program and not the binary file. Also, use "official" expressions such as PgBouncer, GitHub, and CSS. POD file was synced with README. Thanks to Euler Taveira for the patch.
  • Menu is broken when --disable-type top_cancelled_info test and closing list must be inside disable_type test. While in it, ident disable_lock test. Thanks to Euler Taveira for the patch.
  • Fix use of uninitialized value. Thanks to johnkrugger for the report.
  • Remove test to read log file during log format auto-detection when the file is hosted remotly. Thanks to clomdd for the report.
  • Add autodetection of log format in remote mode to allow remote parsing of pgbouncer log file together with PostgreSQL log file.
  • Fix number of sessions wrongly increased after log line validation Thanks to Achilleas Mantzios for the report.
  • Minor reformatting of the pgBadger Description.
  • Fix repeated info in documentation. Thanks to cscatolini for the patch.

New in pgBadger 9.1 (Jan 25, 2017)

  • New features:
  • Add report of error class distribution when SQLState is available in the log_line_prefix (see %e placeholder).
  • Update SQL Beautifier to pgFormatter v1.6 code.
  • Improve error message normalization.
  • Add --normalized-only option to generate a text file containing all normalized queries found in a log with count.
  • Allow %c (session id) to replace %p (pid) as unique session id.
  • Add waiting for lock messages to event reports.
  • Add --start-monday option to start calendar weeks in Monday instead of default to Sunday.
  • Bug fixes and features enhancements:
  • Add report of error class distribution when SQLState is available in the log line prefix. Thanks to jacks33 for the feature request.
  • Fix incremental global index on resize. Thanks to clomdd for the report.
  • Fix command tag log_line_prefix placeholder %i to allow space character.
  • Fix --exclude-line options and removing of obsolete directory when retention is enabled and --noreport is used.
  • Fix typo in "vacuum activity table". Thanks to Nicolas Gollet for the patch.
  • Fix autovacuum report. Thanks to Nicolas Gollet for the patch.
  • Fix author of pgbadger's logo - Damien Cazeils and English in comments. Thanks to Thibaut Madelaine for the patch.
  • Add information about pgbouncer log format in the -f option. Thanks to clomdd for the report.
  • Add --normalized-only information in documentation.
  • Fix broken report of date-time introduced in previous patch.
  • Fix duration/query association when log_duration=on and log_statement=all. Thanks to Eric Jensen for the report.
  • Fix normalization of messages about advisory lock. Thanks to Thibaut Madelaine for the report.
  • Fix report of auto_explain output. Thanks to fch77700 for the report.
  • Fix unwanted log format auto detection with log entry from stdin. Thanks to Jesus Adolfo Parra for the report.
  • Add left open parentheses to the "stop" chars of regex to look for db client in the prefix to handle the PostgreSQL client string format that includes source port. Thanks to Jon Nelson for the patch.
  • Fix some spelling errors. Thanks to Jon Nelson for the patch.
  • Allow %c (session id) to replace %p (pid) as unique session id. Thanks to Jerryliuk for the report.
  • Allow pgbadger to parse default log_line_prefix that will be probably used in 10.0: '%m [%p] '
  • Fix missing first line with interpreter call.
  • Fix missing Avg values in CSV report. Thanks to Yosuke Tomita for the report.
  • Fix error message in autodetect_format() method.
  • Add --start-monday option to start calendar weeks in Monday instead of default to Sunday. Thanks to Joosep Mae for the feature request.
  • Fix --histo-average option. Thanks to Yves Martin for the report.
  • Remove plural form of --ssh-option in documentation. Thanks to mark-a-s for the report.
  • Fix --exclude-time filter and rewrite code to skip unwanted line as well code to update the progress bar. Thanks to Michael Chesterton for the report.
  • Fix support to %r placeholder in prefix instead of %h.

New in pgBadger 9.0 (Sep 2, 2016)

  • This major release of pgBadger is a port to bootstrap 3 and a version upgrade of all resources files (CSS and Javascript). There's also some bugs fixes and features enhancements.

New in pgBadger 8.2 (Aug 12, 2016)

  • This is a maintenance release that fix some minor bug. There is also some performances improvement up to 20% on huge files and some new interesting features:
  • Multiprocessing can be used with pgbouncer log files.
  • pgBouncer and PostgreSQL log files can be used together in incremental mode.
  • With default or same prefix, stderr and syslog file can be parsed together, csvlog format can always be used.
  • Use a modal dialog window to download graphs as png images.
  • Add pl/pgSQL function information to queries when available.
  • Here are the complete list of changes:
  • Fix report of database system messages.
  • Fix multi line statement concatenation after an error.
  • Fix box size for report of queries generating the most temporary files and the most waiting queries.
  • Rewrite code to better handle multi-line queries.
  • Fix garbage in examples of event queries with error only mode (option -w). Thanks to Thomas Reiss for the report.
  • Fix getting dataset related to query duration with the use of auto_explain. Thanks to tom__b for the patch.
  • Use a modal dialog window to download graphs as png images.
  • Huge rewrite of the incremental mechanism applied to log files to handle PostgreSQL and pgbouncer logs at the same time.
  • Multiprocess can be used with pgbouncer log.
  • Add code to remove remaining keyword placeholders tags.
  • Fix an other possible case of truncated date in LAST_PARSED file Thanks to brafaeloliveira for the report.
  • Set default scale 1 in pretty_print_number() js function.
  • Fix auto-detection of pgbouncer files that contain only stats lines. Thanks to Glyn Astill for the patch.
  • Add date to samples of queries generating most temporary files.
  • Do not display warning message of empty log when quiet mode is enable.
  • Fix reading from stdin by disabling pgbouncer format detection. Thanks to Robert Vargason for the patch.
  • Fix case of duplicate normalized error message with "nonstandard use of ...".
  • Fix storage of current temporary file related request.
  • Use the mnemonic rather than a signal number in kill calls. Thanks to Komeda Shinji for the patch.

New in pgBadger 7.3 (Jan 18, 2016)

  • This is a maintenance release to fix a major bug that was breaking the incremental mode in pgBadger. It also adds some more reports and features:
  • Add --timezone=+/-HH to control the timezone used in charts. The javascript library runs at client side so the timezone used is the browser timezone so the displayed time in the charts can be different from the time in the log file.
  • Add /tmp/pgbadger.pid file to prevent cron jobs overlaping on same log files.
  • Add command line option --pid-dir to be able to run two pgbadger at the same time by setting an alternate path to the pid file.
  • Report information about "LOG: skipping analyze of ..." into events reports.
  • Report message "LOG: sending cancel to blocking autovacuum" into events reports. Useful to look for queries generating autovacuum kill on account of a lock issue.

New in pgBadger 7.2 (Jan 5, 2016)

  • This new release fixes some issues especially in temporary files reports and adds some features:
  • Allow pgBadger to parse natively the journalctl command output
  • Add new keywords from PG 9.5 for code formating
  • Add support to %n log_line_prefix option for Unix epoch (PG 9.6)
  • There's also some new command line option:
  • Adds --journalctl_cmd option to enable this functionality and set the command.
  • Here is the full list of changes/fixes:
  • Fix missing detailed information (date, db, etc.) in Queries generating the largest temporary files report.
  • Fix label of sessions histogram. Thanks to Guillaume Lelarge for the patch.
  • Fix to handle cancelled query that generate more than one temporary file and more generally aggregate size on queries with multiple (> 1GB) temporary files.
  • Add "Total size" column in Temporary Files Activity table and fix size increment when a query have multiple 1GB temporary file.
  • Fix temporary file query normalization and examples.
  • Fix incomplete and wrong queries associated to temporary files when STATEMENT level log line was missing. Thanks to Mael Rimbault for the report.
  • When -w or --watch-mode is used, message "canceling statement due to statement timeout" s now reported with other errors.
  • Allow dot in dbname and user name. Thanks to David Turvey for the patch.
  • Remove use of unmaintained flotr2 javascript chart library and use of jqflot instead.
  • Fix bad formatting with anonymized values in queries.
  • Display 0ms instead of 0s when qery time is under the millisecond. Thanks to venkatabn for the report.
  • Normalize cursor names. Patch from Julien Rouhaud
  • Fix unregistered client host name with default pattern. Thanks to Eric S. Lucinger Ruiz for the report.
  • Remove redundant regular expressions.
  • Tweaking awkward phrasing, correcting subject-verb agreements, typos, and misspellings. Patch from Josh Kupershmid.
  • Fix potential incorrect creation of subdirectory in incremental mode.
  • Allow single white space after duration even if this should not appear.
  • Update copyright.

New in pgBadger 7.1 (Jul 14, 2015)

  • This new release fixes some issues and adds a new report:
  • Distribution of sessions per application.
  • Adds Json operators to SQL Beautifier.
  • Changes/fixes:
  • Fix unwanted seek on old parsing position when log entry is stdin.
  • Try to fix a potential issue in log start/end date parsing.
  • Fix broken queries with multiline in bind parameters.
  • Add new report Sessions per application.
  • Add Json Operators to SQL Beautifier.
  • Makefile.PL: changed manpage section from '1' to '1p', fixes #237.
  • Update Copyright date-range and installation instructions that was still refering to version 5.

New in pgBadger 7.0 (May 13, 2015)

  • THIS MAJOR RELEASE ADDS SOME MORE USEFUL REPORTS AND FEATURES:
  • New report about events distribution per 5 minutes.
  • New per application details (total duration and times executed) for each query reported in Top Queries reports. The details are visible from a new button called "App(s) involved".
  • Add support to auto_explain extension. EXPLAIN plan will be added together with top slowest queries when available in log file.
  • Add a link to automatically open the explain plan on http://explain.depesz.com/
  • New report on queries cumulated durations per user.
  • New report about the Number of cancelled queries (graph)
  • New report about Queries generating the most cancellation (N)
  • New report about Queries most cancelled.
  • HERE IS THE FULL LIST OF CHANGES/FIXES:
  • Update documentation with last reports.
  • Fix number of event samples displayed in event reports.
  • Add new report about events distribution per x minutes.
  • Add app=%a default prefix to documentation.
  • Add reports of "App(s) involved" with top queries. Thanks to Antti Koivisto for the feature request.
  • Remove newline between a ) and , in the beautifier.
  • Add link to automatically open the explain plan on http://explain.depesz.com/
  • Add support to auto_explain, EXPLAIN plan will be added together with top slowest queries when available in the log file.
  • Add a graph on distributed duration per user. Thanks to Korriliam for the patch.
  • Add tree new report: Number of cancelled queries (graph), Queries generating
  • the most cancellation (N) and Queries most cancelled lists. Thanks to Thomas
  • Reiss for the feature request.
  • Fix case where temporary file statement must be retrieved from the previous
  • LOG statement and not in the following STATEMENT log entry. Thanks to Mael Rimbault for the report.
  • Add --enable-checksum to show a md5 hash of each reported queries. Thanks to Thomas Reiss for the feature request.

New in pgBadger 6.4 (Apr 14, 2015)

  • This new release fixes a major bugs in SQL beautifier which removed operator
  • and adds some useful improvement in anonymization of parameters values.
  • pgBadger will also try to parse the full csvlog when a broken CSV line is
  • encountered.
  • Make anonymization more useful. Thanks to Hubert depesz Lubaczewski for the patch.
  • Fix previous patch for csvlog generated with a PostgreSQL version before 9.0.
  • Try continue CSV parsing after broken CSV line. Thanks to Sergey Burladyan for the patch.
  • Fix bug in SQL beautifier which removed operator. Thanks to Thomas Reiss for the report.
  • Fix loop exit, check terminate quickly and correct comments indentation. Thanks to Sergey Burladyan for the patch

New in pgBadger 6.3 (Mar 27, 2015)

  • THIS NEW RELEASE FIXES SOME BUGS AND ADDS SOME NEW REPORTS:
  • A new per user details (total duration and times executed) for each query reported in Top Queries reports. The details are visible from a new button called "User(s) involved".
  • Add "Average queries per session" and "Average queries duration per session"
  • in Sessions tab of the Global statistics.
  • Add connection time histogram.
  • Use bar graph for Histogram of query times and sessions times.
  • THERE'S ALSO SOME COOL NEW FEATURES AND OPTIONS:
  • Add -L | --logfile-list option to read a list of logfiles from an external file.
  • Add support to log_timezones with + and - signs for timestamp with milliseconds (%m).
  • Add --noreport option to instruct pgbadger to not build any HTML reports in incremental mode. pgBadger will only create binary files.
  • Add auto detection of client=%h or remote=%h from the log so that adding a prefix is not needed when it respect the default of pgbadger.
  • Redefine sessions duration histogram bound to be more accurate.
  • Add new option -M | --no-multiline to not collect multi-line statement and avoid storing and reporting garbage when needed.
  • Add --log-duration option to force pgbadger to associate log entries generated by both log_duration=on and log_statement=all.
  • THE PGBADGER_TOOLS SCRIPT HAVE ALSO BEEN IMPROVE WITH NEW FEATURES:
  • Add a new tool to pgbadger_tool to output top queries in CSV format for follow-up analysis.
  • Add --explain-time-consuming and --explain-normalized options to generate explain statement about top time consuming and top normalized slowest queries.
  • HERE IS THE FULL LIST OF CHANGES/FIXES:
  • Update flotr2.min.js to latest github code.
  • Add per user detail information (total duration and times executed) for each query reported in "Time consuming queries", "Most frequentm queries" "and Normalized slowest queries". The details are visible from a new button called "User(s) involved" near the "Examples"
  • button. Thanks to Guillaume Le Bihan for the patch and tsn77130 for
  • the feature request.
  • pgbadger_tool: add tool to output top queries to CSV format, for follow-up analysis. Thanks to briklen for the patch.
  • Add geometric operators to SQL beautifier. Thanks to Rodolphe
  • Quiedeville for the report.
  • Fix non closing session when a process crash with message:
  • "terminating connection because of crash of another server process".
  • Thanks to Mael Rimbault for the report.
  • Add -L|--logfile-list command line option to read a list of logfiles from a file. Thanks to Hubert depesz Lubaczewski for the feature request.
  • Automatically remove %q from prefix. Thanks to mbecroft for report.
  • Do not store DEALLOCATE log entries anymore.
  • Fix queries histogram where range was not appears in the right order.
  • Thanks to Grzegorz Garlewicz for the report.
  • Fix min yaxis in histogram graph. Thanks to grzeg1 for the patch.
  • Add --log-duration command line option to force pgbadger to associate log entries generated by both log_duration = on and log_statement=all.
  • Thanks to grzeg1 for the feature request.
  • Small typographical corrections. Thanks to Jefferson Queiroz Venerando and Bill Mitchell the patches.
  • Reformat usage output and add explanation of the --noreport command line option.
  • Fix documentation about minimal pattern in custom log format. Thanks to Julien Rouhaud for the suggestion.
  • Add support to log_timezones with + and - signs to timestamp with milliseconds (%m). Thanks to jacksonfoz for the patch.
  • pgbadger was not recognize log files with timezones like 'GMT+3'.
  • Add --noreport command line option to instruct pgbadger to not build any reports in incremental mode. pgBadger will only create binary files. Thanks to hubert Depesz Lubaczewski for the feature request.
  • Add time consuming information in tables of Queries per database...
  • Thanks to Thomas for the feature request.
  • Add more details about the CSV parser error. It now prints the line number and the last parameter that generate the failure. This should allow to see the malformed log entry.
  • Change substitution markup in attempt to fix a new look-behind
  • assertions error. Thanks to Paolo Cavallini for the report.
  • Use bar graph for Histogram of query times and sessions times.
  • Fix wrong count of min/max queries per second. Thanks to Guillaume
  • Lelarge for the report. Add COPY statement to SELECT or INSERT statements statistics following the copy direction (stdin or stdout).
  • Fix Illegal division by zero at line 3832. Thanks to MarcoTrek for the report.
  • Add "Average queries per session" and "Average queries duration per
  • session" in Sessions tab of the Global stat. Thanks to Guillaume Lelarge for the feature request.
  • Reformat numbers in pie graph tracker. Thanks to jirihlinka for the report.
  • pgbadger_tools: Add --explain-time-consuming and --explain-normalized to generate explain statement about top time consuming and top normalized slowest queries. Thanks to Josh Kupershmid fot the feature request.
  • move everything than error information from json output when -w | --watch-mode is enable. Thanks to jason.
  • Fix undefined subroutine encode_json when using -x json. Thanks to jason for the report.
  • Add auto detection of client=%h or remote=%h from the log so that adding a prefix is not needed when it respect the default of pgbadger.
  • Redefine sessions duration histogram bound to be more accurate. Thanks to Guillaume Lelarge for the report.
  • Add connection time histogram. Thanks to Guillaume Lelarge for the feature request.
  • Add new option -M | --no-multiline to not collect multi-line statement to avoid garbage especially on errors that generate a huge report.
  • Do not return SUCCESS error code 0 when aborted or something fails.
  • Thanks to Bruno Almeida for the patch.

New in pgBadger 6.2 (Oct 17, 2014)

  • This is a maintenance release to fix a regression in SQL traffic graphs and fix some other minor issues. The release also adds a new option -D or --dns-resolv to map client ip addresses to FQDN without having log_hostname enabled on the postgresql's configuration
  • Do not display queries in Slowest individual, Time consuming and Normalized slowest queries reports when there is no duration in log file. Display NO DATASET instead.
  • Fix min/max queries in SQL traffic that was based on duration instead of query count. Fix wrong unit to Synced files in Checkpoints files report. Thanks to Levente Birta for the report.
  • Enable allow_loose_quotes in Text::CSV_XS call to fix CSV parsing error when fields have quote inside an unquoted field. Thanks to Josh Berkus for the report.
  • Add -D | --dns-resolv command line option to replace ip addresses by their DNS name. Be warned that this can slow down pgBagder a lot.
  • Thanks to Jiri Hlinka for the feature request.

New in pgBadger 6.1 (Sep 30, 2014)

  • Change -T | --title position on pgBadger report. Title now override the pgBadger label. Thanks to Julien Rouhauld for the patch.
  • Add --file-per-query and --format-query option to write each slowest query in separate file named qryXXX.sql and perform minimal formating of the queries. Thanks to Rodolphe Quiedeville for the patch.
  • Remove debug query from explain-slowest tool.
  • Fix surge in sessions number report when an exclusion or inclusion option (dbname, user, appname, etc.) is used. Thanks to suyah for the report.
  • Fix fatal error when remote log file is 0 size. Thanks to Julien Rouhaud for the report.
  • Allow pgbadger_tools --explain-slowest to automatically execute the EXPLAIN statements an report the plan. See pgbadger_tools --help for more explanation.
  • Add --analyze option to replace EXPLAIN statements by EXPLAIN (ANALYZE, VERBOSE, BUFFERS).
  • Move pgbadger_tools program and README.tools into the tools/ subdirectory with removing the extension. Add more comments and explanations.
  • Fix case where die with interrupt signal is received when using -e option. Thanks to Lloyd Albin for the report.
  • Add a new program pgbadger_tools to demonstrate how to deal with pgBadger binary files to build your own new feature. The first one 'explain-slowest' allow printing of top slowest queries as EXPLAIN statements.
  • Keep tick formatting when zooming. Thanks to Julien Rouhaud for the patch.
  • Fix automatic detection of rsyslogd logs. Thanks to David Day for the report.
  • Fix issue in calculating min/max/avg in "General Activity" report. It was build on the sum of queries duration per minutes instead of each duration. Thanks to Jayadevan M for the report.
  • The same issue remains with percentile that are build using the sum of duration per minutes and doesn't represent the real queries duration.
  • This commit also include a modification in convert_time() method to reports milliseconds.
  • Add -B or --bar-graph command line option to use bar instead of line in graph. Thanks to Bart Dopheide for the suggestion.
  • Fix Checkpoint Wal files usage graph title.

New in pgBadger 6.0 (Aug 12, 2014)

  • Javascript improvement to use only one call of sql_select and sql_format. Use jQuery selector instead of getElementById to avoid js errors when not found. Thanks to Julien Rouhaud for the patches.
  • Add -R | --retention command line option to set the maximum number of week reports to preserve in the output directory for incremental mode. Thanks to Kong Man for the feature request.
  • Session count is immediately decreased when a FATAL error is received in the current session to prevent overcount of simultaneous session number. Thanks to Josh Berkus for the report.
  • Fix issue in incremental mode when parsing is stopped after rotating log and rotated log has new lines. The new file was not parsed at all. Thanks to CZAirwolf for the report.
  • Fix revert to single thread when last_line_parsed exists. Thanks to Bruno Almeida for the report.
  • Fix issue in handling SIGTERM/SIGINT that cause pgbadger to continue.
  • Add autoclean feature to pgbadger in incremental mode. pgbadger will now removed automatically obsolete binary files unless you specify --noclean at command line.
  • Add new command line option --anonymize to obscure all literals in queries/errors to hide confidential data. Thanks to wmorancfi for the feature request.
  • Fix single "SELECT;" as a query in a report. Thanks to Marc Cousin for the report.
  • Add a copy icon in front of each query in the report to select the entire query. Thanks to Josh Berkus for the feature request.
  • Fix wrong move to beginning of a file if the file was modified after have been parsed a time. Thanks to Herve Werner for the report.
  • Allow pgBadger to write out extra files to outdir when creating incremental reports. Require the use of the -X or --extra-files option in incremental mode. Thanks to Matthew Musgrove for the feature request.
  • Fix incomplete handling of XZ compressed format.
  • Fix move to offset in incremental mode with multiprocess and incomplete condition when file is smaller than the last offset. Thanks to Herve Werner for the report.
  • Allow/improve incremental mode with multiple log file and multiprocess.
  • Fix incorrect location of temporary file storing last parsed line in multiprocess+incremental mode. Thanks to Herve Werner for the report.
  • Fix remote ssh command error sh: 2: Syntax error: "|" unexpected. Thanks to Herve Werner for the report.
  • Fix missing database name in samples of top queries reports. Thanks to Thomas Reiss for the report.
  • Add minimal documentation about JSON output format.
  • Add execute attribute to pgbadger in the source repository, some may find this more helpful when pgbadger is not installed and executed directly from this repository.
  • Fix issue with csv log format and incremental mode. Thanks to Suya for the report and the help to solve the issue. There also a fix to support autovacuum statistic with csv format.
  • Fix bad URL to documentation. Thanks to Rodolphe Quiedeville for the report.
  • Two minor change to made easier to use Tsung scenario: Remove the first empty line and replace probability by weight. Now it is possible to use the scenario as is with Tsung 1.5.
  • Fix incremental mode where weeks on index page start on sunday and week reports start on monday. Thanks to flopma and birkosan for the report.
  • Replace label "More CPU costly" by "Highest CPU-cost". Thanks to Marc Cousin for the suggestion.
  • Add query latency percentile to General Activity table (percentiles are 90, 95, 99). Thanks to Himanchali for the patch.
  • Fix typon pgbadger call. Thanks to Guilhem Rambal for the report.
  • Add JSON support for output format. JSON format is good for sharing data with other languages, which makes it easy to integrate pgBadger's result into other monitoring tools like Cacti or Graphite. Thanks to Shanzhang Lan for the patch.
  • Update documentation about remote mode feature.
  • Update documentation to inform that the xz utility should be at least in version 5.05 to support the --robot command line option. Thanks to Xavier Millies-Lacroix for the report.
  • Fix remote logfile parsing. Thanks to Herve Werner for the report.

New in pgBadger 5.1 (May 6, 2014)

  • Fix parsing of remote log file, forgot to apply some patches.
  • Support to named PREPARE and EXECUTE queries. They are replaced by the real prepare statement and reported into top queries.
  • Add new --exclude-line command line option for excluding immediately log entries matching any regex.
  • Included remote and client information into the most frequent events.
  • pgBadger is now able to parse remote logfiles using a password less ssh connection and generate locally the reports.
  • Histogram granularity can be adjusted using the -A command line option.
  • Add new detail information on top queries to show when the query is a bind query.
  • Support to logfile compressed using the xz compression format.
  • Change week/day menu in incremental index, it is now represented as usual with a calendar view per month.
  • Fix various compatibility issue with Windows and Perl 5.8
  • Changes:
  • fixed calendar display and correct typo. Thanks to brunomgalmeida for the patch.
  • revert to single thread if file is small. Thanks to brunomgalmeida for the patch.
  • print calendars 4+4+4 instead of 3+4+4+1 when looking at full year. Thanks to brunomgalmeida for the patch.
  • Add --exclude-line option for excluding log entries with a regex based on the full log line. Thanks to ferfebles for the feature request.
  • Fix SQL keywords that was beautified twice.
  • Remove duplicate pg_keyword in SQL beautifier.
  • Fix increment of session when --disable-session is activated.
  • Fix missing unit in Checkpoints Activity report when time value is empty. Thanks to Herve Werner for the report.
  • Fix double information in histogram data when period is the hour.
  • Add support to named PREPARE and EXECUTE queries. Calls to EXECUTE statements are now replaced by the prepared query and show samples with parameters. Thanks to Brian DeRocher for the feature request.
  • Included Remote and Client information into the most frequent events examples. Thanks to brunomgalmeida for the patch.
  • Fix documentation about various awkward phrasings, grammar, and spelling. Consistently capitalize "pgBadger" as such, except for command examples which should stay all-lowercase. Thanks to Josh Kupershmidt for the patch.
  • Fix incremental mode on Windows by replacing %F and %u POSIX::strftime format to %Y-%m-%d and %w. Thanks to dthiery for the report.
  • Remove Examples button when there is no examples available.
  • Fix label on tips in histogram of errors reports.
  • Fix error details in incremental mode in Most Frequent Errors/Events report. Thanks to Herve Werner for the report.
  • Fix Sync time value in Checkpoints buffers report. Thanks to Herve Werner for the report.
  • Fix wrong connections per host count. Thanks to Herve Werner for the report.
  • Allow pgBadger to parse remote log file using a password less ssh connection. Thanks to Orange OLPS department for the feature request.
  • Histogram granularity can be adjusted using the -A command line option. By default they will report the mean of each top queries or errors occurring per hour. You can now specify the granularity down to the minute. Thanks to Orange OLPS department for the feature request.
  • Add new detail information on top queries to show when the query is a bind query. Thanks to Orange OLPS department for the feature request.
  • Fix queries that exceed the size of the container.
  • Add unit (seconds) to checkpoint write/sync time in the checkpoints activity report. Thanks to Orange OLPS department for the report.
  • Fix missing -J option in usage.
  • Fix incomplete lines in split logfile to rewind to the beginning of the line. Thanks to brunomgalmeida for the patch.
  • Fix tsung output and add tsung xml header sample to output file.
  • Make it possible to do --sample 0 (prior it was falling back to the default of 3). Thanks to William Moran for the patch.
  • Fix xz command to be script readable and always have size in bytes: xz --robot -l %f | grep totals | awk "{print $5}"
  • Add support to logfile compressed by the xz compression format. Thanks to Adrien Nayrat for the patch.
  • Do not increment queries duration histogram when prepare|parse|bind log are found, but only with execute log. Thanks to Josh Berkus for the report.
  • Fix normalization of error message about unique violation when creating intermediate dirs. Thanks to Tim Sampson for the report.
  • Allow use of Perl metacharacters like [..] in application name. Thanks to Magnus Persson for the report.
  • Fix dataset tip to be displayed above image control button. Thanks to Ronan Dunklau for the fix.
  • Renamed the Reset bouton to "To Chart" to avoid confusion with unzoom feature.
  • Fix writing of empty incremental last parsed file.
  • Fix several other graphs
  • Fix additional message at end of query or error when it was logged from application output. Thanks to Herve Werner for the report.
  • Fix checkpoint and vacuum graphs when all dataset does not have all values. Thanks to Herve Werner for the report.
  • Fix week numbered -1 in calendar view.
  • Change week/day menu in incremental index, it is now represented as usual with a calendar view per month. Thanks to Thom Brown for the feature request.
  • Load FileHandle to fix error: Can not locate object method "seek" via package "IO::Handle" with perl 5.8. Thanks to hkrizek for the report.
  • Fix count of queries in progress bar when there is compressed file and multiprocess is enabled. Thanks to Johnny Tan for the report.
  • Fix debug message "Start parsing at offset"
  • Add ordering in queries times histogram. Thanks to Ulf Renman for the report.
  • Fix various typos. Thanks to Thom Brown for the patch.
  • Fix Makefile error, "WriteMakefile: Need even number of args at Makefile.PL" with Perl 5.8. Thanks to Fangr Zhang for the report.
  • Fix some typo in Changelog

New in pgBadger 5.0 (Feb 8, 2014)

  • Fix size of SQL queries columns to prevent exceeding screen width.
  • Add new histogram reports on top normalized queries and top errors or event. It shows at what hours and in which quantity the queries or errors appears.
  • Add seeking to last parser position in log file in incremental mode. This prevent parsing all the file to find the last line parse from previous run. This only works when parsing a single flat file, -j option is permitted. Thanks to ioguix for the kick.
  • Rewrite reloading of last log time from binary files.
  • Fix missing statistics of last parsed queries in incremental mode.
  • Fix bug in incremental mode that prevent reindexing a previous day. Thanks to Martin Prochazka for the great help.
  • Fix missing label "Avg duration" on column header in details of Most frequent queries (N).
  • Add vertical crosshair on graph.
  • Fix case where queries and events was not updated when using -b and -e command line. Thanks to Nicolas Thauvin for the report.
  • Fix week sorting on incremental report main index page. Thanks to Martin Prochazka for the report.
  • Add "Histogram of query times" report to show statistics like 0-100ms : 80%, 100-500ms :14%, 500-1000ms : 3%, > 1000ms : 1%. Thanks to tmihail for the feature request.
  • Format mouse tracker on graphs to show all dataset value at a time.
  • Add control of -o vs -O option with incremental mode to prevent wrong use.
  • Change log level of missing LAST_PARSED.tmp file to WARNING and add a HINT.
  • Update copyright date to 2014
  • Fix empty reports of connections. Thanks to Reeshna Ramakrishnan for the report.
  • Fix display of connections peak when no connection was reported.
  • Fix warning on META_MERGE for ExtUtils::MakeMaker < 6.46. Thanks to Julien Rouhaud for the patch.
  • Add documentation about automatic incremental mode.
  • Add incremental mode to pgBadger. This mode will build a report per day and a cumulative report per week. It also create an index interface to easiest access to the different report.
  • PostgreSQL log file rotation.
  • Add -O | --outdir path to specify the directory where out file must be saved.
  • Automatic queries formatting is now done on double click event, simple click was painful when you want to copy some part of the queries. Thanks to Guillaume Smet for the feature request.
  • Remove calls of binmode to force html file output to be utf8 as there is some bad side effect. Thanks to akorotkov for the report.
  • Remove use of Time::HiRes Perl module as some distributions does not include this module by default in core Perl install.
  • Fix "Wide character in print" Perl message by setting binmode to :utf8. Thanks to Casey Allen Shobe for the report.
  • Fix application name search regex to handle application name with space like "pgAdmin III - Query Tool".
  • Fix wrong timestamps saved with top queries. Thanks to Herve Werner for the report.
  • Fix missing logs types statitics when using binary mode. Thanks to Herve Werner for the report.
  • Fix Queries by application table column header: Database replaced by Application. Thanks to Herve Werner for the report.
  • Add "Max number of times the same event was reported" report in Global stats Events tab.
  • Replace "Number of errors" by "Number of ERROR entries" and add "Number of FATAL entries".
  • Replace "Number of errors" by "Number of events" and "Total errors found" by "Total events found" in Events reports. Thanks to Herve Werner for the report.
  • Fix title error in Sessions per database.
  • Fix clicking on the info link to not go back to the top of the page. Thanks to Guillaume Smet for the report and solution.
  • Fix incremental report from binary output where binary data was not loaded if no queries were present in log file. Thanks to Herve Werner for the report.
  • Fix parsing issue when log_error_verbosity = verbose. Thanks to vorko for the report.
  • Add Session peak information and a report about Simultaneous sessions. log_connections+log_disconnections must be enabled in postgresql.conf.
  • Fix wrong requests number in Queries by user and by host. Thanks to Jehan-Guillaume de Rorthais for the report.
  • Fix issue with rsyslog format failing to parse logs. Thanks to Tim Sampson for the report.
  • Associate autovacuum and autoanalyze log entry to the corresponding database name. Thanks to Herve Werner for the feature request.
  • Change "Simultaneous Connections" label into "Established Connections”.

New in pgBadger 4.1 (Nov 20, 2013)

  • Add log_autovacuum_min_duration into documentation in chapter about postgresql configuration directives. Thanks to Herve Werner for the report.
  • Add chapter about "Incremental reports" into documentation.
  • Fix reports with per minutes average where last time fraction was not reported. Thanks to Ludovic Levesque and Vincent Laborie for the report.
  • Fix unterminated comment in information popup. Thanks to Ronan Dunklau for the patch.
  • Add --exclude-appname commande line option to eliminate unwanted traffic generated by a specific application. Thanks to Steve Crawford for the feature request.
  • Allow external links use into URL to go to a specific report. Thanks to Hubert depesz Lubaczewski for the feature request.
  • Fix empty reports when parsing compressed files with the -j option which is not allowed with compressed file. Thanks to Vincent Laborie for the report.
  • Prevent progress bar length to increase after 100% when real size is greater than estimated size (issue found with huge compressed file).
  • Correct some spelling and grammar in ChangeLog and pgbadger. Thanks to Thom Brown for the patch.
  • Fix major bug on SQL traffic reports with wrong min value and bad average value on select reports, add min/max for select queries. Thanks to Vincent Laborie for the report.

New in pgBadger 2.2.1 (Nov 22, 2012)

  • Update PostgreSQL keywords list for 9.2
  • Fix number of queries in progress bar with tsung output.
  • Remove obsolete syslog-ng and temporary syslog-ll log format added to
  • fix some syslog autodetection issues. There is now just one syslog
  • format: syslog, differences between syslog formats are detected and
  • the log parser is adaptive.
  • Add comment about the check_incremental_position() method
  • Fix reports with empty graphs when log files were not in chronological
  • order.
  • Add report of current total of queries and events parsed in progress
  • bar. Thanks to Jehan-Guillaume de Rorthais for the patch.
  • Force pgBadger to use an require the XS version of Text::CSV instead
  • of the Pure Perl implementation. It is a good bit faster thanks to
  • David Fetter for the patch. Note that using csvlog is still a bit
  • slower than syslog or stderr log format.
  • Fix several issue with tsung output.
  • Add report of shut down events
  • Add debug information on command line used to pipe compressed log
  • file when -v is provide.
  • Add -U | --exclude-user command line option to generate report
  • excluded user. Thanks to Birta Levente for the feature request.
  • Allow some options to be specified multiple time or be written as a
  • coma separated list of value, here are these options: --dbname,
  • --dbuser, --dbclient, --dbappname, --exclude_user.
  • Add -S | --select-only option to build report only on select queries.
  • Add first support to tsung output, see usage. Thanks to Guillaume
  • Lelarge for the feature request.
  • Add --include-query and --include-file to specify regex of the queries
  • that must only be included in the report. Thanks to Marc Cousin for
  • the feature request.
  • Fix auto detection of log_duration and log_min_duration_statement
  • format.
  • Fix parser issue with Windows logs without timezone information.
  • Thanks to Nicolas Thauvin for the report.
  • Fix bug in %r = remote host and port log line prefix detection.
  • Thanks to Hubert Depesz Lubaczewski for the report.
  • Add -C | --nocomment option to remove comment like /* ... */ from
  • queries. Thanks to Hubert Depesz Lubaczewski for the feature request.
  • Fix escaping of log_line_prefix. Thanks to Hubert Depesz Lubaczewski
  • for the patch.
  • Fix wrong detection of update queries when a query has a object names
  • containing update and set. Thanks to Vincent Laborie for the report.