Sawmill Changelog

What's new in Sawmill 8.7.7.4

Oct 6, 2015
  • Bugs fixed:
  • [1326127] Fixed an issue with info.cfg where table_num_rows would show an incorrect number for main table after a database update was performed.
  • [1326764] Fixed an issue that would cause an error to occur when a Snapon that was attached during profile creation was detached.
  • [1328253] Reports of profiles without date/time support and applied filters show two "Clear Filters" buttons.
  • [1328463] Report elements with "Use overview for totals" and "Table filter expression" may cause an error.
  • [1328683] Email addresses containing an apostrophe or other non-alpha numeric printable ASCII characters did not pass email validation, respectively were indicated as invalid email address.
  • [1329342] The Manage Fields dialog window in the Report Element Editor does not open under certain circumstances such as when creating a new report element which erroneously has a duplicate aggregating field.
  • New features:
  • [1329553] Resolved the issue where the Apple OS X Gatekeeper mistakenly reports the Sawmill MacOS DMG as "damaged".

New in Sawmill 8.7.6.2 (May 21, 2015)

  • Bugs fixed:
  • [1321194] A schedule with a report filter that runs at a configured time sometimes does not apply the filter to the report.
  • [1321466] Setting the CSV empty value to empty string "" still shows the "(empty)" value when the report field has a custom display format type for export defined.
  • [1324830] Inactive report filter items in admin scheduler are displayed as active/checked although the filter has no effect.
  • New features:
  • [1319403] Allows date_time database field to have the number of bits to be configureable via the browser. It had been fixed at 16 bits, which caused this problem due to an overflow.
  • [1320092] Added support for TLS/SSL email connections via SMTP.
  • [1323486] Added support for the new OCLC EZproxy standard log format.

New in Sawmill 8.7.5 (Jan 30, 2015)

  • Bugs fixed:
  • [1314766] Generating a report on Windows with multiple report filters sometimes results in an error like: Can't delete file LogAnalysisInfo\\TemporaryFiles\\illocal_xxx (Permission denied)
  • [1316024] This illocal file deletion bug in Windows has been fixed for release 8.7.5.
  • [1316990] Database Tools > Process Logs, added page description so that it doesn't become confused with Build Database.
  • [1317933] Clicking tabs in Admin/Roles show an alert message.
  • [1317984] HTTPS web server now uses TLS protocol, replacing SSLv3 protocol. This allows it to work in latest version of Firefox.
  • [1320412] Choosing the "Professional" or "Lite" trial tier during setup has no effect.
  • [1320413] Cannot save reports in reports editor after changing the order of reports. The reports editor states "No changes to save".
  • New features:
  • [1309202] Added support for Zentyal log format.
  • [1312748] Added support for Bluecoat SGOS6.2/6.4 format.
  • [1313210] Added support for Checkpoint log file format.
  • [1313664] Added support for Optenet log format.
  • [1313713] Added support for Trend Micro Control Manager.
  • [1313714] Support for Trend Micro Deep Security has been added.
  • [1313715] Support for Microsoft Exchange Server 2013 has been added.
  • [1313899] Support for Kiwi ISO Event Sentry Added
  • [1313900] Added support for i-Filter log file format.
  • [1315968] Added support for IWSVA log file format.
  • [1319225] Added support for Gene6 SARL FTP log format.

New in Sawmill 8.7.3.2 (Apr 22, 2014)

  • Bugs fixed:
  • [1296642] When "Use overview for totals" is checked, the Other Items row can show a negative number. (This is fixed by showing a dash in this case).
  • [1299899] If a report is generating, and a database-writing task like a rebuild or update runs, the database will be altered underneath the running report, possibly causing an error in report generation. This is now fixed by delaying the rebuild or update until no reports are running.
  • [1301487] Apache Custom profiles are created with a duplicate file_type field, resulting in a build error, "Trying to compute 'file_type' field, but there is no 'page' field to compute it from."
  • [1301592] Added info text to Profiles View feature permission in Roles, visible when Edit is checked.
  • [1301593] Added Admin Tools feature permission in Roles.
  • [1301594] Internet Explorer 8, IE9 and IE10 hangs when loading the profiles page and when Internet Explorer's "View all websites in Compatibility View" option is checked.
  • [1301596] Fixed minor language variable issues in Admin/Profiles and Admin/Tools.
  • [1301665] Remainder row shows negative values for fields with aggregation method unique when "Use overview for totals" is checked.
  • [1301907] In the analysis of Juniper SSL VPN log data, "VPN Tunneling: Session ended for user with IP" lines are ignored, possible resulting in very long reported session durations for those sessions.
  • [1301918] When using the internal database, indices do not improve the performance of single-value report filters on single fields.
  • [1302023] Multiprocessor SSQL queries (e.g., from main table reports generated with "query splitting" turned on) crash or generate an internal error.
  • [1302306] Attempting to create a profile from log data in Windows 2000/2003/2008 DNS Log Format, given an error "no date_time in snapons" during the Create Profile Wizard.
  • [1302380] Reports generated from MDaemon 13 logs show no events.
  • [1302651] Sending a report by email with multiple recipient email addresses causes a SMTP server error in some mail servers.
  • [1302908] When analyzing servuftp logs, the Log Detail report fails with an error, "Unexpected text at end of SSQL field description: '_file'"
  • [1303189] The profiles list is not properly sorted after a profile has been renamed.
  • [1303428] Editing the last action in scheduler overrides the first action with the last action.
  • [1303868] Report filter items of type within/matches are not added and not editable in Reports/Filters.
  • [1303979] Manage fields in an overview report element allows to add non-aggregating fields which cause an error in reports. Fixed so that only aggregating fields can be added in overview report elements.
  • [1304170] A report element which displays one or more chronological graphs without a table and with no filters applied causes an "Unable to read file" error. The error only occurs if the profile uses the internal database, profiles with an external database are not affected.
  • [1304248] Sawmill Lite displays a "No Permission" page when navigating to the Config/Log Source page or to a Tools page.
  • [1304349] When creating a MySQL profile with a non-default port, an error occurs at the end of profile creation like, "Failed to connect to MySQL database at 127.0.0.1:3302 with username 'user'"
  • [1304406] When a regular expression table filter, or "omit parenthesized items," is used on a report column of type non-aggregating integer, an error occurs like, "Internal: attempt to get string value from non-varchar column 0 of table 'xref2' with GetStringCell()".
  • New features:
  • [1285416] Display a "Before you start" section in Admin/Profiles to new users, it is only shown after a new setup. The section reminds new users to disable Antivirus software for the Sawmill directory and shows some links to best practice guides when processing large datasets.
  • [1299190] Added support for IronPort S-Series with pseudo-W3C (pattern) headers.
  • [1301348] Added support for WebLogic 10 log format (by extending the existing WebLogic 8 plug-in so it can handle both).
  • [1301358] Added support to show a warning message for missing log fields in the New Profile Wizard. The warning message will be shown if required log fields are defined in a warn_on_missing_log_fields node in the log format plug-in and if these log fields are not detected in the log files.
  • [1301531] Added support for Kerio Control security log format.
  • [1301630] Added rendering of very large byte counts as Terabytes, Petabytes, and Exabytes.
  • [1301689] Substantially rewrote support for the OpenVPN plug-in so it supports the latest format of log data, parses faster, supports any syslog header, and more.
  • [1301696] Enhanced IPCop Syslog support to handle date header lines.
  • [1301859] Added support for NPR Reporting log format.
  • [1301902] When reporting on Cisco PIX/IOS log data, dynamic Teardown lines are ignored, resulting in lower than expected reported duration.
  • [1301976] Added support for Cisco IronPort S-Series (WSA) CSV export format.
  • [1302060] Added option to Report Options/CSV Export to convert the "(empty)" value to any custom defined value.
  • [1302205] Display operating system and architecture in About window.
  • [1302286] Display a warning message in File Manager/Network Shares when the "Map drive letter" is selected.
  • [1302301] Added support for F-Secure HTTP Access log format
  • [1302454] Added support for Zimbra mail server log format
  • [1302671] Added support for IPCop Snort multiline log format.
  • [1302844] Added support for FreeProxy log format
  • [1303041] Added support for Sophos UTM Web Application Firewall log format.
  • [1303190] Display best practice tips in New Profile Wizard on log source and database page and in Scheduler.
  • [1303191] Improved styling of the Help Manual and added a Best Practice Guides section.
  • [1303424] Added support for Websense Server log format.
  • [1303461] Added "Show/Hide Created By Column" in Profiles/View menu. The column can now be set per user. User access can be set in Admin/Roles.
  • [1303462] Moved "Show/Hide Database Info Column" from Profiles View Editor to Profiles/View menu. The column can now be set per user. User access can be set in Admin/Roles.
  • [1303794] Added support for FortiGate 300 Series log format.
  • [1304249] The "database info column" is now visible by default in the Profiles list.
  • [1304335] Added support for InterMapper Chart log format.

New in Sawmill 8.7.2 (Feb 3, 2014)

  • Bugs fixed in version 8.7.2:
  • [1296399] Error messages from errors during parsing in multiprocessor mode, sometimes have embedded HTML code in them.
  • [1296656] If a profile and its database is deleted, and the database is internal and uses a custom database directory, the directory is not fully deleted, but still exists and contains one file, info.cfg.lock.
  • [1299432] If there is no new data in the log source, single-processor database updates will give an error like, "## Attempt to read beyond end of LogAnalysisInfo/Databases/(PROFILE)/main/Tables/f_main_table_p0/data.tbl (fileSize=0); attempted to read from 0 to 214."
  • [1300566] If Flash Media Server log data contains fields with embedded spaces, some field values will be put into the wrong fields during parsing.
  • [1301306] On small mobile devices, In the Scheduler, when adding a New Item to the Report Filters, the window opens partly off-screen, and runs away when chased by scrolling.
  • [1301307] The profiles menu displays profiles for which a user has no access permission, though only when logging in with different usernames and access permissions on the same computer.
  • [1301309] Clicking on a sorted report column does not change the sort direction under certain circumstances.
  • New features in 8.7.2:
  • [1299895] Added support for MOVEit DMZ SSH log format.
  • [1300599] Changed the "strftime:" option for custom "display format type" to use local time instead of UTC.
  • [1300848] Added support for Internet Explorer 11 detection (user agents with "Trident").
  • [1301207] Added support for Bitvice WinSSHD XML log format.

New in Sawmill 8.7.1.3 (Jan 24, 2014)

  • Bugs fixed in version :
  • [1292350] On SPARC Solaris, database builds usually crash.
  • [1293936] Running the delete_database_field command-line action, on a profile with report fields which have no database field (e.g., expression fields), gives an error, "Internal Error: Empty node name"
  • [1294383] When using an "unnormalized string" field with a dataset of more than about 5 million lines, a database build will crash.
  • [1296757] When updating a database, if skip-by-pathname is on and file-by-file is on, the update re-adds previously seen files.
  • [1299145] When importing a database into a fresh profile, the hierarchy tables are only automatically rebuilt for those fields which have hierarchical xref tables. So most hierarchy tables are not built, and hierarchical reports fail with an error like, "Unable to read file LogAnalysisInfo/Databases/{profile}/main/Tables/geo_countrysubitem/header.cfg (Operation timed out)". (As a workaround, rebuild the hierarchies manually with "sawmill -p {profile} -a rdh").
  • [1299296] There are duplicate File Type, Screen Depth, and Screen Dimensions log fields in a profile created with the Web Server Package snapon.
  • [1299697] Clicking next rows on a table which already displayed the last row returned previous rows.
  • [1299698] Missing row numbers and Rows button in Path Through a Page report element.
  • [1299723] After manually adding the Device Type snapon to a profile, a database build fails with the error, "Syntax error: Expected variable, subexpression, or identifier -- found =".
  • [1299892] In sendmail analysis, in status=Sent lines with multiple recipients, the last email address is not reported, and the second-to-last email address is reported twice.
  • [1299960] Profile conversion of older version 8.0 and version 8.1 profiles does not start when clicking "Click here to convert the profiles".
  • [1299961] Delete profiles is not functional when the Admin/Profiles page displays older version 7.x, 8.0 or 8.1 profiles.
  • [1299962] Update Database and Build database hangs at "loading" in the Reports GUI when logged in as non-root-admin user.
  • [1299963] Config Options and Tools menu is not visible in Reports and Config when logged in as non-root-admin with Config and Tools permissions.
  • [1299965] Profiles View permissions are not available in Admin/Roles. Edit Profiles View is only available to root-admin.
  • [1300154] Admin/Profiles page has no margin when no profile exists or when it contains invalid profiles from older versions.
  • [1300156] New profile is not immediately displayed in profiles list.
  • [1300158] Admin/Profiles misses language variables for translation.
  • [1300592] Reports with complex filters run on Windows can give an error like, "Can't delete file LogAnalysisInfo\Databases\PROFILE\main\Tables\filtertmp_7804_2\indices\itemnum\header.dat (Permission denied)"
  • [1300735] Admin/Profiles page does not show any profile when additional report columns are defined and when logged in as non-root-admin user.
  • New features:
  • [1292766] Added support for Watchguard Firebox logging via Syslog Watcher (new plug-in for Syslog Watcher, and enhanced Watchguard Firebox XTM plug-in to handle slight variant format.
  • [1294243] Added support for IBM HTTP Server Common Log Format.
  • [1299192] The remove_database_data action now allows the date_filter (-df) command line option, and discards whatever is *not* in the filter set, so "-df 30day" will expire everything older than 30 days.
  • [1300869] Changed the way report filters are done internally in some cases (filtering on main table with "within" or date range filters), to reduce temporary disk usage and increase performance.

New in Sawmill 8.7.0.4 (Dec 9, 2013)

  • Bugs fixed:
  • [1285838] Profiles using Kiwi (mm/dd/yyyy) syslog format, do not report values for any fields other than the syslog header.
  • [1288265] Deleting a MySQL profile with database results in an error, 'Unknown configuration group "admin_pages" in node ""'
  • [1288440] If a database contains no data, the Sessions Overview gives a cryptic error, "Internal error: mapping 'LogAnalysisInfo\Databases\{profile}\main\Tables\session_users_stage1\sets\sessions' read-only, but its lists.dat"
  • [1288998] PDF rendering cuts off the date range display at the far right of the page.
  • [1289649] Fixed ShoutCAST 1.8 log format support; it was incorrect categorized as a Gateway device instead of a Media Server, so some of the reports were odd.
  • [1290915] After performing a "remove database data" operation on an internal database, filtered reports which use indices may give incorrect (too large) numbers. (workaround: rebuild the indices with "sawmill -p {profile} -a rdi" after rdd).
  • [1291100] When a snapon is detached, if it added a database field during profile creation, some traces of it remain in the profile (and can cause errors), including references in auto-generated cross-reference groups, and auto-generated reports.
  • [1292125] References to row_number in report field expressions give an error, "Unknown variable 'row_number' in expression"
  • [1292191] The Play Duration reported by Wowza and Flash can be slightly too high; it double-counts some duration when a session ends with a pause, followed by a stop.
  • [1292206] When displaying a Log Detail report, with a filter, and paging forward, the second page will sometimes show no results.
  • [1292709] When zooming in the reports UI, the "zoom" window appears too low, to wide, and transparent.
  • [1293226] On multiprocessor systems with Enterprise licensing and "split queries" enabled, even when queries are large enough to be split among processors, they are not.
  • [1293275] When a new field is created in the New Field Wizard, the resulting report element has no label.
  • [1293441] When using a within/matches filter with a report field which has no corresponding database field, an error can occur like "Unknown variable 'page_directory' in expression".
  • [1293446] On non-Windows servers, if a scheduled task to email a report uses a subject containing parentheses but no spaces, an error will occur when the schedule is run, "-bash: syntax error near unexpected token `('" (workaround: put the subject in double-quotes).
  • [1293517] A CSV export of the Cities report contains HTML tags in the city names. And, the Display Format Type of the Cities field appears as "Bandwidth" in the UI.
  • [1293861] Wowza sessions ending with "destroy" instead of "stop" are not counted in play duration.
  • [1293885] IIS web log profiles do not include referrer analysis (search engines and search phrases), and do not simplify the c-referer field.
  • [1294553] When deleting a profile with an external database, and checking the "Drop database" checkbox, the UI hangs after deletion.
  • [1295435] Save As New Report does not save the defined date filter.
  • [1295437] Filtering on report fields without underlying database fields (e.g., expression fields like Bounce Rate), gives a syntax error like, "Syntax error: Expected variable, subexpression, or identifier -- found".
  • [1295520] If a log format contains an "average" field (aggregation method "average"), and the denominator field doesn't exist, and the field is checked in Create Profile Wizard, it will appear in reports as a simple summing field with the same values as the underlying numerator field (rather than not being included in reports).
  • [1295828] Profiles created from Apache custom log data containing user-agent or referrer, have duplicate versions of the reports derived from those fields.
  • [1296318] "Actions emails" do not have a subject or a return address.
  • [1297066] The "time elapsed" part of the progress display is sometimes wrong (suddenly drops down to a lower time).
  • [1297590] A cross-scripting vulnerability exists which allows a carefully crafted URL to run arbitrary JavaScript code in the client browser.
  • [1298158] Changing the date time format in lang_stats.cfg had no effect in reports.
  • [1298879] If an SMTP password contains a plus (+), it is passed incorrectly to the server, resulting in failed authentication.
  • [1298923] Profiles created from Flash Media Server logs have multiple "byte" database fields which are normalized as type string, and non-aggregating, and have xref groups as though they were normal non-aggregating fields; these fields are not used directly in reporting and should not have xrefs; and they are integers and should not be normalized as itemnums. In short, complexity of Flash profiles has been reduced, and performance has been improved.
  • [1299365] Canceling profile deletion by clicking No in the confirm delete window interpreted the click as Yes and deleted the profile in Internet Explorer 9.
  • New features:
  • [1288348] Added a new database field type, "unnormalized string," which does not use itemnums to normalize string fields. This is useful for fields which have a large number of unique values, and can greatly improve scalability while retaining full detail of a field.
  • [1291946] Added support for Savvion BPM log format.
  • [1292044] Added a new option attrs parameter to the built-in Salang function ldap_search(); this allows non-user attributes to be queried, which is necessary to use certain LDAP configurations for login.
  • [1293647] Added support for Lotus Notes log format.
  • [1294046] Added support for Windows Event log format (XML).
  • [1295126] Added support for SiteMinder Apache WebAgent Log Format
  • [1295174] Added support for a variant of McAfee Web Gateway log format (version 7.2).
  • [1295942] Added support for Limelight SHOUTcast Service log format.
  • [1296470] Added support for WebLogics Diagnostic log format.
  • [1296714] Extended the Filemaker Access log format plug-in to support version 11.
  • [1297159] Changed the "remove database data" query in MSSQL to remove data in batches of 1 million rows, to keep it from overloading the transaction log for very large datasets.
  • [1297215] Added support to define a custom filename in Reports Export Table.
  • [1297216] Improved reports and config navigation menus and fixed miscellaneous styling issues.
  • [1297217] Added a profiles drop down menu in Reports and Config navigation bar. This allows to switch between profiles without navigating to Admin/Profiles.
  • [1297218] Added a filter field in Admin Profiles. This allows to filter/search the profiles list by profile names.
  • [1297219] Improved admin profiles list loading performance by caching the profiles list in the web browser.
  • [1297221] Improved report element controls, layout and style.
  • [1297222] Added support to run Process Logs from the web user interface. Process Logs is located in the new Tools menu in Config and Reports.
  • [1297405] Enhanced Hurricane MTA support to report the EHLO responses, so it is possible to filter on them, for instance to show only connections supporting TLS.
  • [1298119] Added support for MOVEit DMZ log format
  • [1299364] Added Admin/Profiles dashboard to display basic report values, display the profiles database state and to update/build profiles databases.

New in Sawmill 8.6.2.1 (Jul 10, 2013)

  • Bugs fixed:
  • [1288964] In Wowza analysis, for profiles created with Sawmill 8.6.2, play duration is overreported in cases where there are "unpause" events in the logs.
  • [1289068] When attempting to delete an MS SQL profile through the web interface, along with its database, an error occurs, 'Unknown configuration group "admin_pages" in node ""'.
  • [1289278] The new profile wizard does not show the database page in the Pro version.
  • [1289335] Config Reports Editor - "Link to report" list is not properly updated when renaming, adding or deleting new reports in reports editor.
  • [1289722] In some circumstances, a Flash Media Server profile (or possibly other profiles which redefine their log fields during database building) will fail to build with an "Empty node name" error.
  • [1289735] "Link to report" did not open the report in static (generated) report files.
  • [1289736] The reports menu was not shown in static (generated) report files.
  • [1289900] When creating a profile using MSSQL as the database, an error can occur when entering the database name, 'The MS SQL database name must start with a Unicode letter or the characters "_", "@", "#"; followed by one or more letters, numbers or the characters "_", "@", "#", "$". Please define a different database name.'
  • [1289956] When reporting on Chinese Windows syslog files, some characters are garbled in reports.
  • [1290318] A Flash Media Server profile created with the duration field unchecked, has no reports.
  • [1290495] If a SQL prefix is used, an error can occur on database update, "Unable to Execute ODBC Query='select count(*) from main_table_update'; diagnostics=ODBC error: rec1: SQLstate: S0002; msg=[Oracle][ODBC][Ora]ORA-00942: table or view does not exist"
  • [1290731] Fixed a security issue in the update and build database page.
  • [1291076] Profiles created from Akamai Streaming W3C logs, do not track the final field in log.
  • [1291263] Enhanced display format type settings in report fields. Display format types can now be specified for all aggregating fields in exported reports, they can also be different from the HTML settings.
  • New features:
  • [1288297] Added support for Cisco eCDS log format.
  • [1288692] Added support for SocketLabs Hurricane MTA log format.
  • [1289692] Added "Bounces" and "Bounce Rate" as a standard part of the Web Server Package (affecting all plug-ins which use that package, including Apache Extended, IIS, and many others.
  • [1290423] Added support for DataEnter Xwall log format.
  • [1290844] Added support for {==} (Salang) sections in the database directory parameter, enabling, for instance, automatic creation of a new database every day.

New in Sawmill 8.6.0.2 (Dec 13, 2012)

  • Bugs fixed:
  • [1263664] The built-in web server fails to load pages or files, sporadically, on non-Windows systems.
  • [1272644] When importing a v7 profile without a database to 8.5, session information (reports and fields) is not carried over to the new profile.
  • [1272731] "Remove Database Data" operations did not correctly check for existing database-writing processes, potentially causing corruption of a database if they begin during a database update or rebuild.
  • [1274531] When building a database from Juniper MFC 12 (W3C) data without suppress_cs_range or suppress_etag fields, an error occurs, reporting the absence of these fields.
  • [1274718] Network actions like create_user, which shouldn't require -p, generate an error if p is not specified, "Unknown variable 'internal.profile_name' in expression"
  • [1274963] After attaching a "Report Field Ratio" snapon, reports give an error, "Unknown database field '{fieldname}' in v.query_result.header"
  • [1275020] Report filters on numerical fields using >= give incorrect results (often, filtering out nothing).
  • [1275153] If a version 8.1 profile has a "unique" database field of Type "string", conversion to 8.5 will result in a version 8.5 profile which fails on database build with an error like, "Internal: Attempt to find main table column number from database field 29 [visitors], but there is no such column in main_table"
  • [1275902] If a database field is non-hierarchical, and there is a cross-reference group of it which is hierarchical, the corresponding report will show an extra blank line counting all events.
  • [1276172] If the web server is running more than 30 days straight, tasks may begin to fail immediately after starting (this is due to MasterProcessLock files being prematurely deleted).
  • [1276578] Attaching the Geographic Location Information from Config -> Snapons gives an error, "Snapon attempted to add database field 'location', which already exists"
  • [1276952] Improved detection of libcrypto during "configure" on Linux, to handle systems with limited versions of the library.
  • [1276987] The "Other Rows" line in tables is incorrect, containing one of the rows which is visible.
  • [1277076] The "omit parenthesized items" item is not saved when using the Customize link in Reports for pivot tables.
  • [1277141] Session information is lost when importing a Sawmill 7 profile through the web interface.
  • [1277448] When using an SFTP log source, entering "/" for the pathname does not show the files in /. (workaround: use "/*").
  • [1277702] Generating a report where one of the report elements has a label containing a double-quote, gives an error like, "Unexpected = in group node (v.progress.step.0.abc def ))".
  • [1278066] The delete_user action (or network action) gives an error, "Syntax error: Unknown variable 'profile_name' in expression."
  • [1278917] The command-line progress display often displays less than 100% at the end of a successfully completed action.
  • [1279444] A database build can crash if there are no xref and no indices and no database filters.
  • New features:
  • [1272161] itemnum tables are no longer indexed, when the corresponding database field is set to not be indexed.
  • [1273614] Separately implemented country/region/city support as a snapon, for better modularity; this also makes it possible to have more than one Geo analysis in a single profile (based on different IP fields).
  • [1274335] Greatly improved the performance of filtered Log Detail reports, by eliminating the calculation and display of the total available rows. This decreases the time from 169 seconds to 7 seconds in one 180-million-line dataset.
  • [1274617] Created a new snapon, Create Default Xref Groups, which at the time it is attached, creates default xref groups for all database fields (each xref group having that field, date/time, and all aggregating fields), and all reports (each xref group having all fields in the report). This is similar to what has happened automatically at profile creation, in earlier versions, but it is now possible to reset the xrefs to optimal configuration after adding or removing fields or reports.
  • [1274618] Log Detail is now ordered automatically with the timestamp at the left, followed by the non-aggregating fields in the order they appear in the Report Fields, followed by the aggregating fields in the order they appear in Report Fields. In previous versions, all fields were in Report Fields order, which can give undesirable orderings, especially when some fields are created with snapons.
  • [1275211] Added error message display when JavaScript is disabled.
  • [1275333] Added a white line between adjacent slices in 2D pie charts, for better contrast.
  • [1275344] Enhanced LDAP login plug-in so user roles and profile permissions can be managed through the Sawmill web interface, and will not be overwritten each time by the LDAP login.
  • [1277169] Added support for McAfee Web Gateway log format.
  • [1277726] The Countries/Regions/Cities reports have been somewhat restructured, when they are created from a snapon (as is currently the case for Apache logs, Common Access Log Format, and IIS logs; more will follow). The reports no longer use a hierarchy, but instead use custom fields with custom formatting, which give a cleaner and easier-to-read appearance to the names of the regions and cities. Some city and region categories have also been consolidated.
  • [1277785] Added support for Sonicwall NSA (Network Security Appliance) log format.
  • [1277837] Added support for Smoothwall Network Guardian and Advanced Firewall log format.
  • [1278135] Added support for IceCast Playlist log format.
  • [1278289] Added a snapon to report Service Name, e.g., "HTTP" computed from port 80, protocol TCP.
  • [1278764] Improved progress reporting to show "scanning log source" as a separate stage after "erasing database."
  • [1278851] Added support for Websense log format.
  • [1278870] Enhanced support for Microsoft DHCP log format, to handle non-syslog version, and missing field values in some fields.
  • [1279121] Added the option to build all database field hierarchies from the command line with "-a rdh" by omitting the -fn parameter.
  • [1280817] Added Dashboard functionality. This is a collection of features—side-by-side report elements that flow and wrap to maximize visible data, simpler and smaller versions of reports and graphs, and a number of other report element options—which can be used to implement simple "dashboard" style reports with many small graphs or tables in a two-dimensional layout. Used this functionality in the Web Server Package snapon to implement a Dashboard for Apache Combined, IIS, and Common Access log formats (more to come).
  • [1280820] Enhanced reporting of Web Browser information, for plug-ins that use Web Server Package (currently, IIS, Apache Combined, and Common Access). The new report shows browser name, major version, and full browser version in three separate report elements in one report.
  • [1280821] Cleaned up and improved the appearance of reports and graphs in a variety of small ways.

New in Sawmill 8.5.7.3 (Jun 14, 2012)

  • Bugs fixed:
  • [1258482] Certain large datasets can crash during the database filtering step of a database build.
  • [1261954] Conversion of a Sawmill 7 database files with an empty error; conversion of a Sawmill 7 profile seems to succeed, but gives an error on viewing reports.
  • [1264206] When using SFTP to process files compressed with bzip2, an error can occur like, "SSH connection failed: read_packet(): Packet len too high(1608634376 5fe1d008)"
  • [1264277] If a custom report field uses another report field to calculate its value, and that report field is not a visible column in the report, it will give an error like, "Unknown column 'accesses' in cell_by_name()."
  • [1264861] Fixed a bug where conversion of a MSSQL Sawmill 8.1 profile to 8.5 fails with an error like, "Unable to Execute ODBC Query='create table main_table_plus_dfc select loadorder, db_filters_computed, ... from main_table'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near the keyword 'select'.;"
  • [1265145] A "remove database data" operation on a MySQL 5.5 database, gives an error like, "SQL query failed: 'delete from main_table using main_table x where (x.date_time < '2012-03-22 02:15:01')' error=Unknown table 'main_table' in MULTI DELETE"
  • [1265527] Reports can fail to generate, with an error, "Unknown variable 'lang_stats.general.' in expression."
  • [1266068] Reports sorted by "string" columns can crash in some circumstances.
  • [1266187] CSV export from the Scheduler, when "all rows" is specified, exports only one row.
  • [1266371] When using multiple SFTP log sources, database updates can skip an entire log source, if some data in the previous log source has already been imported.
  • [1266544] When viewing an unfiltered Log Detail report, paging forward changes the line numbers, but not the data displayed.
  • [1266546] In FTP or SFTP log sources, if the pathname ends with a /, no files will be selected.
  • [1267210] When an unqualified hostname is used in a URL to access the Sawmill web interface, the error message displayed states that there was error while trying to display an error. There should be an error; but it should state that hostnames must be fully qualified.
  • [1267261] The new report field option "Skip Escaping" is on by default, which can cause an error, "Syntax error: Unknown operator in expression," while displaying reports containing literal $s and other special characters.
  • [1267732] Attaching the "Gateway Reports" from the Snapons page gives an error, "Unknown variable 'lang_admin.snapons.gateway_reports.parameters.have_client_ip_field.form_element_label' in expression"
  • [1267856] Building the database of a profile using Nortel ACD format causes an error, "#### Internal: Attempt to find main table column number from database field 19 [average_tsf], but there is no such column in main_table"
  • [1268317] Per-user report filters have no effect.
  • [1268763] If an 8.1 profile has a "unique" database field pointing to a database field which doesn't exist, the converted 8.5 profile will give an error like, "Unknown database field 'cs_cookie' as source field of database field 'visitors'".
  • [1268972] If all log sources are disabled, a database build can crash.
  • [1269054] Large MySQL database builds, and other operations, can use large amounts of memory (more than permitted by the Preferences).
  • [1270484] When using an external SQL database, if all xref and indices are turned off for a profile, the main table will not be populated (or only 1000 lines of it).
  • New features:
  • [321927] Added support for Kerio Connect 7 logs. Previous versions supported only Keri Mail Server through versions 6.5. The new version also reports separately on SMTP, HTTP, and WebDAV events, reports usernames when available, and reports SSL status.
  • [1261075] Added rotation of the tagging server log file, and moved it to a subdirectory LogAnalysisInfo/logs/tagging.
  • [1261253] Added support for conditions in snapon parameters, so when attaching a snapon, certain parameters can appear or disappear depending on the values of other parameters. Used this to implement a better Aggregating Field snapon, which prompts for the source field when the aggregation operator is "unique", or prompts for a log filter or database filter expression otherwise.
  • [1262435] Added an "index granularity" option for database fields, which specifies the precision of the indexing of database fields in the main table. Previous versions had a default of 0, resulting in highly precise indices; the new default is 1000, which provides only general regional indexing of the table. With the new value, indices are 10x smaller or more, and index builds are as much as 5x faster. However, filtered reports can be as much as 50% slower, if a cross-reference table is not available to provide the report.
  • [1263556] Changed internal database deletion so it deletes only the "main" subdirectory of the database directory, rather than the entire database directory. This prevents an occasional issue which could cause data loss if the database directory was set to an existing directory with other data (like the log source directory).
  • [1265556] Reduced database disk usage by about 50% in most cases, by using smaller integers when reasonable.
  • [1266416] Enhanced Wowza analysis to include new Media Usage reports for a simple top-level view of usage. Also added referrer tracing with search engine and search phrases analysis. Improved grouping of reports menu, and removed a few extraneous fields for better performance.
  • [1266466] Improved performance of command-line index building (-a rdi) by building all indices in a single pass through the main table (which is how it works during a database build already). This can make the index builds 10x faster or more, when rebuilding all indices from the command line.
  • [1268454] Added support for Cisco NetFlow logs, created with "nfdump -o long"
  • [1268470] Implemented Mail Server Reports snapon, with general reports appropriate to mail servers: Sender Domains, Recipient Domains, Senders, Recipients, Recipients by Sender. Added this snapon to the Postfix plug-in.
  • [1269005] Added support for Akamai HTTP Streaming Log Format.
  • [1269200] Added a built-in Salang function, dns_resolve_hostname_to_ip_address(), for resolving hostnames to IP addresses.

New in Sawmill 8.5.6.3 (Mar 24, 2012)

  • Bugs fixed:
  • [1250051] When parsing very large datasets (billions of lines) with multiple processors, an error can occur during log parsing, "Invalid format of PARSED response from parsing server."
  • [1256945] Sawmill 8.1 profiles which include "session entrances" or "session exits" as report element columns do not convert properly; the resulting profile will give an error like, '"The report field "ssession_entrances" does not exist in columns of report element "year"' in the Report Editor.
  • [1257518] On Windows, if a LogAnalysisInfo has been moved using LogAnalysisInfoDirLoc, the upgrade installer installs the latest version in the default location, rather than in the version specified by LogAnalysisInfoDirLoc.
  • [1257840] If there is only one row in the database (typically, because there is only one line in the log data), viewing reports after a rebuild will give an error, "#### Unable to read file LogAnalysisInfo/Databases/apacheextended/main/Tables/xref0/header.cfg (Operation timed out)"
  • [1258384] When mapping a drive or share through the Network Disks section of the File Browser, the disk does not immediately appear after being mapped.
  • [1258619] After about 2 billion lines processed, the progress display shows the number of line processed as a negative number
  • [1258808] If a report field is added by a snapon at profile creation time, and its display format is not "integer," it is overridden by profile creation to be an "integer" display field.
  • [1258809] On Windows, if a custom report field displays as "integer," and contains a value larger than about 2 billion, it will display as about 2 billion in HTML reports.
  • [1259019] If a syslog-required field has a database field called "duration," the Log Fields Editor shows a blank field label.
  • [1259034] After a Remove Database Data operation, the Calendar report still shows links for days that have been removed.
  • [1259120] The "concurrent connections" snapon (used to do concurrent connection analysis for several media server plug-ins, including Microsoft Media Server, can overcount concurrent connections in some cases, as it fails to register the end of certain connections. These errors tend to accumulate as more lines are processed, so the number of concurrent connections will incorrectly slow a gradual upward slope.
  • [1259396] Fixed a bug where, if using a SQL table prefix or suffix, a Date Range filter would fail with an error like, "SQL query failed: 'select * from bottomleveldateitemnum' error=Table '.bottomleveldateitemnum' doesn't exist"
  • [1260066] If the "Log field separator" is a tab, the Config -> Log Processing -> Format page will show an error if you make a change to it, "No value. Please define a value."
  • [1260232] Sending mail to a qpsmtpd server, using username/password authentication, gives an error, "503 AUTH not defined for HELO."
  • [1260425] When a version 8.1 profile has a "session begin" or "session end" column in a report, and it is converted to 8.5 format, the Report Editor gives an error like, 'The report field "ssession_begin" does not exist in columns of report element "year".'
  • [1260484] Viewing Log Detail with a filter which discards all events gives an error like, "Attempt to read beyond end of LogAnalysisInfo/Databases/{profile}/main/Tables/rep_9a7ada7d8bbfe1830f4751a689e58504/data.tbl (fileSize=0); attempted to read from 0 to 280"
  • [1260610] Fixed a bug where the Log4j parser did not properly handle %d dates without a curly-bracket format, in PatternLayout (and rejected all entries).
  • [1261320] When using an Oracle database, the Log Detail report gives an error like, "Unable to Execute ODBC Query='select x.broken_link, x.date_time, x.day_of_week, x.hour_of_day, x.s_ip, x.cs_method, x.cs_uri_stem, x.cs_uri_stem, x.file_type, x.screen_dimensions, x.screen_depth, x.worm, x.s_port, x.cs_username, x.c_ip, x.domain_description, x.location, x.organization, x.domain, x.isp, x.web_browser, x.operating_system, x.spider, x.sc_status, x.sc_substatus, x.sc_win32_status, x.hits, x.page_views, x.time_taken, x.time_taken, x.session_entrances, x.session_duration, x.bounces from main_table xrownum between 1 AND 50'; diagnostics=ODBC error: rec1: SQLstate: S1000; msg=[Oracle][ODBC][Ora]ORA-00933: SQL command not properly ended ;"
  • [1261498] Flash Media Server profiles do not report concurrent connections.
  • [1262050] When using an FTP or SFTP log source on Windows, if the log file or pathname contains a colon (:), it gives an error like, "Unable to create folder LogAnalysisInfo\TempLogs\1330129826\7768\directory\\subdir:with:colon"
  • [1262299] When using "-a pv" to display database fields summaries from the command line, max/min fields based on date_time display values from 1970.
  • [1262381] Temporary tables with names like xref0_update are not properly delete when the database is built, making the database about 25% larger than it needs to be.
  • [1262595] If a relative date filter is used which selects just one day, on a Days report with a graph, it will give an error, "#### Unknown variable 'lang_stats.months_short.e_t' in expression". If it is used on a Months report with a graph, it will give an error, "#### Couldn't find node 0 in volatile.temp_month"
  • [1262794] Enhanced (and in some cases fixed) memory management, to better limit memory usage. The previous version generally kept its memory usage under control (under the specified limit), but certain types of memory usage were not counted toward the limit, which could result in substantially higher memory usage than allowed.
  • [1262847] The Create Profile Wizard does not prompt for the Access Log Valve pattern, when creating a profile using the format, "Tomcat (using Access Log Valve pattern)"
  • [1263906] If LogAnalysisInfo is relocated using LogAnalysisInfoDirLoc on Windows, the service will not properly shut down Sawmill.exe processes when it stops.
  • [1264170] Create a new report element "Hour of day" with "Display: Graphs". On tab Graph Options change the graph type to "Line graphs". On tab Graphs change sort by to "Hour of day", keep sort direction "Ascending". Click save changes and view the report. The report will show the graph with sort_direction "descending" although the report element editor indicates "ascending".
  • [1264241] When using MySQL or MSSQL as the back-end database server, if the database already exists and is already a Sawmill database, and if the profile uses the sessions snapon, an error will occur on profile creation, "Duplicate column name 'session_id'".
  • New features:
  • [1102604] Added support for arbitrary (almost) Log4J parsing, through support for most PatternLayout values.
  • [1256999] Improved reporting performance has been improved for large reports with "omit parenthesized items" turned on (an example of this is a standard web server "search engines" report). In one example (200 million line dataset), reporting performance increased 24x.
  • [1258492] Added support for Clavister SG log format.
  • [1258816] Added support for a variant of GroupWise Post Office Agent Log Format, which logs Net Id.
  • [1259024] Added a new snapon, "Top Level Domain" which creates a "top level domain" field, and populates it with a log filter, using the list of known top- and -second-level domains to convert a URL to a reasonable top domain name, e.g., "abc.xyz.com" becomes "xyz.com" and "abc.xyz.co.de" becomes "xyz.co.de".
  • [1259026] Added a new snapon, "Gateway Reports" which creates a category of four simple reports for gateway decides, for HR purposes: Users Summary, Categories Summary, Domains summary (using the new Domains field snapon), and Usage Detail. The Summary reports include pie charts, and the Usage Detail reports shows category, user, site, start and end time, and duration. This this snapon is attached by default to Palo Alto Integrated, and Squid (without category); other formats will follow.
  • [1259220] Greatly improved performance of report generation, for reports containing fields using custom expressions, including "average" fields (e.g., page views per session). Reports using this kind of field, and containing millions of rows, generate as much as 100x faster now. (Reports with few rows are not much affected).
  • [1259630] Improved performance of database builds of the internal database, especially multiprocessor builds and profiles without database filters (e.g., profiles without session analysis, or other snapon functionality which creates a database filter). Performance improvements vary by profile, but may be 40%-100% faster than previously, on a multiprocessor system.
  • [1259704] Added back support for the rebuild_cross_reference_tables action, which allows all xrefs, or any single xref (with -crt N) to be rebuilt from the command line, without rebuilding all the rest of database filters, or indices.
  • [1259821] Added support for numerical reporting of the content_bytes (%B) field, in Apache Custom format strings.
  • [1259842] Improved cleanup of Sawmill's "recycling bin" (LogAnalysisInfo\TemporaryFiles\DeleteMe), by immediately deleting everything put in it (simultaneously with whatever else is going on), rather than waiting for the next cleanup cycle. This can make a huge different in the amount of disk space temporarily used during certain operations, especially 8.1 profile conversion.
  • [1260326] Add a new clean_up_database action (e.g. sawmill -p {profile} -a cud), which drops all temporary tables from the database (except those whose parent process is still running). This also now occurs automatically at the beginning of any database update, or "remove database data." This eliminates the clutter which sometimes results when reports or other processes terminate abnormally, and fail to clean up their temporary tables.
  • [1260480] Added functionality to include multiple log filter initializations, and multiple log filter finalizations, in a single profile (as subnodes of log.filter_initialization and log.filter_finalization). They are run in order. Old-style initializations and finalizations (expressions directly on log.filter_initialization, etc.) are still supported. Added a new snapon operation to add a filter initialization or finalization to a profile. Together, these features allow snapons to add independent filter initializations and finalizations to a profile.
  • [1260482] Implemented the "Advanced Example: Rejecting spiders based on JS and /robots.txt access" log filter example as a snapon, for much easier implementation in a profile.
  • [1261082] Added a new action, rebuild_database_filters, which rebuilds all database filters.
  • [1262382] Enhanced Wowza analysis to use the new concurrency snapon to track concurrent streams, instead of the older "session" style analysis.
  • [1264537] Added support for Microsoft Forefront log format.

New in Sawmill 8.5.5.1 (Jan 18, 2012)

  • 948006] Removing data from an Oracle database using a filter gives an error like, "#### Unable to Execute ODBC Query='delete from main_table x where not (filtertmp_3932_0.itemnum IS NULL)'; diagnostics=ODBC error: rec1: SQLstate: S0022; msg=[Oracle][ODBC][Ora]ORA-00904: "FILTERTMP_3932_0"."ITEMNUM": invalid identifier"
  • [1244697] Duration fields are incorrect for Wowza Media Server profiles created with Sawmill 8.5.3 or 8.5.4.
  • [1246049] Reports, especially for a database which hasn't been rebuilt for a long time, can fail with an error like, "Attempt to read beyond end of LogAnalysisInfo/Databases/{profile}/main/Tables/_select_result_75103_1/sets/visitors/header.dat (fileSize=0); attempted to read from 0 to 64."
  • [1256046] When using a MS SQL database with Palo Alto log data, the Sessions Overview gives an error like, "select count(distinct x.user), sum(x.page_views), max(x.date_time), min(x.date_time), count(distinct x.session_id), sum(x.session_duration) from main_table x where 1=1'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near the keyword 'user"
  • [1256170] The "Maximum caching buffer memory full load" field is empty, in the Config UI.
  • [1256189] If there is a database field which has a different source than itself (e.g., a unique field like "visitors" which counts another field), database export will fail with an error like, "Can't find tableAlias=, fieldName=visitors in table main_table."
  • [1256296] Non-root-admin users cannot change their password; the "Save Changes" button has no effect.
  • [1256368] A license installation with sublicense only allows the number of profiles specified by the main license.
  • [1256450] Report fields which compute an average value of a field (x/y), show 0.
  • [1256501] When using a MS SQL database with a 64-bit integer field, aggregated numbers larger than about 2 billion can cause an overflow error which terminates the build.
  • [1256506] The delete_database_field action does not delete columns from report table (and report tables themselves) which are derived from the field, e.g., the "city" column and table are not deleted when the "location" field is deleted.
  • [1256517] An unfiltered Log Detail report takes a long time to generate (scans the whole main table, instead of just the visible rows).
  • [1256528] Some temporary tables are not removed from internal databases, especially during filtered report generation, resulting in a large number of unnecessary database tables (and unnecessary files on disk) after a long period with many reports generated.
  • [1256952] Profiles created from the Juniper Networks Secure Access 4000/6000 plug-in give an error when reports are displayed: 'Unknown configuration group "session_paths" in node "profiles.in.statistics.reports"'
  • [1257001] Fixed an issue with the performance of some queries, which made reports slow with large datasets.
  • [1257002] Fixed a bug which could cause a crash on database build, if there were more cross-reference groups than database fields.
  • [1257516] If a Sawmill 8.1 profile has a custom "session user" field (session visitor ID), and that field has no corresponding report field, the profile when converted to 8.5 will fail to generate the Individual Sessions report with the error, 'The report field {customfield} does not exist in columns of report element "individual_sessions".'
  • [1257752] PIX logs with high hit-cnt values can take a very long time to process, or even fail with an error like, "Unexpected response from SPS server: PARSED 43969360 289230 0 0"
  • New features:
  • [916304] Added support for Amazon Cloudfront Streaming logs, including a database filter which emulates Amazon's own "bytes transferred" calculation.
  • [1052024] Enhanced the Coradiant TrueSight log format support, to handle variable field lists in the header, to report on all known numerical fields, and to categorize all known non-numerical field reports.
  • [1218318] Added a new "Execute command line" action to the Scheduler, to run an arbitrary command line.
  • [1256677] Added a performance warning to the Progress display, when a particularly complex normalized database field is detected during database building.
  • [1257003] Added an xref for each report, by default (with no date range info). This can significantly improve the performance of unfiltered top-level reports, and especially the Single-Page Summary.
  • [1257026] Improved the selection of the xref table used for a query, so the smallest (fastest) match is chosen, rather than the first match. This improves the performance of some unfiltered, or date-filtered, reports.

New in Sawmill 8.5.4.1 (Dec 14, 2011)

  • [1243631] Generating a Sessions Overview for a dataset with no session events gives an error, "Internal error: mapping 'LogAnalysisInfo/Databases/{profile}/main/Tables/session_users_stage1/sets/sessions' read-only, but its lists.dat (LogAnalysisInfo/Databases/{profile}/main/Tables/session_users_stage1/sets/sessions/lists.dat) does not exist"
  • [1246830] Converting a v8 profile whose name contains a dot, gives an error, 'Unknown configuration group "options" in node "profiles.{profilename}.database"'
  • [1247484] If no -er (ending_row) is specified for a get_report action, it defaults to 0, which shows just one row (it should really show ten).
  • [1252329] On Window, if an index (or certain other database files) exceeds 2GB, it can grow without bound, causing the disk to fill and the database build to fail. This typically happens with large datasets, of 200 million lines or more, but it can happen with certain smaller ones.
  • [1256100] For certain (uncommon) large datasets, on Windows, a database build can crash during database filtering.
  • [1256165] Filtering reports, especially on single days, sometimes gives an error like, "Attempt to read beyond end of LogAnalysisInfo\Databases\{profile}\main\Tables\filtertmp_5676_1\data.tbl (fileSize=4); attempted to read from 882531391810568340 to 882531391810568344."

New in Sawmill 8.5.3.1 (Nov 22, 2011)

  • Bugs fixed in version 8.5.3:
  • [1189781] Bug reports always go to [email protected], even if Support Email is set to something different in lang_stats or Preferences
  • [1192538] When using real-time reporting with multiprocessor parsing and the internal database, the reports do not show the latest data.
  • [1201875] Setting the thousands divider, or the decimal divider, in the user's settings, as no effect in reports.
  • [1217367] Adding a new license to an already licensed installation gives an error, "Unknown operator in expression."
  • [1218812] Filtered reports which cannot use xrefs, do not take advantage of database indices, resulting in report generation time linear with the size of the database. This makes some reports, especially those filtered on an item with few associated events, much slower than they could be. With this enhancement/fix, these reports are now 30x-50x faster for a 10 million row dataset, and will be proportionally even faster (or rather, were proportionally slower), for larger datasets.
  • New features in 8.5.3:
  • [1191655] Support has been added for the Mikrotik Web Proxy log format.

New in Sawmill 8.5.1.1 (Sep 7, 2011)

  • Bugs fixed:
  • [1007266] Various counts in the Postfix Mail Server or Brightmail Gateway plug-in, such as "Messages blocked" were being counted for each recipient, inflating the count.
  • [1115018] The Sessions snapon is always installed with a timeout of 1800, even when the log format plug-in has a different timeout. Also, the Sessions snapon does not recognize "logout" events.
  • [1117001] When using MS SQL as the back-end database, and using bulk_insert as the insert mode, and using a wildcard report filter, report generation gives an error like, "BULK INSERT filtertmp_5852_0 FROM 'C:\LoadDataDirectory\results.aec.gov.au.2011.to.2012\sql_load_data_5852_seq0.tsv"
  • [1134950] On Mac OS, the Sawmill logo and and Use Sawmill window graphic are out of date.
  • [1136138] Network action accesses give an error, "Syntax error: Unknown variable 'action_fail' in expression"
  • [1137776] The "upgrade these 8.1 profiles to 8.5" text does not appear, when using Internet Explorer 6.
  • [1139624] A bug report submission can fail on 32-bit systems, given an error like, "Unable to allocate 4294958111 bytes of memory; maximum memory is 939524096, but 3654011 is already used, and no further memory can be freed."
  • [1144553] If a profile has session information, and the database directory is non-default, profile conversion will fail with an error, "Can't find tableAlias=, fieldName=ssession_id in table main_table," or "Unknown database field "ssessions" in cross-reference group."
  • [1149031] Fixed a bug where conversion of an 8.1 database to 8.5 would fail silently if the profile had no session information, eventually causing an error like, "Unable to read file LogAnalysisInfo/Databases//main/Tables/xref0/header.cfg."
  • [1156064] The get_report action does not include the main (non-aggregating) column in the XML result.
  • [1164125] In Sawmill Lite click on Log Source, Database Info, Update Database or Build Database in Config. Sawmill shows the "No permission to view this page ... " alert.
  • [1164128] In IE6 try to open the Config Option Database Info, Update Database, Build Database, Log Parsing Filters or Database Filters. They show a blank page indicating a javascript error.
  • [1164135] In IE6 go to Config Report Options and click the Config Options link. The navigation bar shows form elements (list boxes) from the underlying form.
  • [1164140] In IE6 open Config Log Parsing Filters or Database Filters. The textarea element aligns to the right.
  • [1164142] In IE6 or IE8 open Config Log Filters and click on individual log filters, then view a different page. Sawmill shows an alert about unsaved log filters changes although no changes have been made.
  • [1165249] Fixed a bug which could cause sporadic crashes of the subordinate web server process (resulting in images and other support files failing to load in the web interface).
  • [1165798] Go to Admin/User Settings and enter a new password, mistype the second password and click Save Changes. The changes are not saved and no error message is shown about the mistyped password.
  • New features:
  • [1135664] Added a Subjects report and parsing of the email subject from lines with Subject: and subject= to the Postfix Mail Server plug-in.

New in Sawmill 8.5.0.1 (Jul 29, 2011)

  • [1084636] Asking for password to be emailed gives an error, "Invalid sender in send_email()."
  • [1093134] Database builds can crash if some xrefs are disabled.
  • [1094316] The "+" icons in the Session Paths report have no effect.
  • [1097200] The Pages/Directories report isn't hierarchically zoomable.
  • [1101001] Queries on real-time databases give an error, "Unable to read file LogAnalysisInfo/Databases/wri/main/Tables/xref0/header.cfg (Operation timed out)."
  • [1113948] A Flash Media Server profile created with x-duration unchecked in the New Profile Wizard, gives an error on build, "Syntax error: Unknown variable 'x_duration' in expression"
  • [1114148] Updating a database for a Flash Media Server profile, or any other profile with a "node" type database filter variable, may give an error like, "Couldn't find node 4616027 in untitleds."
  • [1114349] Network action access to get_report gives an error, "Couldn't find node header in untitleds."
  • [1114357] The Admin menu appears obscured by the Config menu, when the Config menu is pulled down.
  • [1123802] The Individual Sessions report contains enormous bogus session IDs in some rows.
  • [1129556] Viewing reports after attaching a "Particular File Access" snapon gives an error like, 'the database field node "accesses_on__robots_txt" refers to the log field node "" but this log field does not exist.'
  • [1130487] Clicking "Customize Report In Config" doesn't open the report--it opens the Report Editor, but not the report itself.

New in Sawmill 8.5 Beta 5 (Jun 25, 2011)

  • [952934] Progress display shows "Generating report" during snapon attachment.
  • [1064630] Attaching the "Particular file access" snapon gives an unexpanded variable value "$lang_admin.snapons.particular_file_access.parameters.pathname.parameter_value" as the default value in the "Pathname of file" field.
  • [1069711] Attaching double_hits snapon gives the error: "Expected ADD or DROP in ALTER TABLE query, found 'double_hits'"
  • [1069746] Progress displays on Windows during database filtering contain odd characters.
  • [1070815] Parsing of Flash Media Server logs (or any other profile with many "float" type aggregating fields) on multiprocessor Windows systems is about 5x slower than it ought to be.
  • [1074140] When attempting to attach the "Unique Values" snapon to a Flash profile, an error occurs, "Unknown variable 'lang_admin.snapons.unique_values.unique_field_name.ratio_field.parameter_value' in expression."
  • [1075902] When creating a profile with a Shoutcast w3c log, after clicking "Finish" this error is displayed: "Snapon attempted to add database field 'session_id', which already exists"
  • [1075940] Byte numbers are wrong (much too high) for Flash profiles.

New in Sawmill 8.5 Beta 4 (May 11, 2011)

  • [991021] Viewing reports of a v8 profile converted to new format, gives, "Can't find tableAlias=, fieldName= in table xref0."
  • [1058009] Visitors, and other "unique" fields, show zeros in some reports.
  • [1062806] The Overview shows 0 for visitors, or other unique fields, for some (larger?) datasets.
  • [1063606] Attempting to attach the "Particular File Access" snapon immediately gives an error, "Unknown variable 'lang_admin.snapons.particular_file_access.parameters.page_field.label' in expression".

New in Sawmill 8.5 Beta 3 (Apr 29, 2011)

  • Bugs fixed:
  • Building the database for a PIX profile gives an error, 'Unknown index colum "user" in main_table.'
  • For larger datasets, unique numbers (e.g., visitors) may be too low, or zero, in the Overview.
  • he Log Processing page in Config is empty, for some profiles.
  • New features:
  • Language, Thousands divider, and Decimal divider, are all now per-user options, and editable in Admin -> Settings.
  • Enhanced the Create Profile Wizard to allow plug-ins to prompt for snapon parameters at profile creation time.

New in Sawmill 8.1.8.1 (Jan 18, 2011)

  • Bugs fixed:
  • [946520] Selecting Apache Custom in the log formats list gives a blank page, instead of prompting for the format string.
  • [947994] Expiring (removing) data from an profile with a back-end Oracle database gives an error, "SQL command not properly ended"
  • [955556] Indices can become corrupt on database update, causing the database to be much larger than it needs to be, and the update to take much longer than it should.
  • [960979] When expiring all rows of a database, an error can occur, "Internal: Attempt to write header for read-only table unique_loadorder."
  • [963660] Duplicating the first action in the scheduler inserts it first, but the display shows it second.
  • [966350] In Config -> More Options -> Miscellaneous, the name of the DNS Lookup appears initially as "Support & Action Email."
  • [966655] Relative date filters like "yesterday" offset the time zone in the wrong direction on Windows, sometimes resulting in the wrong day being displayed.
  • [966926] Some PDF reports give an error when displayed with Windows Acrobat Reader, "An error exists on this page."
  • [975496] Multiple copies of the same item, differing only by case (e.g., /Dir1 and /dir1) may appear in tables, even when the field is case insensitive.
  • [975637] The "Distributed processing/Parsing server distribution method" options appear in Lite and Professional modes, though they have no effect except in Enterprise.
  • [984449] If a profile is created by a particular user, then than user is deleted, and later a new user with the same name is created, the profile is shown to have been created by the new user.
  • New features:
  • [957993] Support for SHOUTcast log format versions 1.6, 1.8 and 1.9 are now in a single plug-in. The name is "SHOUTcast Media Server / DNAS (Distributed Network Audio Server)". A Players report has been added to this plug-in. The W3C version has also changed. The Web Browsers report is not called Players, and the Spiders report is gone from the Vistor System report group. The plug-in name is now "SHOUTcast Media Server / DNAS (Distributed Network Audio Server) (W3C)".

New in Sawmill 8.1.7.3 (Oct 12, 2010)

  • Bugs fixed:
  • [871194] Documentation does not display in the user's selected language--it displays in the Preference language, regardless of user settings.
  • [889872] Fixed a bug which, when MP query splitting was turned on, could cause an error like, "Lock disappeared on Task-85306-Lock, but SubqueryDone_85015_3 still does not exist; subquery process must have crashed!"
  • [892452] Profiles which handle their own header parsing using filter_preprocessor, re-add data on a database update, if one of the log files is nothing but a header.
  • [896386] An XSS vulnerability exists, potentially allowing an attacker to execute arbitrary JavaScript code on another user's system.
  • [913690] Using a date filter on the Single-Page Summary gives an error like, 'Unknown configuration group "date_filter_info" in node "sessions_cache.1b41dc18092a36480b171fb9508386ec.profiles.access.report_ jobs.69931523589d08bcfe8b7dc9990d403c.report_elements.0" (030FADDC)'
  • [914013] The profiles list in the User editor is not sorted.
  • [923130] Profiles including a user-agent field can crash during database build (but usually don't).
  • [927348] If there is an extra closing parenthesis in a log filter, at the top level, the remainder of the log filter will be quietly ignored.
  • [927438] Viewing reports of Nortel ACD gives an error: 'The database field node "average_tsf" refers to the log field node "average_tsf" but this log field does not exist. It is recommended to manually correct the "average_tsf" database field node.'
  • [933940] Turning on DNS lookup in a profile created with 8.1.6 gives an error on build, "Couldn't find node hash_table_expansion_factor in profiles..database.tuning"
  • [934837] Command-line database updates do not correctly detect a running database build, and proceed with the update, potentially corrupting the database.
  • [936430] Running Sawmill in CGI mode, with the directory containing Sawmill not writable by Sawmill, gives an error, "Can't open lock file Lock (Permission denied)."
  • New features:
  • [922988] Added detection of iPad/iPod in user-agent string parsing

New in Sawmill 8.1.6.3 (Sep 8, 2010)

  • [841842] Creating and building the database of a profile with a real-time log source on Windows sometimes results in an error, "Unable to read file LogAnalysisInfo\Databases\sawmill_realtime\main\Tables\bottomleveldatebo ttomlevelitem\header.cfg (Broken link)"
  • [851291] Creating a profile with Microsoft Media Server log data, and unchecking "session events" gives an error, "Syntax error: Unknown variable 'session_events' in expression" on database build.
  • [851829] Added local time zone support for the date filter string option; defaulted to local time zone for UI date filters.
  • [860107] When using the "auto" date format, years past 2030 are considered corrupt (which is a problem for Thai log data, since Thai years are around 2553).
  • [862673] This was an escaping issue with control characters for all filter expressions added in Config (per profile, per report or per report element filter expression) or entered via the command line.
  • [868898] The Salang function current_log_pathname() returns the wrong pathname, for the first file of a multi-file log source, when using multiprocessor log processing.
  • [869075] Field values containing ASCII code 26 (control-Z) cause an error on report generation like "Unterminated quote in LogAnalysisInfo\profiles_cache\\raw_report_elements\39f3bc8e94a923143b0ef079f5dc4805.cfg at line 1282."
  • [870672] After importing a profile from v7 to v8, the Database Fields page of Config may show "Select log field" as the log field, or may not display at all.
  • [877311] Using a "session start" filter on the Individual Sessions report, causes an error like, "Invalid integer size 63 in LocalFileTable::GetIntCell()."
  • [880490] Building Sawmill from the encrypted source code, on Fedora Core 13, gives an error, "error: invalid conversion from ‘const SSL_METHOD*’ to ‘SSL_METHOD*"
  • [883332] Build a database with Oracle back-end, with a xref table with four or more non-aggregating fields, gives an error when indexing the xref table similar to: "create index x19_3xbx19_13xbx19_17xbx19_18xb on x19l0_0_0_0ux19 (bottomleveldate, cs_username, sc_filter_result, sc_filter_category)'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Oracle][ODBC][Ora]ORA-00972: identifier is too long"
  • [884348] When using an Oracle database with the Oracle client driver, if a string field has a value longer than 200 characters, but shorter than 255 characters, it will generate an error "value too long."
  • [885674] When using the internal database, main table indices can become larger and larger (more than expected) as database updates occur.
  • [886062] Building a database with an external SQL database back-end, and with a SQL prefix, results in indices with names do not include the prefix, potentially causing index name collisions on Oracle, if using multiple databases in the same server.
  • [887931] If a database field name is too long, and an external SQL database is being used, report generation will fail with an error like, "Unable to Execute ODBC Query='select x.bottomleveldate, count(x.bottomleveldate) from zxref0 x inner join zbottomleveldatebottomlevelitem b on x.bottomleveldate = b.bottomlevelitem group by x.bottomleveldate'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Oracle][ODBC][Ora]ORA-00972: identifier is too long." This is more common on Oracle, where any hierarchical database field longer than 15 characters will cause this error.
  • [888885] Many plug-ins, including Tomcat, show duplicate Day of Week and Hour of Day reports, both in the "Date and time" group and at the top level of the report menu.
  • [890225] The built-in function convert_charset() deallocates memory improperly, resulting in crashes or incorrect results in some cases where it is used from a custom log filter.
  • [891575] The default "web browser" is reported as "Netscape Navigator" for unknown "Mozilla" user agents (changed it to "Unknown Mozilla").
  • [894120] When a report is filtered with a filter which excludes all rows, it generates an error like, "Unable to read 304 bytes from LogAnalysisInfo/Databases//main/Tables/_select_result_21095_0/data.tbl.moved; got only 0 bytes (No such file or directory)"
  • [894684] When using a SQL table prefix with non-internal database, database export fails with an error like "Table 'profile.main_table' doesn't exist"
  • [896378] A non-administrator can access Setup Wizard with a specially formatted URL.
  • [896380] A non-administrator can gain access to user information with a specially-formatted URL.
  • [896381] Non-administrators can create/delete user accounts with a specially formatted URL.
  • [896383] By changing local JavaScript variables, a malicious non-administrator can gain access to some sections of the user interface restricted to administrators.
  • [896387] A non-administrator can change the administrator password using a specially crafted URL.
  • [906782] Import of exported database into an Oracle database gives an error like, "#### Unable to prepare ODBC query: ODBC error: rec1: SQLstate: S0002; msg=[Oracle][ODBC][Ora]ORA-00942: table or view does not exist."
  • [911657] On Oracle databases using Unicode charset, database builds with fields with long string values can give an error like, 'ORA-12899: value too large for column "SYSTEM"."FILE_TYPEITEMNUM"."FILE_TYPE" (actual: 378, maximum: 255).'

New in Sawmill 8.1.5.1 (Jun 4, 2010)

  • Bugs fixed:
  • [797770] Update a database after removing data can cause an error like, "Short read in SingleWindowCachingBuffer file LogAnalysisInfo/TemporaryFiles/Task24630/ris_main_table_x_loadorder_le_1014599626/data.tbl; tried to read 1 bytes from position 1014599626, but only got 0 bytes."
  • [798195] Creating a profile using the WarFTPalt plug-in causes a Sawmill alert, 'Unknown configuration group "destination_ip" in node "profiles.profilename.log.fields"'
  • [798479] In the Session Paths report the pages are not in chronological order.
  • [806410] Bug fixed where percentages were wrong when "Use overview for totals" was set for reports, and "Show total value" was set for columns of type Unique.
  • [808447] If Users/Roles (RBAC) has "view only" changes made to permissions, it now no longer displays a "confirm message" asking the user to save changes before leaving the page. The toolbar message is also improved to be "Your grants for this page are limited to view and changes cannot be saved."
  • [814810] When an incorrect password is entered for an SFTP log source, the error message is "unknown keyboard-interactive response: 1".
  • [816641] Fixed bug which could cause 0's in the Overview, after removing data, and updating with new data. This could also cause an error about reading past the end of a temporary file.
  • [818357] Using an FTP log source to certain versions of Pure-FTPd server can result in an error like, "Unexpected response from FTP server: 150 4.977 seconds (measured here), 0.50 Mbytes per second."
  • [823230] Bug fixed where Internet Explorer 8 shows skewed calendar months.
  • [839414] Filtering on a single second gives an error like "No date applied. The date filter '7apr1998_165306'" is invalid.'"
  • [842123] Export of a database from the command line (with -a ed) generates empty itemnum tables.
  • [843395] Command-line export (-a ed) of a MySQL database gives an error, "No database selected"
  • [844142] Building an Oracle database with some versions of the Oracle Client driver gives an error, "Internal Error: Attempt to get SQL_INTEGER value from a SQL_FLOAT"
  • New features:
  • [712684] Slightly enhanced the Admin menu so clicking Profiles reloads the profiles list, even when already displaying the profiles list.
  • [800148] Added support for Microsoft Exchange Server log data, logged through a syslog server.
  • [814809] Extended the command-line authentication protocol so the script can print *ROLE*:N to indicate that the user logging in is part of role N (N is the internal role node name).
  • [815421] Added sorting of the Log Detail report.
  • [816854] Added a "Automatically direct to reports upon login if user accesses only one profile" to the Users editor.
  • [821749] Added support for an additional UNIX-syslog style header (after the first syslog header) in ISC DHCP log format.

New in Sawmill 8.1.4 (Mar 31, 2010)

  • Bugs fixed:
  • [767223] When using multiple log sources, the progress displays count of bytes processed restarts at zero for each log source, instead of showing the total cumulative bytes processed.
  • [770870] The built-in Salang function convert_charset() allocates memory equal to the string to be converted, and does not deallocate it. This can result in high memory usage over long builds, if log filters use this function.
  • [785338] After doing a "remove database data" operation, the Overview hangs for a long time with no progress display.
  • [785650] The week_of_year field uses 0 for the first week of the year, instead of 1 as documented.
  • [790229] The "config" file of a database is not updated when xrefs are rebuilt using the "-a rcrt" command-line option.
  • [791207] When using the command-line authentication script, the script is called for every click, rather than just once per login session.
  • [793905] After creating a report using "Generic W3C Web Server" as the format, reports give an error, "Unknown variable 'volatile.new_profile_name' in expression"
  • [794485] On MacOS 10.6, the "run at startup" option does not start Sawmill successfully.
  • [795215] Multiprocessor builds of profiles using the global date regular expressions option, can incorrectly reject some log entries.
  • [796244] Fixed problem where files with Kiwi ISO syslog headers autodetect as "Kiwi (mm-dd-yyyy dates)" as well as "Kiwi (ISO/Sawmill)". The ISO files have the year first (yyyy-mm-dd), so selecting the wrong syslog completely prevents the logs from parsing. Files with the year last in the date are detected as both "Kiwi (mm-dd-yyyy dates)" and "Kiwi (mm-dd-yyyy dates)" so that the correct date can be selected. Now files with the year first are only detected as "Kiwi (ISO/Sawmill)".
  • [798584] The SHOUTcast W3C Log Format plug-in did not parse correctly after encountering a second W3C field header in the same file. This bug was introduced in Sawmill 8.1.3 in a change to the plug-in that worked around for a logging problem in SHOUTcast 1.9.8.
  • [798843] Log Detail gives an error, if the number of rows in the filter set is less than the number of rows displayed in the table, like: "Unable to read 392 bytes from LogAnalysisInfo\Databases\profile\main\Tables\_select_result_1344_ 0\data.tbl.moved; got only 0 bytes".
  • [799951] The built-in Salang function collect_listed_fields() does not work properly when the divider or separate parameters are not constants.
  • [804591] When using command line authentication, logging in as root administrator causes an error like: Unknown configuration group "user_grants" in node "sessions_cache.479a3ff1bcb1ea9ba79e4f2113018196.session_info" (02705354).
  • [806853] Profile patterns entered in the Scheduler were are treated as patterns unless they are preceded by "pattern:".
  • [806929] The Windows installer does not properly install the Microsoft 2008 redistributable packages, which are required for Sawmill to start on some older versions of Windows.
  • New features:
  • [686796] Support for a new Windows Event Log variant has been added. It is comma separated and has a m/d/yyyy date format. The fields are Level, Date, Time, Source, Event ID, Task Category and Message. The plug-in was tested using files from Vista logs with 24 hour times and Windows Server 2008 logs with AM/PM times.
  • [781872] Support has been added to the FortiGate Traffic Log Format plug-in for a format variation that is comma instead of space separated.
  • [799782] Added support for a format variant of the Astaro SMTP Proxy Log Format which has a single space after between the To email address and the "Reason". Other variants have a field there whose value is ignored. The result of not allowing this was rejection of most To lines (=> lines).

New in Sawmill 8.1.3.1 (Feb 15, 2010)

  • Bugs fixed:
  • [742760] When the language in the Preferences does not match the language for the current user, reports can contain a mix of languages.
  • [744621] Database build, using an Oracle database, with a database field whose internal name exceeds 30 characters, fails with an error "ORA-00972: identifier is too long"
  • [752760] If all lines of log data are rejected during log processing, the resulting database, which correctly shows zeros for all values, incorrectly shows "01/Jan/1970, 1 day (entire date range)" as the date range in the Reports page.
  • [752760] The date range at the top of Reports shows 1970, when there are no entries in the database.
  • [758276] When the Log Detail is filtered, then paged, the second page data matches that of the first page.
  • [759965] Building a database with more than 64 main table indices (e.g., a default profile with more than 64 non-aggregating field) gives an error: Too many keys specified; max 64 keys allowed.
  • [763994] Apache Custom format strings containing literal \" values do not parse properly.
  • [765835] The Concurrent Session number is computed from the unfiltered session data--report filters have no effect on it.
  • [767731] The "+" operator does not match properly in regular expression log sources when using Show Matching Files in Config->Log Sources.
  • [767843] The Windows installer does not install the VC++ 2008 redistributable libraries, which can cause errors when trying to start Sawmill, if the libraries are not already installed.
  • [772469] SFTP log sources give an error "Unknown SSH server prompt 'Password: ' -- only 'Password:' is supported" on certain SFTP servers (specifically, SUSE Linux Enterprise Server 10).
  • [775640] Building a database with Microsoft SQL Server, in a profile with a session analysis, where the name of the session page field is "file" or some other reserved SQL keyword, gives an error like, "Incorrect syntax near the keyword 'file'."
  • [776369] When password expiration is configured in Preferences, an error occurs when the password expires: "Couldn't find node login_plugins in".
  • [776937] The Session Paths report can cycle back on itself, making sessions infinitely deep, when using the web UI.
  • New features:
  • [704687] Merged changes from Sawmill 7 that were never replicated in Sawmill 8. This adds support for a new format variant and extracts some additional fields.
  • [717071] Added support for an all caps string instead of a hex number after the url in the Barracuda Spyware Firewall / Web Filter Log Format plug-in. This field is ignored in all cases because it's purpose is unknown.
  • [718244] Reporting of pass throughs (BP events) was added to the Cisco Wide Area Application Services (WAAS) TCP Proxy log format plug-in for WAAS 4.1.x. Counts of connection starts, pass throughs and two types of connection ends were added to the numeric fields.
  • [748047] Implemented support in the SHOUTcast W3C Log Format plug-in for a logging problem in SHOUTcast 1.9.8 which causes numeric values Play Duration, Server-to-client bytes and Average Bandwidth. The W3C log format specifies that if there is no value in a field, it must be replaced with a dash, but SHOUTcast logs sometimes have a completely empty cs-user-agent (player) field which caused the remaining fields to be shifted and the value of sc-bytes to end up in the x-duration field. The modified plug-in compensates for this problem. This is an important concern for Soundexchange RIAA reporting.

New in Sawmill 8.1.2.2 (Dec 10, 2009)

  • Bugs fixed:
  • [629538] When using MYSQL 5.1, database data removal fails with an error, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'as x using main_table as x where (x.date_time < 'YYYY-MM-DD HH:MM:SS')' at line 1"
  • [634747] Changed the way play duration is distinguished from non-play duration in the Wowza Media Server Pro plug-in. This fixes a bug in 8.1.1 where the stream duration is incorrect when there are pauses or seeks. Also restored full tracking of x_duration in the stream_duration field and added the play_duration and pause_duration numeric fields. Added check for no publishing start time because we it is possible to have no c_client_id values on unpublish events.
  • [676495] Reports show incorrect results, when zooming on multiple days in the Days report, and simultaneously zooming on multiple items in some other report.
  • [680285] Profile creation fails, when using Aventail Client Server log format, with an error, "Could not find main_report_field_name in log format report description 'connect_tunnel_sessions_overview'".
  • [686802] When using the Microsoft ODBC for Oracle driver, Sawmill gets an error on database build: Internal Error: Attempt to get SQL_DOUBLE value from a SQL_DECIMAL ODBC table column: 0. Note: with this bug fix, this error no longer occurs; however, other errors may occur because Sawmill requires an ODBC 3 driver, and the Microsoft ODBC Driver for Oracle does not support ODBC 3 (it supports only ODBC 2). Therefore, do not use this driver with Sawmill; instead use the Oracle Client driver (provided by Oracle), or another third-party ODBC driver for Oracle.
  • [686819] Building a database in a profile with session information, using an Oracle back-end database with the Oracle Client driver, gives an error, "Internal Error: currentBufferColumn=8 in UploadChunk() for sessions_update."
  • [686901] If two reports are run simultaneously against a real-time database, the second report can "collide" with the resuming database build, causing various errors.
  • [688118] Database xref tables, in the internal database, are much larger on disk that they need to be, for some datasets.
  • [691995] Internal database indices skip a row when being updated. Among other possible effects, this can cause one day of data to be missing from the reports, and the Date Picker.
  • [692073] The order of log sources in Config -> Log Source is not permanently saved when they are manually reordered.
  • [693811] The name of the plug-in the Generic W3C Log Format has changed to Generic W3C Web Server Log Format to indicate the basic kinds of fields it expects to find. The way fields are created has been changed overcome a bug where the creation of some derived fields with functions caused an error because the database fields did not exist.
  • [699372] A Microsoft SQL Server log source give the error "Invalid log source. Numeric valid out of range" when one of the selected columns is a BIGINT, and one of the values is larger than about 2 billion.
  • [703189] Use of "create_many_profiles" gives an error: "Syntax error: Unknown operator in expression" due to a syntax error at the keyword "string".
  • [707419] HTTPS does not work on Windows (it starts an HTTP server instead).
  • [707422] Database updates using an Oracle database server give an error, "ORA-00001: unique constraint (STAT.LOADORDER) violated"
  • [710643] Double-quoted W3C fields values which with with a literal double-quote (escaped as two double quotes), e.g. appearing literally in the log data as """"hello""" to indicate a value of a double-quoted value hello, are treated as an empty value.
  • [713843] Importing the database of a Sawmill 7 profile using MySQL, and changing the database name during import, gives an error, "Table '.locationitemnum' doesn't exist at ../src/run.cpp:2768"
  • [717967] Database updates can crash during the index build/merge step, for very large datasets.
  • [725411] The Overview report in the Single-page Summary sometimes shows incorrect average bandwidth values.
  • [737620] ISA CSV log data is very slow to process, due to a problem normalizing the highly unique filter_info field.
  • [741574] With IronPort C-Series logs, relayed SBRS events are reported as "rejected."
  • [743313] Filtered date/time reports can display graphs from previous cached filtered versions of the report, instead of the correct graph.
  • New features:
  • [573418] Added basic support for keyboard-interactive sshd authentication; this allows SFTP access to default FreeBSD systems, and others using keyboard-interactive with a "Password:" prompt.
  • [731045] Added support for Windows 7 detection in user-agent string.

New in Sawmill 8.1.1.1 (Oct 7, 2009)

  • Bugs fixed:
  • [659410] Creating a profile with Mirapoint SMTP Log Format gives an error, 'Unknown configuration group "date_time" in node "profiles..log.fields"'.
  • [662875] When using Professional trial mode, Save Report To Menu gives an error, "Checksum does not match for file 'templates/statistics/save_as_new_report/get_report_dat.cfv'. Your licensing does not permit editing of template files, but it looks like that file has been edited. Please upgrade to Enterprise licensing if you need to edit templates, or remove the edits from that file to continue with your current licensing."
  • [662959] Updating a database when the database has not yet been build gives an error, "Can't find tableAlias=main_table, fieldName=bottomleveldate in table main_table_join_sessions_join".
  • [665590] Use of an ODBC log source on a Microsoft SQL Server with a BIGINT field, results in an error, "Invalid log source Unknown ODBC type; 5"
  • [665606] Cross-reference builds done with the internal database, with large datasets, sometimes give an error like, "Internal: list 99 in IntegerLists .... ends with -3412465, which is the start of a range"
  • [671050] In some circumstances, dates which have data are not clickable in the Date Picker.
  • [671133] If a scheduled task is running when a second scheduled task is scheduled to run, the second task does not start, in some cases.
  • [671684] A session contains filter, using wildcards, on a page which does not exist, gives an error, "Internal Error: Empty node name."
  • [671949] The global_date_filename_regular_expression has no effect in multiprocessor database builds.
  • [679482] Fixed a bug which could cause an error on database build or update, similar to "#### Internal: listitem=338931424, which is larger than the array cache size (308152)."
  • [680155] Database builds may crash at the end of log reading, if DNS lookup is turned on, and parsing servers are being used for multiprocessor building.
  • [680181] Numeric comparisons in report filters give incorrect results.
  • [680671] Running a multiprocessor "split" build (using Sawmill to split data into multiple per-profile log datasets) can result in an error "mismatched brackets" in the info.cfg file.
  • [681790] Removing data from a database without session information gives an error, "Can't delete file LogAnalysisInfo/Databases//main/session_table_info.cfg".
  • New features:
  • [606985] Added a plug-in to report on OpenFire IM logs. These are XML logs and the lines can be extremely long. Testing was done by introducing newlines between closing and opening packet tags.
  • [661926] Added new support for AIX CPU Utilization log format.
  • [667394] Added support for Communigate Pro 5.2 to the Communigate Pro Log Format plug-in. Improved tracking of multiple recipients. Eliminated counting a message twice when ACCOUNT/delivered and DEQUEUER/LOCAL/delivered are both in the log. Added basic support for DEQUEUER/LIST/relayed. Eliminated underused operation field and added an action field just for delivery type. Added Queue ID field.

New in Sawmill 8.0.7.1 (Apr 17, 2009)

  • Bugs fixed:
  • [520725] Common Access Log profiles with large datasets use a huge amount of memory on database builds.
  • [528835] The parser adds "{default}" to the end of any log field value which ends in the hierarchy divider specified by its log field, even if the field is not hierarchical. I.e., {default} or "(default page)" sometimes appears unexpectedly in reports of non-hierarchical, non-page fields.
  • [529667] Some strings in the web interface are in English, even when non-English translations are used.
  • [533349] Ironport C Series reports do not show recipients when message rewriting occurs in the log data.
  • [534986] Session ID reports appear as integers in the Individual Sessions report, on profiles where the session ID is computed by log filters, instead of appearing as the computed value.
  • [535446] The IPC folder of LogAnalysisInfo collects many files with names HTTPRequest*, which linger for hours; these could be cleaned up much sooner.
  • [535576] Building a database for a MS SQL profile will give an error like, "drop the index 'main_table.mt_field_name', because it does not exist in the system catalog," if another profile, or a profile using different prefix/suffix, has used that same database to build a Sawmill database in the past.
  • [536085] Firewall-1 NG (text export) Log Format format does not support hh:mm:ss format for the "elapsed" field.
  • [536108] Database builds started from Config -> Database Info immediately display a "build completed" page, before the build is actually completed, when a profile is "real time."
  • [536525] PDF report generation sometimes crashes on x64 Windows.
  • [538883] Filters created in the Filters window using "is NOT item name" appear as "is item name" in the filter description of the report.
  • [539107] During a multiprocessor database build, the IPC folder of LogAnalysisInfo collects several files with names like ParsingServerPort_*.done, which linger for days; these could be cleaned up much sooner.
  • [539683] PDF export fails from the scheduler when using a drive letter in the pathname, with an error like "Can't create directory c: (File exists)".
  • [539696] Session duration shows 0 for profiles without a session ID field, and with "maximum session duration" set to 0.
  • [539814] Individual Sessions report of imported Sawmill 7 profile gives error, "Unable to read file LogAnalysisInfo/Databases/mms_pub_imported/main/Tables/ssession_idsubitem/header.cfg".
  • [540871] Deleting a database field does not delete it from all cross-references tables.
  • [541655] Schedules configured to "update all profiles" run all updates simultaneously, rather than sequentially.
  • [541975] The "Creating Many Profile in a Batch" topic is missing from the FAQ.
  • [543436] Some command line operations give the rather cryptic error message "Couldn't find node licenses in" when there is no license installed.
  • [543560] When using Firefox, page formatting is messed up if a report is generated, the server is stopped, the Admin page is accessed, the server is started, and the Admin page is accessed again.
  • [543657] Profiles with a field named "level" give an error "invalid identifier" when used with Oracle.
  • [543851] An internal string management issue could cause crashes during database builds or updates, when using a SQL database server.
  • [544036] The "Session ID" profile option is not customizable in the web UI.
  • [544295] Database updates can give an error, "Unable to allocate X bytes of memory (allocating preconversion buffer)" where X is a huge number.
  • [545134] Running the Windows installer to upgrade overwrites the preferences.cfg and default_profile.cfg files from the previous installation.
  • [546690] The label of the report element filter field has a typo: "epxression"
  • [548666] Log data with pseudo-W3C headers starting with "# Date Time" (like some Exchange 2000 logs) does not parse, resulting in empty reports.
  • [550414] Exporting PDF tables containing certain data can give an error, "Unknown HTML entity"
  • [553375] Improved performance of building indices, for the internal database, for large datasets.
  • [553713] Memory usage can be very high on profiles using the internal database, with very complex fields (fields with many unique values).
  • New features:
  • [538958] Added a -pp ("path page") command-line option for specifying the focus page for command-line export of the Paths Through A Page report.
  • [539823] Added better detection of mobile web browsers.
  • [541059] Enhanced Firewall-1 (fw log -ftn export) Log Format to handle logs with lines starting with dates; added some new fields.
  • [545835] Added support for Linksys VPN Router log format
  • [546006] Added the option to disable cross-reference tables individually, for better control over database build and report performance.
  • [548920] Enhanced support for IPtraf logs, to support a variant with single-digit days.
  • [550396] Improved performance of report generation in most cases. This is partly due to an increase in the default value of the "maximum paging caching buffer memory usage", and partly due to other optimizations. Profiles will benefit most when setting "maximum paging caching buffer memory usage" to the new default value of 64MB. In one example (a large pivot table), report generation speed increased from 13 minutes to 2 minutes; memory usage increased by less than 2x.
  • [553685] Improved progress reporting for cross-reference builds and index builds, to show more granular progress while building each xref table or index (especially when using the internal database).

New in Sawmill 8.0.6.3 (Mar 20, 2009)

  • Bugs fixed:
  • When using a prefix or suffix with MS SQL or Oracle or MySQL, the expiration query fails with a "no such table" error.
  • The "file by file" option for log processing is not editable in the web UI.
  • Database build performance is slow in some cases due to splitting processing across N+1 threads (when N is the number of processors or cores) instead of the more efficient N.
  • Log parsing uses arbitrarily large amounts of memory when reading log data from corrupt log data or log data with extremely long lines.
  • Command-line authentication login fails with error, "No Permission You don't have grants to view this page or profile. Please contact your system administrator for more details."
  • Profiles using filter_finalization give an error on database build, "Internal: sourceFileNode=NULL during ConfigNode::ParseInfixStatements()"
  • Database build generates an error, "Error in writing ODBC table main_table: ODBC error: rec1: SQLstate: 22003; msg=[Microsoft][ODBC SQL Server Driver]Numeric value out of range;" when importing data into MS SQL which has negative integers in it.
  • Browsing to a UNC pathname like \pubpublogs, on Windows, shows nothing in the right panel of the File Browser.
  • Viewing reports sometimes gives an error, "Attempt to get node number 0 from node v.temp_x_ticks."
  • Browse button gives the directory above the entered pathname, if the pathname ends with a slash.
  • Global page headers and footers do not appear in reports.
  • The "Automatically update database when older than" option is not saved properly in the UI.
  • Database builds could sometimes give an error, "Unable to read contents of directory LogAnalysisInfo/TemporaryFiles/DeleteMe_... (No such file or directory)"
  • Tasks scheduled to run at the same time run in sequence instead.
  • Temporary files are not cleaned up quickly enough, resulting in large numbers of DeleteMe files in the TemporaryFiles directory.
  • Memory usage is very high when zooming to large reports.
  • Date/time graphs with one-minute granularity show no data.
  • "Default date filter" (global per-profile date range filter) has no effect.
  • Imported Sawmill 7 profiles sometimes give error, "Couldn't find node 0 in language.english.lang_stats.weekdays_short."
  • Removing all cross-reference groups gives an error, "Couldn't find node xrefs in v.fp."
  • The Paths Through A Page report takes a very long time, and a huge amount of disk space.
  • The date filter expression "last1month" (and similar ones) give an error about incorrect date range format.
  • A database removal operation on a MS SQL database gives an error, "Unable to Execute ODBC Query='delete from main_table x using main_table x where 1=1'; diagnostics=ODBC error: rec1: SQLstate: 37000; msg=[Microsoft][ODBC SQL Server Driver][SQL Server]Line 1: Incorrect syntax near 'x'.;"
  • Months report of imported Sawmill 7 profile, where the date/time report had been edited in Sawmill 7, gives an error, "Unable to read file LogAnalysisInfoDatabases\mainTables ows_...oheader.cfg (No such file or directory)"
  • Microsoft Media Server plug-in gives wrong results for the c-rate field.
  • Generating a report with a customer date range filter string gives an error "Couldn't find node in."
  • Reports give an error, "Couldn't find node modification_time in v.lang_stats_file_info," when English language is not installed.
  • Reports show wrong numbers for fields containing single field values greater than 2 billion, when using multiprocessor database builds on a 32-bit system.
  • The "Process subdirectories" checkbox erroneously states that it is for "local folders only."
  • ODBC log sources do not support LONGVARCHAR (a.k.a. TEXT) or BIGINT fields.
  • Microsoft Media Server profiles show 0 for session duration.
  • W3C log data (like IIS log data) is very slow to parse, when using multiprocessor database building.
  • Date filter strings of the format "last1d", "last1m", and others using single-letter units, give an error, "The date filter is invalid."
  • The Profile menu in an action in the Scheduler is not alphabetized.
  • The "automatically update when older than" option sometimes has no effect.
  • The "configure" script does not complain when there is no C++ compiler, when building Sawmill from encrypted source, which causes the compilation to fail later when it attempts to compile the C++ files.
  • New features:
  • A new option has been added for doing LOAD DATA imports into local MySQL database servers, which is faster than the default LOAD DATA LOCAL INFILE, and works on servers where local infile is disabled, but requires the MySQL server to be on the Sawmill server.
  • A new variant of Unix Daemon Messages log format is now supported.
  • The name of the Sawmill executable has been changed to just "sawmill" on non-Windows platforms, instead of having the version number in the name.
  • Sawmill now determines the amount of physical RAM on the system, and when splitting queries across multiple threads with "auto," it ensures that each thread has at least 2GB of RAM.
  • A new option, "maximum read block size," controls how much memory can be allocated to holding a single line of log data.

New in Sawmill 8.0.5.1 (Feb 27, 2009)

  • Bugs fixed:
  • Fixed a bug which could cause an error when searching the documentation in Windows.
  • Fixed a bug which could cause an error on builds with MS SQL databases, if there was a database field called "rule."
  • Fixed a bug which could cause database corruption if the main table of the database was more than 4GB, with 32-bit versions of Sawmill. This generally happened with datasets more than about 30 million lines. It could cause a variety of symptoms, including out-of-memory errors while building indices, crashes during reporting, and incorrect numbers in the reports.
  • Fixed a bug which could cause an error "Parsing server returned itemnums list for unknown database field 'fieldname'" when building two databases simultaneously on Windows.
  • Fixed a bug which could cause an error "Unable to read file ... sessions_join" when no valid log entries were found in the dataset.
  • Moved the MySQL DLL to the Sawmill installation directory, to reduce conflicts with PHP and other programs which install different versions of the MySQL DLL.
  • Added support for MySQL on the x86 Linux ES4 platform, where is was disabled.
  • Restored the "create many profiles" script (now in LogAnalysisInfo/util), which was missing earlier versions of Sawmill 8.
  • Fixed a bug where multiprocessor log processing would fail with a parsing server error, if the "Server hostname" was not empty in Preferences.
  • Fixed a bug where lines of log data could be split incorrectly, for log formats where quoted values can span multiple lines, if the lines spanned the boundary between log reading blocks. This could result in slightly low numbers.
  • Fixed a bug in Shoutcast 1.8 processing, which could cause a regular expression error on parsing.
  • Fixed a bug which could cause incorrect sorting of rows in tables on 32-bit systems, if they contained large floating-point numbers (more than about 4 billion).
  • Fixed a bug which caused the graph tick marks and labels to be in the wrong locations, when rendered by Firefox.
  • Fixed a bug which could cause an error if all log filters were deleted.
  • Fixed a bug where the Create Profile Wizard incorrectly validated the pathname when regular expressions were used, causing a "no such pathname" error in some cases where the pathname was correct.
  • Added the "maximum paging caching buffer memory usage" option to the web UI.
  • Removed some database optimization options which no longer have any effect in Sawmill 8.
  • New features:
  • Enhanced support for Smarter Mail log format to handle dates containing dots.
  • Enhanced support for Wowza log format to handle large negative x-duration values (due to logging or server bugs).
  • Enhanced support for Limelight log format, to handle spaces in #Fields lines, in addition to tabs.
  • Added the "integer bits" option for database fields to the web GUI.
  • Improved handling of v7-to-v8 database conversion in the case where the database directory was overridden--it now prompts for a new directory for the Sawmill 8 database directory, so it won't overwrite the Sawmill 7 one.

New in Sawmill 8.0.4.4 (Feb 14, 2009)

  • Bugs fixed:
  • Fixed a bug which could cause progress reporting to halt partway through long database builds, on 32-bit Windows.
  • Fixed a bug in the Intermapper Event Log Format plug-in that caused an error if Events was not selected as a numeric field during profile creation.
  • Fixed a bug where reporting would hang in CGI mode, if a "temporary directory" was set.
  • Added "always include bottom-level items" to the database fields editor.
  • Fixed a bug which could cause an error when processing Blue Coat W3C logs, if the log data contained both a cs-uri-path and a cs-uri-stem field.
  • Fixed an error, "Attempt to join main_table to filtertmp_5652_1 on 's.session_id = filtertmp_5652_1.itemnum', but there must be one column from each of the two tables in the ON condition, and there doesn't seem to be", which could occur when zooming in on a particular session in DIndividual Sessions.
  • Fixed a bug which would cause an error if a template was edited in an Advanced installation, though template editing is permitted by Advanced licensing.
  • Fixed a bug which could cause an overflow when reporting processing time in ISA 2004 log data.
  • Fixed/improved cleanup of temporary files in LogAnalysisInfo.
  • Fixed a bug where PDF generation would fail if the report contained a pie chart with a legend.
  • Fixed a bug where "-a rdi" (rebuilding indices from the command line) gave an error when used with the internal database.
  • Corrected references to SawmillCL.exe, which is now called Sawmill.exe (in version 8).
  • Fixed a bug where the Cities report showed States instead, when reporting on W3C log data (like QTSS).
  • Fixed a bug where the Logout link did not work on the first click, in Firefox.
  • Fixed a bug where the Support Email Address in Config -> Miscellaneous -> Support & Action Email, did not save.
  • Fixed a bug where the Config -> Reports page would not display when logged in as a Manager.
  • Fixed a bug which could cause crashes when using the page tagging server on Windows.
  • Fixed a bug which could cause 0s in reports, when clicking single date item while displaying a date report at a different level (e.g., clicking a month, while displaying the Days report).
  • Fixed a bug which could cause an error when there was a database field named "count".
  • Fixed a bug where long values could overlap the columns to the right of them, in HTML tables.
  • Fixed a bug where database updates reported themselves as "Rebuild Database" in the Database Info page.
  • Fixed a bug where the command-line progress display did not show the number of lines processed.
  • Fixed a bug where the Sessions Overview report did not change when filters were applied.
  • Fixed a bug which could cause some log data not to parse, if the "sessions visitor ID" field was empty.
  • Added the End User License Agreement file to the distribution directory (it was in the installer, but not the installation).
  • Fixed a bug which could cause an overflow when displaying time-taken for Blue Coat W3C logs, on a 32-bit system.
  • Fixed a bug which would cause an error when filtering a report, if there was a SQL table prefix or suffix specified in the profile.
  • Fixed a bug which would cause 0 reports when using a regular expression filter on a non-itemnum field (like hour of day).
  • Fixed a bug which could cause an error, "Unknown variable 'lang_stats.field_labels.time_stamp' in expression", when reporting on Common Access log data.
  • Fixed a problem with the formatting of FAQ pages.
  • New features:
  • Modified the Intermapper Event Log Format plug-in to find the year in the log file name it contains the pattern LogYYYYMMDD.
  • Added an option to RBAC to disable password changing.
  • Extended Sidewinder Firewall reporting to show countries, regions, and cities for each IP.
  • Added suport for Squid format logging to Unix Syslog, with a Squid timestamp.
  • Added support for a new variant of Mirapoint SMTP log format.
  • Improved the performance of some internal database operations used for indices and uniques tracking.
  • Improved the performance of some internal database queries involving the sessions table.

New in Sawmill 8.0.3 RC2 (Jan 25, 2009)

  • Bugs fixed:
  • Fixed a bug which could cause database builds to terminate unexpectedly. This occurred with ASA log data, but could theoretically happen with any log data.
  • Fixed an issue with the copyright in the Czech translation.
  • Fixed/enhanced the TaskLog file to suppress the execute_sql_query, which were numerous and mostly useless.
  • Added documentation for the "Remove Reloads From Sessions" option.
  • Corrected the MacOS Install.txt file to remove incorrect upgrade instructions.
  • Fixed a bug which would cause an error when zooming on a session user, and zooming to the Session Pages report.
  • Fixed a bug which could cause an error, "Error in get_progress_state.cfv" when displaying the Overview. This bug was caused by a failure during the database build, so profiles showing this behavior will need to have their databases rebuilt.
  • Fixed a bug which could cause database builds to hang, when built from Config -> Database Info, if an error occurred while the database was being deleted.
  • Fixed a bug with the v7-to-v8 profile converter, which would cause errors when viewing reports, if the v7 profile contained references to database fields which were not translated in Sawmill 8.
  • Fixed a bug which could cause an error during database build, if the database directory was on a different drive or partition from the LogAnalysisInfo (installation) folder.
  • Fixed a bug where the current_log_pathname() function did not work when using parsing servers (multithreaded database builds).
  • Fixed a bug which would cause an error, "Syntax error: Expected ) in expression; found '" when creating a profile from BlueCoat W3C data.
  • Fixed a bug which could cause an error when relocating the database directory.
  • Fixed a bug which could cause sporadic (though harmless) crashes of the web server.
  • Fixed a bug which could cause an error, "Unknown configuration group 'cs_uri_stem_bottom_level_items'" when viewing reports for a Microsoft Media Server profile.
  • New features:
  • Improved the Windows installer to omit the minor "rcN" from the version number, to simplify the Sawmill version number display, and make it clear that these is a production releases (successful release candidates), not pre-production releases (unsuccessful release candidates).
  • Improved documentation of "session events" and "sessions."
  • Added section to documentation, "Adjusting date/time values for Daylight Savings Time".
  • Enhanced the Limelight plug-in to handle arbitrary W3C headers, allowing variable field layout.
  • Enhanced the Squid plug-in to handle "action" field values containing spaces (like "TCP MISS").

New in Sawmill 8.0.2 RC2 (Jan 17, 2009)

  • Updated the Czech, German, and Polish translations.
  • Enhanced Radware DefensePro plug-in to handle ip:port variation.
  • Enhanced Flash Media Server plug-in to support analysis when there is no cs_uri_stem field.
  • Implemented a custom action, reset_root_admin, for resetting the root administrator password from the command line.
  • Added more documentation about real-time log importing. Bugs fixed this version:
  • Fixed a bug where the "automatically update if older than" option was not saved in Config.
  • Fixed a bug which caused an error when creating an ISA profile: 'Unknown configuration group "filterinfo" in node "profiles.isa_logs.log.fields".'
  • Fixed a bug where some hierarchical reports would show no results when drilled into.
  • Fixed a bug where direct login by URL would sometimes give an error.
  • Fixed a bug which would give the error 'column "" does not exist' when analyzing Bluecoat log data.
  • Fixed a bug which could cause an error when using a literal backslash in a custom report expression.
  • Fixed a bug which could cause an error while analyzing Nortel ACD log data: 'Unknown configuration group "avg_agent_time_busy" in node "profiles.test.log.fields.'
  • Fixed a bug which could cause an error when clicking Scheduler after importing v7 profiles with field names containing numbers.
  • Fixed a bug where full-month filters on the Overview gave zero results.
  • Fixed errors compiling the encrypted source on SuSe 11, and other recent operating systems.
  • Fixed a bug where the progress display for one profile could show in another profile's Reports display, if there were two simultaneous profiles in use.
  • Fixed a bug where x86 Windows was reported as x64, and vice versa, in automated bug reports.
  • Fixed several places where images were broken in CGI mode.
  • Fixed a bug where very long field values could generate an error, when using ODBC.
  • Fixed a bug which could cause an error when analyzing Sidewinder log data with Microsoft SQL Server.
  • Fixed a bug where Log Detail report was empty when using ODBC databases.
  • Added support for not splitting queries at all, in the web interface.
  • Fixed a bug where some URLs used HTTP, even when running Sawmill in CGI mode under an HTTPS server.
  • Fixed a bug where parsing servers could linger after the process that spawned them was gone, in the event of a parsing error.
  • Fixed a bug which caused an error, 'Internal Error: Empty node name,' when setting a session field to "(None)".
  • Fixed a bug which caused an error when generating the Top Malware report in Ironport S-Series logs, 'Couldn't find node display_format_type in sessions_cache.xyz.profiles.profilename.extended_profile_dat.'
  • Fixed a bug which could cause an error like, 'Unable to read file LogAnalysisInfoDatabasesprofilemainTables ows_1_6248oheader.cfg' on location reports, in imported v7 profiles.
  • Fixed a bug which could cause sporadic crashes of the web server process.
  • Added the update_xrefs_on_update option to the web interface.
  • Fixed a bug which caused the Help link to sometimes have no effect.
  • Fixed a bug where pivot tables would sometimes show NULL rows.
  • Fixed a bug which could cause errors when using a SQL prefix or suffix.
  • Fixed a bug which could cause an error, "Unknown variable 'lang_stats.field_labels.ms_ras_client_name' in expression."
  • Fixed a bug which could cause an error when analyzing Sidewinder logs with MS SQL.
  • Fixed a bug where Config/Reports did not prompt to save changes.
  • Converted some hard-coded strings to language module strings, for internationalization.
  • Fixed a bug which would cause the error, "Couldn't find node next_pages in v.query_result.data" when viewing a Session Paths report with no data.
  • Fixed a bug which prevented error messages from being viewed when not logged in.
  • Fixed several broken images in the documentation, in CGI mode.
  • Fixed a bug where non-root-admin users could not see the version number of Sawmill in the web UI.
  • Fixed a bug which could cause the error, "Unable to read...mainTablessessions_joinheader.cfg (Day of week)."
  • Added the missing language module variable lang_admin.log_filters.simplify_playerid_label.
  • Fixed a bug which would cause error when rebuilding a database, if it was stored a different drive or partition from the LogAnalysisInfo folder.
  • Fixed a bug which would cause database build errors when the log data contained extremely large or infinite numbers.
  • Restored the missing FAQ about emailed report and Outlook 2003.
  • Fixed the FAQ about resetting the admin password to describe the new method.
  • Fixed a bug where the SPARC Solaris 9 distribution did not have the necessary libraries included.
  • Fixed a bug where Config -> Log Source did not support multiple log sources in Professional tier.
  • Fixed a bug where once a trial license expired, it would not accept another license key.
  • Restore the Support page from Sawmill 7, to Sawmill 8.
  • Restored the HTML comment describing the filters, from Sawmill 7, to Sawmill 8.

New in Sawmill 8.0.1 RC3 (Dec 25, 2008)

  • Bugs fixed:
  • Fixed a bug where once a trial license expired, Sawmill would not accept a new license.
  • Fixed a bug where a Sawmill 7 license would be called "invalid" instead of being reported as a valid older license, no longer valid for Sawmill 8.
  • Fixed the upgrade instructions in the README of some platforms, which were still describing the Sawmill 7 upgrade method.
  • Fixed a bug where Sawmill 8 would not install its service, if Sawmill 7 was already installed, and would uninstall the Sawmil 7 service when Sawmill 8 was uninstalled.
  • Fixed a bug in the Create Profile wizard, which would cause an error if the Pathname was not literally a valid existing pathname, even if it contained wildcards which should have matched valid pathnames. This almost always caused wildcard or regular expression log sources to fail on profile creation.
  • Fixed a bug where wildcard and regular expression report filters did not work (generated an error when attempted) with Microsoft SQL Server profiles.
  • New features:
  • Added a separate chapter to the documentation about real-time importing.
  • Added a Support link to the Admin page.
  • Added the Salang expression of the current filter to the HTML of the report, as an HTML comment (useful for creating command-line or scheduled filters, or for debugging).
  • Chopped off the "rcN" part of the version number in the web interface, to make it look better.
  • Switched the default port of the web server to 8988. This makes it much simpler to run Sawmill 7 and Sawmill 8 together on the same system.
  • Enhanced support for ISA W3C format, to handle a significant variant (2007)
  • Added support for Unicode with Microsoft SQL Server as back-end database, so non-ASCII log data can be imported and stored in MS SQL, and queried from outside Sawmill.

New in Sawmill 8.0.0 RC5 (Dec 16, 2008)

  • Changed the GUI concept from html frames to single pages.
  • Added report fields for more flexibility and fine tuning of report elements and table data.
  • Added a simplified date_filter syntax (i.e: 2m, last2m, etc.) for date/time filtering.
  • Added a new caching system which caches various report components and database data independently.
  • Added RBAC (Role Based Access Control)
  • Added support for sequential actions per schedule in scheduler.
  • Added a "Run Now" button in the Scheduler, to run any task immediately.
  • Added log fields editor
  • Added database fields editor
  • Added session fields editor
  • Added report fields editor
  • Added new field wizard (which allows to create a log field, database field and report field at once)
  • Added URL support to view reports by URL definition by defining: profile name, report name, date filter, filter expression, filter comment
  • Added support to view any hierarchical depth of a hierarchical database field as non-hierarchical report. This allows i.e.: to view a months report or a region or city report. (These reports have been added by default)
  • Added the calendar as optional report.
  • Added support to dynamically create a pivot table within the reports GUI
  • Added support for different sort field and sort direction of the drill down field.
  • Added support to drill down data on a table with multiple string fields.
  • Added a new date picker which combines single date, date range and relative date selection.
  • Changed the zoom concept in that zoom automatically adds the zoomed item to filters.
  • Added support to zoom to multiple items at once.
  • Added support to save filter items as filter group
  • Improved the filters editor.
  • Added support to email a report within the reports GUI
  • Added support to define the row numbers and aggregation rows when exporting a table within the reports GUI
  • Added min and max aggregation rows to tables.
  • Added a row_visibility_expression per report element. This expression allows to show/hide table rows by an expression, i.e. show only rows where page_views > 300 and page_views < 1500.
  • Added support for a default date filter per profile.
  • Added support for a date filter per report or per report element.
  • Improved the Customize Report Element form/options.
  • Added table column info support.
  • Added table row selection support (to mark a row in yellow color).
  • Added support for 3D pie charts
  • Added support for antialiased PNG graphs
  • Added support for use of MS SQL or Oracle databases as back-end databases (where processed log data is stored, and reports generated from).
  • Added support for use of MS SQL, Oracle, or MySQL database as log sources
  • Implemented multiple scalability improvements, to allow Sawmill to process log data with less memory usage. These largely lift the restrictions of log processing on 32-bit systems, so any amount of log data can be processed without exceeding the address space of a 32-bit system.
  • Added real-time reporting. Reports can be generated while data is being imported, and will be up-to-the-moment, based on the latest imported data. Data can be streamed continually into the database, without any reporting downtime.
  • Implemented major SQL performance improvements, especially in the building of cross-reference tables, and in the performance of database update.
  • Enhanced the internal database to support SQL syntax, for universal SQL querying of any Sawmill database.
  • Added support for reading log data from a SFTP server
  • Added directory recursion on (S)FTP servers, so a log source can point to a directory, and all subdirectories will be processed.
  • Added filtering of reports on numerical fields; e.g., show all events where bytes > 1000
  • Implemented multiprocessor splitting of report queries: report calculations are split across multiple processors for better performance.
  • Enhanced multiprocessor and multi-system log processing. Log process is now done more efficiently on a single system, and does not use the disk as much; and it is also possible to split log processing across multiple "parsing servers" on the same network.
  • Added database import and export.
  • Added support for user-created actions (-a options), with fully customizable parameters and behavior.
  • Enhanced the functionality and performance of the Session Paths report.

New in Sawmill 8.0.0 Beta 3 RC3 (Nov 24, 2008)

  • Fixed an uninstaller bug where the Sawmill 8 icon remained on the desktop after uninstall.
  • Fixed/improved the progress display for database builds, so it includes all steps, and shows better descriptions of each step.
  • Fixed a bug with v7-to-v8 profile converter, which did not set up the database tuning options properly, resulting in an error when viewing the Database section of Config.
  • Fixed conversion of v7 MySQL database in the import wizard.
  • Fixed a bug where Sawmill could not see files on a Microsoft FTP server.
  • Fixed a bug which could cause an error when zooming on session fields, and displaying reports generated without cross-reference tables.
  • Fixed a bug where indices were completely rebuilt after database updates; they are now properly incrementally updated from the new new data.
  • Fixed a bug which could cause a "duplicate key" error when viewing reports from a MS SQL database.
  • Added support for input of "node" licenses through the web interface.
  • Fixed bug where the database build would fail if no entries were accepted.
  • Fixed error which could occur when building from Microsoft Media Server logs.
  • Fixed a bug where CSV export in CGI mode had a broken link for the CSV file.
  • Fixed a bug which could cause an error when building a database from Ironport S-Series logs.
  • Fixed a bug where references to non-existent template pages would give an error "no node 'templates' in 'templates'".
  • Fixed the timestamp of emailed reports sent from Windows.
  • New features:
  • Improved performance of a common type of query on in internal database.
  • Reduced memory required by xref builds and other queries.

New in Sawmill 7.2.15 (Jul 1, 2008)

  • Fixed bug in the Helix Universal Server (Style 5) Log Format where the File Time field was being treated as milliseconds while the Sent Time field was being treated as seconds. According to documentation at real.com, both fields contain times expressed in seconds.
  • Fixed a bug which could cause a crash (which would appear in the Sawmill GUI as a hang) when autodetecting data on an FTP or HTTP server.
  • Fixed memory leak which could occur in various circumstances; the specific known circumstance occurred when building a database from a profile with more than 1500 log sources, which caused more than 1GB of memory to be used.
  • Fixed a bug where the number of visitors could be overstated by 1 in Microsoft Media Server log format.
  • Fixed a bug in the Critical Path POP3/IMAP plug-in which could cause an error when creating a profile.
  • Fixed a bug where the "day of year" and "week of year" fields split the day at 23:00, instead of 0:00, on days under daylight savings time.
  • Fixed a bug subtable Table options were not saved and restored properly, when editing a "table with subtable" report in the report editor.
  • Fixed a bug in RACF Security log format, which prevented it from importing the final record in a file.
  • Fixed a bug in RACF Security log format, which prevented it from importing lines where the username contained no spaces.
  • Fixed a bug IronPort C-Series parsing, where SBRS rejects were not reported.
  • Fix incorrect reporting of sessions in the Flash Media Server plug-in by only creating session events when x-event eq disconnect and x-category eq session.
  • Fixed bug in Sidewinder analysis (logged to firewall) which caused incorrect dates when there was a date= field listed.
  • Fixed a bug where certain filters (especially, ORs of "within" filters) could cause main table scans, when they could have been handled by xrefs. This made some filtered reports slower than they should have been.
  • Enhanced Sawmill.app (on Mac) to detect when there is a running installation of Sawmill already, and give an appropriate error message (rather than hanging while it waits to bind to the port).
  • Deprecated the "maximum CPU usage percent" option. The option never worked very well, and has done absolutely nothing since Sawmill 7.0.0, so it serves no purpose. Instead, use operating system priorites to minimize the impact of Sawmill's CPU usage on other processes.
  • Added support for CP Secure Content Security Gateway log format.
  • Added support for a new version/variant of Aruba Wireless Switch.
  • Added tracking of "Context" lines in Citrix Netscaler log format.
  • Added support for Unix Auth log format.
  • Added support for Unix Cron daemon log format.
  • Added tracking of VOF quarantine lines in IronPort C-Series logs.
  • Added reporting of Amavis information in Postfix logs.
  • In the Kiwi YYYYMMDD Comma Syslog plug-in, added stripping of double quotes from around the syslog message since these can break autodetection. If the message is quoted, the plug-in also now changes doubled double quotes back to single double quotes. Doubling is the way Kiwi escapes them.
  • Added support for a new FortiGate 100 Firewall format with additional fields to the FortiGate Comma Separated Log Format plug-in.
  • Added support for Symantec Gateway Security Log Format (via syslog).
  • Added alias domain reporting to Microsoft Exchange 2000 log format.
  • Added support for automatic charset conversion of search engines which do not use UTF-8 in their search URLs (specifically, Yandex).
  • Added reporting of MailScanner lines in Postfix log data.
  • Added a new plug-in to support the SNARE Epilog Collected Oracle Listener log format. The plug-in was contributed by a Sawmill user.
  • Expanded the plug-in for the Nortel Meridian 1 Automatic Call Distribution (ACD) log format to include some additional fields from the logs and an additional graph in the Date/Time reports.
  • Added session analysis to the Flash Media Server plug-in for the purpose of reporting the Maximum Concurrent Connections.
  • Added support for the Users field and a Unique Users numeric field to the Proxy Plus log format plug-in.
  • Added support for Tipping Point SMS Log Format.
  • Added reporting for ARP request and ARP reply lines in Cisco VPN Concentrator.
  • Added support for AspEmail (Active Server Pages Component for Email) log format.
  • Fixed a problem with Cisco VPN Concentrator log format, which caused certain "disconnected" lines to be ignored.
  • Added support for tracking/reporting of the usr field in SonicWall format.
  • Changed label for the Barracuda Spyware Firewall Log Format plug-in to Barracuda Spyware Firewall / Web Filter Log Format to reflect new product name. Added support for standalone (no syslog header) format. Added support for lines where the action is "sniff" instead of "httpscan". Added Action report.
  • Made extensive changes to the Anti-Spam SMTP Proxy (ASSP) log format plug-in. Messages, which are described on multiple lines of the log, are now captured in one database entry so reports are more clear and counts are more accurate. These changes apply to log formats for 1.3.3.1, 1.3.3.8 (and in between, presumably, though they have not been tested). Reports for earlier versions of ASSP that have a different log structure are not changed.
  • Enhanced the JBoss application server plug-in to support a slightly variant.

New in Sawmill 7.2.14 (Mar 27, 2008)

  • Fixed a bug in the IceCast Log Format plug-in where the User Agent field was not being set causing the fields that are derived from it, such as Web Browser, to be empty.
  • Fixed a date/time parsing bug in Barracuda Spam Firewall, where some lines were reverting to the syslog collected date/time instead of the Barracuda's date/time.
  • Fixed a bug in the FirePass SSL VPN Log Format caused by an incorrect variable name. The bug would only have been seen if lang_stats.cfg did not have the firepass_ssl_vpn status code mapping section.
  • Fixed a memory leak which could cause very high memory usage when building a MySQL-based database from a database with many unique values in one or more fields.
  • Fixed a bug in the Unix Syslog With Year plug-in where the syslog message was being lost.
  • Fixed a bug which could cause an error in various circumstances (but usually when building a database) on 64-bit Windows, when one of the mapped files in the internal database exceeded 2GB. This is rare, but can happen to the indices if the "main table segment size" option is set to a very high value.
  • Fixed a bug in the parsing regular expression where the report of multiple Stats or square brackets in the client_info field would cause the entry to be rejected.
  • Changed the IceCast Log Format plug-in to get the duration in seconds from the duration field instead of calculating the duration from the size and an assumed speed. Apparently the duration field was not available at the time the plug-in was first created so a workaround was used.
  • Enhanced Ironport C-Series plug-in to extract more information about antivirus scanning.
  • Added support for charset conversion on 64-bit Windows.
  • Enhanced Tipping Point IPS log format to handle log lines generated by the 2.4.3 firmware revision.
  • Added support for OpenVPN log format.
  • Added support for CRYPTO lines in Cisco PIX/IOS/etc. format.
  • Added a new "Save To Menu" button to the Reports page, to save a filtered report directly to the reports menu.
  • Added support for a format variation with a date as well as a time to the Windows 2003 DNS Log Format plug-in and increased the flexibility of the autodetect regular expression.
  • Added support for Tipping Point 2.5.3 log format.
  • Improved performance of hierarchy builds for MySQL databases. With this change, the time to build the hierarchies for a specific database with 16 million unique IPs dropped from 2:15 hours to 0:40 hours.
  • Added a new profile option, "Use Overview For Totals." This option controls a recent new feature, which computes the Total rows of report using an Overview report.