Softpedia >Mac >Math/Scientific >LONI Pipeline >  Changelog

LONI Pipeline Changelog

What's new in LONI Pipeline 7.0.1

Apr 10, 2017
  • Improved: certain issues pertaining to the design and functioning of the grid usage thread
  • Improved: reduced server stat plugin message size that is sent to the clients to 50,000 jobs to improve performance
  • Fixed: various Hibernate/DAO problems and table configurations
  • Fixed: permissions problems of files uploaded through script editor
  • Fixed: bug on Mac systems that caused the update dialog to keep appearing
  • Fixed: transformation bug preventing double replace operation
  • Fixed: bug sometimes preventing change requests to server library
  • Fixed: job-based sorting bug in the grid stats window

New in LONI Pipeline 7.0 (Apr 3, 2017)

  • Added:
  • New window in the help menu which summarizes how Pipeline works together with a compute cluster
  • New window for displaying detailed information about the grid using new grid stat plugin
  • Remote and local file editor feature added to module definition window and Tools menu
  • Job management system by module name across all users
  • List of program packages available to users on a server is now displayed in the module creation interface
  • Upgraded:
  • Pipeline server database upgraded to Postgres from HSQLDB with new schema.
  • Pipeline server’s database access framework updated to Hibernate from JDBC
  • Improved:
  • User quota’s calculation now factor in overlapping grid queue resources. Quota will more accurately display to users how many resources are available to them
  • More reliable checking of module output files existence
  • Variables can now be used in data sources with the “Number” data type
  • Efficiency of getting data from IDA within Pipeline.
  • Workflow loading statuses are now displayed
  • Loading times of module information
  • XNAT can now run locally on all operating systems, previously worked only on remote Pipeline server

New in LONI Pipeline 6.4 (Nov 5, 2016)

  • Added: Webapp beta is now available at valve.ini.usc.edu/pipeline/
  • Added: Workflows that fail to validate are now editable. Ignore warnings button will become unavailable when the workflow is modified
  • Added: Feature for users to customize server library. Requests to add, rename, update, change package, or delete workflows are submitted and approved/denied by admins
  • Added: New icons for connection button, reset/stop/play/pause buttons, and new workflow and load workflow buttons
  • Added: Colors in the parameters tab of modules to indicate inputs (blue) and outputs (red)
  • Added: Error and warning colors to parameters and connection lines to indicate any validation problems
  • Added: Server address displayed in the active sessions window when user has more than 1 connection or local sessions
  • Added: API for requesting session and job statuses
  • Improved: Method of querying database for invalid sessions to avoid database lockups
  • Improved: Multiple clients open at the same time will sync their connections list
  • Improved: XML view for workflows is now a toggle. You can view the xml while a workflow is executing reconnect to the session when switched back to workflow
  • Improved: During validation, if the package mapped path doesn't exist, the user entered value will be used
  • Improved: Validation errors box automatically sorts by module. Module groups, modules, sources, and sinks, can be right clicked to "Show Errors" pertaining to them
  • Improved: Users can now use shell functions in their modules without grid engine array jobs enabled
  • Improved: Study modules now saves user inputted entries during the module creation process
  • Improved: The way the authentication process is displayed is more informative and clear
  • Changed: Processes will now explicitly run from users home directory
  • Changed: Study modules are now called "Data sources with Metadata"
  • Changed: Max cardinality increased from 3,000 to 10,000. This is the number of total commands generated by a workflow.
  • Changed: Transformation replace function will now replace parts of entire path including the base
  • Changed: Validation Warnings feature enabled by default for local execution (ignorable validation errors)
  • Changed: Dropbox account linking updated and NDAR authentication protocol updated
  • Changed: Various popup menus have been organized and simplified
  • Fixed: Bug where metadata wasn't being copied properly in many circumstances related to periods in file/directory names
  • Fixed: Download file window was not displaying transfer speeds and status accurately
  • Fixed: Server changer tool was not changing sources and sinks properly
  • Fixed: "Copy with input" and "Copy output" were not sorting their values so that they match original order and also any paired metadata files
  • Fixed: Various problems viewing metadata files with strict permissions set or when using relative (package mapped) paths when you have no connections
  • Fixed: Bug where session will not appear in active sessions if you reconnect to a session, reset it and run it again
  • Fixed: Module execution hanging issue on certain servers when a user connects to the server using it's public ip which differs from it's private ip
  • Fixed: Issue where a module connected to a data sink that has no slashes "/" in it's path will not execute
  • Fixed: Execution status of rotated modules not repainting properly
  • Fixed: Module group status was not taking into account contained modules in the waiting state, would show complete despite waiting modules
  • Fixed: Problem related to execution statuses when restarting a module connected to a module group who's contained modules are dependant on restarted module
  • Fixed: Instance statuses in the execution info dialog were not updating properly
  • Fixed: Bugs preventing users from closing the execution info dialog
  • Fixed: Status of files not updated if you check them in the output files tab before they are in their final state
  • Fixed: paths using global variables $username and $tempdir showed null
  • Fixed: visual bugs when switching between the different toggles (i.e. Import data mode, merged/separated views)

New in LONI Pipeline 6.3 (Apr 22, 2016)

  • Added: LRUCache to decrease number of DB queries and improve overall server’s performance.
  • Updated: IDA communication library, older versions of Pipeline won’t be able to communicate with the new version of IDA server.
  • Fixed: Remote file browser to support filenames with spaces.
  • Fixed: Issue when file viewer would not open if another file from the same module was downloaded before viewing the current file.
  • Fixed: issue when data sinks were shown as incomplete, while they were already complete.
  • Fixed: double slash appearing in output/error streams paths.
  • Fixed: issue when some modules after conditional modules would get cancelled after module restart.
  • Fixed: to properly transfer output parameter arguments of conditionals to child modules.
  • Fixed: End time of conditional's instance was not properly reported.
  • Fixed: issue with some conditional modules with metadata not being properly marked as complete.
  • Fixed: issue related to restarting modules with data extract feature.
  • Fixed: issue related to execution of local modules with spaces in their path.
  • Fixed: When workflow gets modified outside of pipeline and it is in "Executing" state, reloading option won't be offered.
  • Fixed: Removing exported parameters from Transformer modules will now properly remove them from parent module group.
  • Fixed: When delete connections between conditional modules and modules with metadata, conditional's true/false parameters won't shift around anymore.
  • Fixed: to not show "Executing Workflow 100% done" in GUI Client after a module gets restarted.
  • Fixed: avoid processing messages in a new thread for CLI.
  • Fixed: to not show reset confirmation box when workflow has validation errors.
  • Fixed: data extract bug, which was missing some values of its instances.

New in LONI Pipeline 6.2.1 (Dec 9, 2015)

  • Pipeline Client / Server:
  • Fixed: bug preventing removal of session from Active Sessions window
  • Fixed: conditionals with multiple parameters and multiple instances would hang forever.
  • Fixed: issue when reconnecting to backlogged workflow would show it as "waiting"
  • Fixed: issue which was preventing deletion of some invalid sessions upon server startup.
  • Fixed: issue when short taking file transfers to data sink would fail
  • Fixed: to properly show data sink statuses
  • Fixed: issue with conditionals which are connected to module groups.
  • Fixed: issue when instances would stay backlogged even after module restart
  • Fixed: issue related to IDA key file generation on servers where privilege escalation is disabled.
  • Fixed: issue when study module groups would fail to properly update
  • Added: a new logic to conditionals: When True and False parameters are connected to same parameter, parameter arguments will be delivered in order.

New in LONI Pipeline 6.2 (Dec 9, 2015)

  • Pipeline Client / Server:
  • Added: An advanced option to Modules which will always force an outer product multiplication (even when all parameters have the same amount of data).
  • Added: Session ID field in Execution Log dialog
  • Added: A confirmation box for Reset Workflow action to prevent accidental removal of workflows.
  • Fixed: Infinite waiting issue in local execution.
  • Fixed: Command line and Client to properly show the workflow completeness percentage.
  • Fixed: An issue which was showing non-existing files as Pending in data sinks.
  • Fixed: to create only 1 symlink when modules have connections to multiple data sinks.
  • Fixed: Issues when sometimes some workflows stopped updating or some instances failed to initialize.
  • Fixed: Metadata augmentation to properly use user-specified values.
  • Fixed: Bugs related to downloading and showing metadata logs.
  • Fixed: issue when Pipeline was not properly post validating files with prefixes.
  • Server Terminal:
  • Improved: Speed of workflow list retrieval
  • Fixed: Issue when some Paused workflows would show up as Running

New in LONI Pipeline 6.1.1 (Nov 4, 2015)

  • Pipeline Client / Server:
  • Redesigned execution events structure. Redesigned Execution Graph Controller to support multiple graphs.
  • Improved: Restart module speed.
  • Improved: Status updates
  • Fixed: Issue with restart modules which would hang the workflow for indefinite time
  • Fixed: Incorrect user management limit calculation for users within a defined group.
  • Fixed: The wrong name for Session History table.
  • Fixed: Package mapping issue so when a package has a specific version as well as a * version, it will only use the specific version of the package mapping.

New in LONI Pipeline 6.1 (Aug 12, 2015)

  • Pipeline Client / Server:
  • Added: Transformer modules. A standalone module that allows complex file path transformations with simple user interface.
  • Added: user group management. System admins can create custom user groups and apply quota for the users in the group as well as total quota for the group. Quota can be a percentage, a hard limit, or a fixed/reserved hard limit.
  • Added: unknown cardinality for string extract of output parameters. If enabled (match all occurrences), data extract will pass all matched items and the next module will have n number of instances.
  • Added: output files information dialog for data sinks. Similar to module execution dialog's output tab, it shows list of output files and their paths, and allows user to download/view the files.
  • Added: support for multiple status lines (instead of just 2) for Modules.
  • Fixed: when "show parameter names" is enabled, adding new parameter will have its name shown.
  • Fixed: for thumbnail generator to allow multiple files to be processed with 1 process, and for slicer to process the first output files instead of all files. These will improve module's post processing time.
  • Fixed: for restarting module with data extract, the correct cardinality for the next modules will be calculated.
  • Fixed: problem which would cause some modules to not have some of their arguments places in the command if they are places after conditionals or any module which changed their cardinalities.
  • Fixed: to only reset statuses only of modules which are children of module group receiving the event.
  • Fixed: to properly shutdown any child controllers when workflows stop or reset.
  • Fixed: for server manager's combined settings message not to do busy loop, null value settings won't be sent to client.
  • Fixed: bug which was changing transformation's parameter when selecting a transformation from the table.
  • Fixed: bugs when use study module's group as output parameter option.
  • Fixed: for IDA dialog when user create IDAGet module with data source, it will include subject id that's parsed from metadata.
  • Fixed: shortcut key conflicts for transformers and image annotations.
  • Fixed: for data sink output dialog to not show pending next to file names until it's confirmed pending. Also fixed to have correct parent for error dialog.
  • Fixed: for module's execution log dialog to not run things (like updates on labels and buttons) every 1 second when it is invisible.
  • Fixed: for IDA dialog when user selects file type but it's not defined (in userdata), the filetype will be created.
  • Fixed: for data sink output status dialog to open viewer when user double-clicks on an item. Also fixed dialog title.
  • Fixed: for nfs file check for non existing files or no permission files, to check recursively its parent directory.
  • Fixed: to clone data module's general metadata fields.
  • Fixed: to have "show in finder" right click menu option for downloaded output files.
  • Fixed: for ValidationAllowedPaths/NFS output validation check to only check files from required output parameters.
  • Fixed: for edit study module's matrix tab to have predefined set of columns for IDA studies, also fixed to show the last element name as matrix title (instead of full path).
  • Fixed: to have an environment variable for all modules submitted by guest users, with name "pl_guest_username" and the value is the guest username.
  • Fixed: for IDA to correctly save guest users credential file. Also fixed to check if the directory ~/.pipeline exists before running mkdir.
  • Fixed: for server to apply commands of thumbnail generator and data view (slicer) separately, as they have different sets of supported file formats.
  • Fixed: to automatically delete directory with port name only from libraryCache to prevent showing outdated modules and workflows in server library.
  • Fixed: bug which was creating a directory with port name only in library cache (but should be hostname and port). This was causing some libraries to appear outdated.
  • Fixed: for restart module/instance to remove output and error streams and output files of previous run.
  • Fixed: server preferences option to automatically check if a file is in NFS (with stat command). This option can be used in combination with ValidationAllowedPaths (renamed from NFSValidationPaths).
  • Server Terminal:
  • Added: interface for creating and editing users and quota for user group management.
  • Fixed: to correctly display name and selected index for server terminal launched from GUI menu.

New in LONI Pipeline 6.0.1 (Apr 15, 2015)

  • This version includes new server features such as NFS file validation, output files check, and server library cache. It also includes improvements and bug fixes for client such as restart failed instances, reconnect workflows, XNAT and Box libraries, local executions, string extract and much more.

New in LONI Pipeline 6.0 (Feb 25, 2015)

  • Pipeline Server:
  • Redesigned: The database schema, this means that databases used for previous versions won’t work with this version. It needs a fresh instance of DB.
  • Redesigned: Job submission thread pooling, execution management and user management
  • Redesigned: Module execution and workflow status events
  • Added: Directory Access Restrictions feature. This feature will allow administrators to set multiple directory access control with more options and should replace the DACMode feature.
  • Fixed: to clear old (last modified >30 days) client's log files and client's temp persistent db files (the ones with long timestamps).
  • Fixed: IDA to automatically renew key (if available) when user submits a workflow with IDAGet.
  • Pipeline Client:
  • Added: "add metadata" and "remove metadata" buttons. They provide functionality of deprecated Study modules.
  • Added: to show node errors for modules.
  • Added: for data extract/string output parameter the option to extract the entire output/error stream.
  • Added: Auto select file type button in Data Module
  • Added: access to Server Terminal from Tools menu
  • Fixed: for edit and add file type to strip leading dot on extension and needs (e.g. if user gives .img, it will save as img).
  • Fixed: for making connections with shift key to overwrite smartline settings, pressing shift key while holing the mouse will enable/disable smartline.
  • Fixed: to properly show status of module groups which contain cancelled modules.
  • Fixed: for validation to catch error when module has output string/number parameter that contains values (i.e. not connected to another module).
  • Fixed: bug which was causing some module groups to show as complete instead of waiting.
  • Fixed: to report an error and stop workflow submission in case there is an error during session initialization.
  • Fixed: not to set information about module outputs when the list is empty.
  • Fixed: to properly cancel modules for which all needed parents have been failed or cancelled.
  • Fixed: to cancel nodes for which all parents are either cancelled or failed.
  • Fixed: concurrency issues when multiple nodes were started together.
  • Fixed: a few ungroup bugs, including exceptions that could hang the ungroup action, and incorrect connections after ungroup.
  • Fixed: to transfer output arguments into child modules after the translation is done. This fixes the problem of some data sinks which instead of linking would try to transfer files.
  • Fixed: for smartline not to have memory value specified by default.
  • Fixed: on GUI validation with missing input, when user clicks "Ignore Warnings", module's package mapping will work as usual.
  • Fixed: for previewer to correctly show black blank images instead of showing Not Available error.
  • Fixed: for the hidden metadata output parameter to be checked at workflow loading time, in addition to connection line create/delete time.
  • Fixed: for workflows with study module, when a connection is removed, it checks and removes the hidden output metadata parameters recursively.
  • Fixed: for workflows with incomplete modules to show "errors found" instead of "completed" workflow status message, it will have orange (same as incomplete module) color.
  • Fixed: on GUI validation with missing input, when user clicks "Ignore Warnings", module's package mapping will work as usual.
  • Fixed: When server library gets corrupted for some reason, clear library cache button will correctly reset/remove existing server library cache.
  • Fixed: for client logging file to append instead of create new for every start.
  • Fixed: to only get URLs from valid URLs (not strings or numbers) when using server changer.
  • Fixed: bug when exported parameters are removed, the parent module group's parameter nodes will be updated.
  • Fixed: to cache the user permission which is used for restricted resources.
  • Fixed: to report error when node tries to submit more jobs than allowed
  • Fixed: for copy with input issue with data extraction/string output parameters.
  • Fixed: server exceptions when sending pre-delete email on sessions that are paused for x days.
  • Fixed: to show error when user runs an empty workflow, or workflow contains only data source, study, cloud source modules.
  • Fixed: to check multiple User type rules if there are any before restricting the user.
  • Command Line Interface:
  • Added: the options to list all active sessions, show status of a given session, and stop/reset/pause/resume a given session.
  • Added: the option to ignore validation warnings to submit workflows (warnings will be printed).
  • Fixed: to show workflows that are completed with error the correct status (instead of failed).
  • Fixed: not to require preferences.xml in user's home, also fixed to remove persistence*.tmp file in user's home.
  • Fixed: the ability to assign values to data sources and sinks. It uses "parameter_id=value" format.
  • Fixed: if there's parameters value assignments, the value(s) will replace the existing ones from .pipe file. Before this commit it appends the values to the existing ones.
  • Fixed: validation error message when using predefined variables ($tempdir and $username).
  • Fixed: manage options to support short session IDs in addition to complete session IDs.
  • Fixed: to show workflow status as complete for workflows with canceled instances (due to conditional module) but no errored ones.
  • Server Terminal:
  • Added: synchronization/update checking mechanism. When user saves preferences that are outdated (saved by another session), it will inform either overwrite or discard/refresh options.
  • Added: multiple selection support, which allows bulk reset/stop/pause/resume workflows. Also added highlight for long running workflows when user clicks on the label.
  • Added: for server terminal the option to show customized warning message when there's a server update.
  • Fixed: to hide "Retrieving data" label after data received from server.
  • Fixed: not show interface when not connected to server and added a message to users about sudden disconnection (happens when regular - non admin user tries to connect to server).
  • Fixed: for server terminal reset workflow button to update total counts (number of workflows, running jobs, etc).
  • Fixed: not append unnecessary comma when guests, shell or noshell are used with other usernames.
  • Fixed: reset and pause/resume buttons not to continuously call refresh after button action. Removed stop action and replaced with reset action.
  • Fixed: to have correct total instances count for running workflows.
  • Pipeline Web Start:
  • Added: CSV Metadata URL as argument. This will allow web start launching with import CSV file as data module with metadata.

New in LONI Pipeline 5.9.1 (Aug 22, 2013)

  • Added: Access control for guests, it can be specified as {guests} in server preferences and terminal.
  • Fixed: User management bugs related to comma separated queue enumeration in GridSubmissionQueue property.
  • Fixed: To have the option to have all data formats in NDARGet module.
  • Fixed: To report usage of non used queues.
  • Fixed: To not put jobId as a terminating signal when job finishes.
  • Fixed: Bug for transformation with parameter based on itself, the transformation will correctly apply.
  • Fixed: Problem with some ISPs reporting the false USC move.

New in LONI Pipeline 5.9 (Jul 24, 2013)

  • Added: User Management for guests and separate queues.
  • Added: Metadata downloading for cloud modules
  • Fixed: Bug for command line to avoid duplicate transformations.
  • Fixed: Module restart speed for modules with many instances.
  • Removed: Grammar View feature, please use Workflow Miner instead.
  • Other bug fixes

New in LONI Pipeline 5.8 (Jul 2, 2013)

  • Pipeline Client / Server:
  • Added: Cloud Storage Module for Amazon S3, Box and Dropbox. The module downloads and uploads files to and from cloud storage services and the Pipeline server, without going to local machine. It lets user link their cloud accounts, browse and select files.
  • Added: NDAR database support. Similar to IDA/XNAT, it creates a downloading module (NDARGet) and downloads input data on server.
  • Added: The ability to have multiple clients (GUI and CLI) connect to the same session (with the same credential).
  • Added: Session (read only) sharing feature for guests. It's a server preference, and requires server admin to specify sharing username. Once enabled, guests can see all workflows by this user, but cannot modify (stop/pause/restart) them.
  • Changed: Server to run less SQL queries wherever found; optimized number of DB queries to improve speed of server restart and module translation.
  • Fixed: Java Swing issue with latest Mac update. It made Pipeline client hang at startup.
  • Fixed: When guest user runs workflows with metadata, metadata files will be properly copied.
  • Fixed: When LONI Viewer (or any local module) completes with with remote modules, the output files for those remote modules no longer disappears.
  • Fixed: To send status update when workflow is stopped, this update will be useful when multiple clients are connecting to the same workflow session.
  • Fixed: To check for expired database connections and replace them with new ones.
  • Fixed: Bug that caused transformation to be applied twice.
  • Fixed: To deregister/remove status update subscription from server for a session when user closes the workflow tab.
  • Fixed: For xml view mode of workflow to preserve viewport position when switching between xml mode and gui mode.
  • Fixed: To auto select study module import rules based on file path.
  • Fixed: For study module import from csv file, metadata will be properly created with guest usernames.
  • Fixed: To properly reset instance status when resubmitting jobs.
  • Fixed: Bug that was causing transformation not to be applied.
  • Fixed: Validation bug when {tempdir} variable is used on data sources of type string.
  • Fixed: To have server's command line file check to check user's group only when it's needed.
  • Fixed: For data sink with metadata files to have link instead of hard copy. Also for data sink to maintain a list of already linked files.
  • Fixed: Server bug when restarting modules, module's output files will be removed.
  • Fixed: Server bug where metadata generator was incorrectly added to command line.
  • Fixed: To save workflow xml data to file instead of db when workflows are reset.
  • Fixed: Server to check only for directories when cleaning up the server.
  • Fixed: Server bug to improve server restart restoration speed.
  • Fixed: Metadata augmentation when the module has cross products and it uses value from input parameters.
  • Fixed: Server to remove not only directories, but also files that are old in the temp file location.
  • Command Line Interface:
  • Fixed: to report missing credentials file, also updated help string.
  • Server Terminal:
  • Added: Server library testing instructions under server terminals, it provides commands to launch CLI to validate and execute workflows in server library.
  • Fixed: Server to not retrieve all sessions from database when administrator requests running sessions only.

New in LONI Pipeline 5.2.2 (Jul 22, 2011)

  • [ Pipeline Client / Server ]
  • Added: for smartline module the ability to edit/change heap size. User can right click and give a number. The sge memory will be updated too accordingly.
  • Added: new functionality to annotations including the ability to expand/collapse one or all annotations and the ability create and resize annotations with images
  • Added: new annotation GUI functionality, workflow toolbar.
  • Added: balloon hints.
  • Added: a reminder popup when user tries to close a workflow that contains local modules/local files waiting to be executed, or user tries to disconnect to the server where they have open workflows with local waiting modules/files.
  • Added: option to preserve local input file names when staging. It is disabled by default so old workflow will have the same behavior (workflows with inputs that have same filename but different paths will continue to work). User can specify under module's advanced options.
  • Changed: to store Staging information in the database instead of memory. This fixes Out of memory exception for server in some cases ( for example when there is a huge local directory assigned to remote module dir type parameter ).
  • Fixed: the bug which was promtping for credentials for workflows with local modules.
  • Fixed: the bug which was trying to add Transformation base's need file to the list of output files
  • Fixed: to include parameter in command line when the transformation base is set but the list of transformations is empty.
  • Fixed: module grouping bug that caused a cyclic connection between a module group and a module.
  • Fixed: bug that left the prefix of a parameter off of the command line if the parameter had no arguments and the 'Include transformed parameter on command line' checkbox was unchecked.
  • Fixed: the bug which was showing Incomplete status for restarting module.
  • Fixed: IDAGet and XNATGet bugs where downloading locally failed because the smartline parameter in the call to IDAGet was not properly specified.
  • Fixed: the bug which was using commas at invalid places for Module tags when hovering on a module/workflow in server library.
  • Fixed: copy with input bugs: when one parent is connecting to multiple child modules, it will be properly copied.
  • [ Server Monitor Terminal ]
  • Added: 'Executables' tab, which behaves much like the Packages panel, but can be used to store package-independent executable information for common tools such as java, python, perl, R; the GUI changes are accompanied by changes to the server side to properly read and write the 'Executable' information from/to userdata.xml
  • Added: to show total number of completed instances for all workflows.