JetS3t Changelog

What's new in JetS3t 0.9.3

Feb 2, 2015

TOOLKIT:
S3 Service - Amazon signature version 4 (AWS4-HMAC-SHA256):
Support Amazon signature version 4 (AWS4-HMAC-SHA256) when signing requests, in particular to permit access to version-4-only region eu-central-1 (#183)
Add some smarts, voodoo, magic, and fairy dust to JetS3t to make version 4 (AWS4-HMAC-SHA256) signatures work in most cases without requiring the user to always know a bucket's S3 region in advance. See in particular the bucket-to-region cache RegionEndpointCache in RestStorageService (#183)
Automatically switch to version 4 signatures when accessing buckets in a region that doesn't support legacy version 2 signatures, namely "eu-central-1" (#183)
Support generating pre-signed URLs using AWS signature version 4 with the new S3Service.createSignedUrlUsingSignatureVersion method (#202)
Fix bug preventing JetS3t applications Sycnronize and Cockpit etc from working with AWSv4 signatures (#203)
General:
Updated BouncyCastle crypto library requirements to latest versions: bcprov-jdk15on-1.51 (#196)
Updated HttpComponents library requirements to latest versions: httpclient 4.3.6; httpcore 4.3.3 (#197)
MultipartUtils.uploadObject now closes file streams on upload failure (#198)
Fix null pointer exception when building XML to update object ACL settings based on an ACL with missing DisplayName (#201)
MAVEN:
Remove problematic config files that were being included in Maven distribution, especially a default logging config, that could break things for some users (#199)

New in JetS3t 0.9.2 (Jul 22, 2014)

New in JetS3t 0.9.1 (May 2, 2014)

TOOLKIT:
General:
Updated library requirements to latest versions: commons-logging 1.1.3; httpclient 4.3.2; httpcore 4.3.1; commons-codec 1.8; java-xmlbuilder 0.6
Storage objects now make fine-grained metadata maps available when possible including: user-specified metadata via #getUserMetadataMap; service-specified metadata via #getServiceMetadataMap; a complete and unfiltered set of metadata via #getCompleteMetadataMap (#171)
Fixed httpclient.read-throttle upload bandwidth throttling implementation which wasn't working for values under 128KB. It should now throttle correctly down to single-digit KB/s values.
Fixed bug in object/file comparison logic that prevented object metadata from being downloaded when a service-side target path was used (e.g. using a bucket path like target-bucket/some/path in Synchronize)
Improved automatic time adjustment feature to use the 'Date' timestamp in a RequestTimeTooSkewed error response to calculate a time offset, instead of performing an extra GET request (#173).
Remove dependency on safehaus UUI library, as UUID is now included in Java 6.
Fix file comparison bugs with metadata lookup and local MD5 hash caching (#159, #161)
S3Service:
Added support to apply canned ACL settings when starting a multipart upload, and to apply arbitrary ACL settings (canned or otherwise) to a multipart object uploaded with the putObjectMaybeAsMultipart convenience method (#184)
Close INPUT form tags in S3 POST upload generated form (#148)
Fix bugs when using AWS server-side encryption
Add API-level support for S3 bucket lifecycle configuration (#158)
Add STORAGECLASSGLACIER storage class.
Apply ACL settings to multipart uploads: canned ACL settings are applied in all cases, non-canned ACLs are applied by #putObjectMaybeAsMultipart (#184)
Fix bug that caused failure of multi-part uploads with service-side encryption.
CloudFrontService:
Added support for setting time-to-live (TTL) values as low as zero seconds.
Support for API version 2012-05-05 (#150)
SimpleThreadedStorageService / ThreadedStorageService:
Added mechanism to selectively permit individual object-level failures within a multi-threaded service operation without causing the whole operation to abort. This is particularly useful for handling missing objects in service during multi-threaded operations without failing the whole operation.
Error conditions are permitted by providing an ErrorPermitter callback handler class, while object-level failures result in generation of a ThrowableBearningStorageObject being generated instead of a normal StorageObject. (#181)
GoogleStorageService:
Google Storage website configuration and other improvements (by dkocher)
SYNCHRONIZE:
Fix bug that prevented Synchronize from deleting service-side objects when files within a pseudo-directory were downloaded to the local system with the --move option.
COCKPIT:
Distribution management dialog now supports viewing and setting the MinTTL value for non-streaming distributions.
UTILITIES:
New console utility to delete outdated incomplete Multipart Upload parts: DeleteMultipartUploads
KUDOS TO:
Chris Baker for catching the httpclient.read-throttle and object/file comparison issues, investigating the bugs and testing the fixes.
David Kocher for a multitude of fixes and improvements, see his pull requests and contributions at https://bitbucket.org/dkocher/jets3t/
Michael Howard (michaelthoward) for impetus and code contributions leading to improved handling of missing objects in multi-threaded operations.
Matteo Bertozzi (matteobertozzi) for fixing bug preventing use of service-side encryption for multipart objects.

New in JetS3t 0.9.0 (Aug 27, 2012)

New in JetS3t 0.8.1 (Apr 11, 2011)

TOOLKIT
General:
StorageObject and its subclasses (e.g. S3Object) now include a convenience constructor that accepts byte array data.
Fixed incorrect file path for XMLBuilder library in shell scripts.
Fixed issue in FileComparer utility that prevented locally-deleted files within directories from being noticed and removed from the remote service. Also affected JetS3t's Synchronize and Cockpit apps. See https://bitbucket.org/jmurty/jets3t/issue/69
Greatly reduced memory usage in FileComparer file comparison utilities when working with large numbers of files.
Service credentials classes now always return ProviderCredentials from #load methods, where the actual credentials instance is AWSCredentials by default or GSCredentials if explicitly loaded.
FileComparer utility now skips/ignores 'special' files on the local file system, to avoid problems synchronizing system directories that contain files that are not really data and cannot be synchronized to a storage service.
S3Service:
Added support for the Multipart Uploads feature, including the utility class MultipartUtils to make it easier to use.
Added support for the Website Configuration feature, which allows static S3 bucket to behave more like a dynamic web site with custom index and error documents.
Added support for configuring buckets to send Simple Notification Service (SNS) messages.
ThreadedS3Service:
Added threaded service subclass of ThreadedStorageService for features specific to Amazon's S3, namely multipart upload.
CloudFrontService:
Added support for custom (non-S3) distribution origins (API 2010-11-01). This improvement involved changes to the CloudFrontService API which are not backwards-compatible, so code that relies on the prior version will need to be updated.
SYNCHRONIZE:
Improved batching algorithm to reduce memory usage when working with large numbers of files.
Added support for the Amazon S3 service's Multipart Upload feature, which allows large files to be uploaded in smaller parts for improved transfer performance or to upload files larger than 5 GB.
Added new property upload.max-part-size to synchronize.properties file, to allow the maximum upload part size to be configured when using the Amazon S3 service. The maximum part size defaults to 5 GB.
Improved file comparison logic so object metadata is only retrieved from a service when it is required.
Removed support for --skipmetadata option. This option is no longer necessary since metadata retrieval is more intelligent, and is not desirable because metadata retrieval is no longer optional when comparing objects uploaded using the Multipart Upload mechanism.
COCKPIT:
Fixed nasty bug that prevented directories from being included when uploading files.

New in JetS3t 0.8.0 (Oct 6, 2010)

General:
New generic bucket, object, and multi-threaded service classes for interacting with either of the S3 or Google Storage services.
No long compatible with JDK 1.4. JDK 6+ is now required to run JetS3t.
Removed antiquated and unsupported SOAP S3 service implementation from toolkit.
FileComparer and related tools are now compatible with both S3 and Google Storage services.
FileComparer tool, and therefore all JetS3t apps, now uses a trailing slash to denote directory placeholder objects in S3 instead of the custom "application/x-directory" content-type. This should improve compatibility with other storage management tools, especially Amazon's AWS Console.
JetS3t Property File changes
Renamed "s3service.internal-error-retry-max" to "storage-service.internal-error-retry-max"
Renamed "s3service.stream-retry-buffer-size" to "uploads.stream-retry-buffer-size"
Renamed "s3service.defaultStorageClass" to "s3service.default-storage-class"
Added properties for new ThreadedStorageService: "threaded-service.max-thread-count" "threaded-service.admin-max-thread-count" "threaded-service.ignore-exceptions-in-multi" * ThreadedStorageService / SimpleThreadedStorageService
New multi-threaded service classes that allow operations to be performed in bulk on either the RestS3Service or GoogleStorageService service implementations.
GoogleStorageService:
Google Storage (GS) service class added to provide specific support for Google's storage provider.
S3Service:
Added support for bucket policy documents: set, get, and delete.
Methods for setting and using AWS DevPay credentials moved from S3Service to RestS3Service.
CloudFront Service:
Added support for setting Default Root Object for CloudFront distributions.
Added support for API version 2010-08-01 and invalidation of objects.
Modified CloudFrontService#signUrl and #signUrlCanned to accept raw resource URL or path strings, instead of expressing all resources as HTTP URLs. This allows for URL signing of HTTPS and RTMP streaming resources.
SOAP implementation:
REMOVED from project.
COCKPIT:
Added explicit support for logging in to either the Google Storage or Amazon S3 service in the startup/login dialog.
A CloudFront distribution's Default Root Object setting can be configured in the "CloudFront Distributions" dialog.
SYNCHRONIZE:
Directly supports new GoogleStorageService, allowing synchronization with Google Storage accounts through the native API.
Choose target service end-point with the new "--provider" command argument.

New in JetS3t 0.7.4 (Jul 20, 2010)

General:
Added support for the new Reduced Redundancy Storage (RRS) class for objects. This discounted storage class can be applied when an object is created, overwritten or copied. The storage class can be set per-object, or a default value can be set with the property 's3service.defaultStorageClass'. Storage classes can be disabled altogether for services that don't support them using the 's3service.enable-storage-classes' property.
Added support for buckets located in the Asia Pacific (Singapore) location "ap-southeast-1".
Eucalyptus-friendly generated URLs (#14). JetS3t configuration properties 's3service.s3-endpoint-virtual-path', 's3service.s3-endpoint-http-port' and 's3service.s3-endpoint-https-port' are no longer ignored when generating signed or torrent URLs.
Bash shell scripts now support cygwin by automatically converting classpaths to Windows format.
Improved configurability and flexibility by removing dependence on the VM-wide static Constants#S3_HOSTNAME constant value (see API Changes below)
CloudFront Service:
NOTE: These changes break backward-compatibility for some of the CloudFrontService and related class interfaces.
Updated service to support API version 2010-06-01 which adds support for HTTPS-only distributions (a.k.a. RequiredProtocols)
Updated service to support API version 2010-05-01 which adds logging support for streaming distributions.
API Changes:
Removed support for methods and metadata attributes that have been obsolete (and deprecated) since version 0.6.0.
Deprecated static methods in S3Service for generating signed URLs. The new non-static method equivalents should be used from now on to avoid dependency on a VM-wide S3 endpoint constant.
SOAP implementation:
Deprecated in preparation for imminent removal from project.
COCKPIT:
Added support for the REDUCED_REDUNDANCY storage class to the Preferences dialog (set default storage class for uploads), the Copy or Move dialog (set the storage class for destination objects), and the Object Attributes dialog (see the current storage class property of objects)
Added support for buckets located in the Asia Pacific (Singapore) location.
Added support for creating HTTPS-only CloudFront distributions.
Removed overly fussy string-length checks in credentials dialog.
SYNCHRONIZE:
Allow synchronization with third-party buckets that are not owned by the user, provided the user has the necessary access to that bucket.

New in JetS3t 0.7.3 (Apr 19, 2010)

New in JetS3t 0.7.0 (Feb 3, 2009)

COCKPIT:
Added ability to view and manage Amazon CloudFront distributions using a dialog box.
Added support for configuring Requester Pays buckets.
Improved support for DevPay credentials, which can now be set and saved directly in the login dialog box instead of only in a properties file.
User preferences can now be remembered on your computer, so they do not need to be re-set every time you start Cockpit.
Improved dialog box for generating Signed URLs. URLs for multiple objects can now be generated at once.
SYNCHRONIZE:
AWS credentials and the cryptographic password can now be provided via prompts on the command-line, rather than merely through a properties file.
Added the --credentials option, which allows AWS credentials to be loaded from an encrypted file rather than an insecure properties file. The encrypted file can be created with the AWSCredentials API or the Cockpit application.
Synchronize will act as an anonymous client if empty values are provided as the AWS access and secret keys. When the S3 connection is anonymous, only public buckets will be accessible.
Fixed a bug that prevented Synchronize from recognizing duplicate file names when uploading to an S3 subdirectory path when using the --batch option.
Synchronize will now prompt for HTTP Proxy login credentials if required. If proxy credentials are pre-specified in the jets3t.properties file, you should not be prompted.
Improved handling of uploads of many files where the files must first be transformed (compressed or encrypted). In this case, files are now transformed and uploaded in batches, where the batch size is set by the synchronize.properties setting 'upload.transformed-files-batch-size'.
TOOLKIT General:
Added initial implementation of CloudFront service.
Added property "cloudfront-service.internal-error-retry-max" for defining the retry limit for CloudFront internal service errors.
Added support for configuring and accessing Requester Pays buckets.
Added property "httpclient.requester-pays-buckets-enabled" for defining whether the RestS3Service supports Requester Pays buckets by default.
Improved support for accessing S3 using DevPay credentials with a new credentials class for defining and storing these credentials: AWSDevPayCredentials
Added a class (AWSDevPayProduct) to represent a DevPay product, and to load information about pre-defined products from the properties file devpay_products.properties.
Improved the interpretation of .jets3t-ignore files so that child paths or arbitrary paths can be specified and ignored. For example, to ignore all CVS directories you can now add a single ignore path at the top level: **/CVS.
TOOLKIT REST Implementation:
Added support for proxy authentication settings in jets3t.properties: "proxy-user", "proxy-password", and "proxy-domain"
The service's HTTP proxy settings can be updated an reapplied on-demand using a range of #initHttpProxy methods.
Added property settings to allow the default HTTP and HTTPS ports to be changed, which can be handy when running the service through a proxy for
testing: s3service.s3-endpoint-http-port, s3service.s3-endpoint-https-port.
The HttpConnectionManager and HttpClient objects used by the REST service can now be reinitialised on-demand using the #initHttpConnection method.
The underlying HttpClient and HttpConnectionManager objects can be accessed to provide greater control and flexibility after a service is created.
The automatic time adjustment performed in response to RequestTimeTooSkewed errors will now work through proxies.

New in JetS3t 0.6.1 (Jan 24, 2009)

COCKPIT:
Added support for copying or moving objects within and between buckets.
Added dialog to assist in moving/renaming multiple objects within a bucket.
Added support for updating objects with new metadata values.
Fixed issue where downloading items to a directory containing many files and directories (eg 250K+) would cause Cockpit to fail with OutOfMemory errors.
Added a Confirm Password field to the encryption settings, to help prevent problems where a password is mistyped.
Cockpit can now generate valid signed GET URLs for DevPay S3 accounts.
SYNCHRONIZE:
Synchonize now recognizes when you have specified a partial S3 path when synchronizing DOWN. A partial path is one that does not exactly match a directory path. Partial paths can now act as a prefix test that identifies objects in an S3 path, where only the objects in S3 that match the prefix are downloaded. For example, in a bucket that has a Docs subdirectory containing objects named with a timestamp prefix, the S3 path "my-bucket/Docs/2008" will identify the objects in the Docs subdirectory with names beginning with 2008.
Added --batch option that causes Synchronize to compare and download/upload files in batches of 1000, rather than all at once. This option will reduce the memory required to synchronize buckets with many objects, and will allow file transfers to commence as soon as possible rather than after the slow comparison process.
Added --skipmetadata option that causes Synchronize to skip the retrieval of object metadata information from S3. This makes synchs much faster for large buckets, but leave the app with less info to make decisions.
Added --reportlevel option that allows the user to control how much report detail is printed, from 0 (no reporting) to 3 (all reporting).
Added --move option that deletes local files after they have been uploaded to S3, or deletes objects from S3 after they have been downloaded.
Property settings in the synchronize.properties file, or in the file referred to by the --properties option, will override properties of the same name in jets3t.properties. This makes it easy to create task-specific properties files.
TOOLKIT:
General:
Added support for the new Copy Object functionality that allows you to copy, move, rename and update objects in your S3 account. These operations are available through the S3Service as the methods copyObject, moveObject, renameObject and updateObjectMetadata.
Added support for conditional copying of objects, based on ETag and Modified Date comparisons.
Made it easier to verify that your data is correctly stored in S3 without corruption during transit, by calculating Content-MD5 hash values by default in commonly-used S3Object constructors and providing S3Object#verifyData methods that make it easy to verify data downloaded from S3.
Added basic support for accessing DevPay S3 accounts. DevPay user and product tokens can be provided to the S3Service class directly using the #setDevPayUserToken and #setDevPayProductToken methods, and default token values can be specified in the jets3t.properties file using the settings "devpay.user-token" and "devpay.product-token".
Modified Bucket Logging Status changing behaviour to update ACL settings of the target bucket before (re)directing log files to the bucket.
Fixed bug in RestS3Service that caused failures in some circumstances when buckets were created with the "US" location.
S3Service instances can now be configured individually by providing a JetS3tProperties object when constructing the service. The property values in this object can also be updated programmatically after the object has been constructed.
FileComparer now supports an optional setting that makes it assume local files are the latest version when there is a clash between the modification dates and hash values of a local file and an object stored in S3. This option should only be used as a work-around for users who synchronize Microsoft Excel files to S3, as these are the only documents that exhibit the mismatch. To enable the option, set the jets3t.properties item "filecomparer.assume-local-latest-in-mismatch" to true.
Added support to FileComparer for listing objects based on prefix "partitions" that allow a bucket to be listed by multiple simultaneous threads. This can speed up listings for large buckets containing many virtual subdirectories. To use multiple partitions when building S3 object maps, configure the jets3t.properties item "filecomparer.bucket-listing." to specify a delimiter string and a traversal depth value separated by a comma, eg: filecomparer.bucket-listing.my-bucket=/,2 to partition the bucket my-bucket based on the '/' delimiter to a depth of 2 virtual subdirectories.
REST Implementation:
Added support for the new Copy Object functionality, and conditional copying.
The XML parsing code now copes better when object names contain a carriage return character. A work-around is applied to prevent the Java XML parser from misinterpreting these characters, a fault which could cause some objects to become un-deletable. This new feature is enabled by default, but can be disabled with the jets3t.properties setting "xmlparser.sanitize-listings".
Changed default AuthenticationPreemptive setting for HttpClient connections from true to false, to improve compatibility with NTLM proxies. Preemptive auth can be turned on by setting "httpclient.authentication-preemptive" to true in jets3t.properties.
Refactored repeatable input streams implementations to recognize and support standard InputStream objects that can be reset. Thanks to Keith Bonawitz for the idea and patch for this.
Multi-threaded Service:
Added a new listObjects method that performs a multi-threaded listing of a bucket's contents. You provide an array of prefix strings which serve to divide your objects into a number of "partitions", and the service performs these prefix-based listings in parallel. See src/org/jets3t/samples/ThreadedObjectListing.java for example usage.
Added a new notification event ServiceEvent#EVENT_IGNORED_ERRORS that provides information about exceptions that were ignored during a multi-threaded S3 operation. Exceptions are only ignored if the JetS3t property "s3service.ignore-exceptions-in-multi" is set to true.