Softpedia >Mac >Developer Tools >Apache Sqoop >  Changelog

Apache Sqoop Changelog

What's new in Apache Sqoop 1.99.6

May 6, 2015
  • SUB-TASK:
  • SQOOP-1528: Sqoop2: Kerberos support on DoAs function
  • SQOOP-1757: Sqoop2: Document generic jdbc connector
  • SQOOP-1961: Sqoop2: addDelegationTokens is not exposed in SqoopClient
  • SQOOP-1978: Sqoop2: Command line support
  • SQOOP-1979: Sqoop2: Rest API support
  • SQOOP-1998: Sqoop2: Kite Connector: Hive support
  • SQOOP-2016: Sqoop2: Create integration test for JDBC to Hive
  • SQOOP-2019: Sqoop2: Handle post-function for updating/deleting resource
  • SQOOP-2079: Sqoop2: Remove useless functions for RBAC
  • SQOOP-2080: Sqoop2: Combine privileges into READ and WRITE
  • SQOOP-2092: Support for LONG in MInputType
  • SQOOP-2094: Sqoop2: Update RESTAPI.rst
  • SQOOP-2101: Sqoop2: Add SqoopClient support to call Restful API
  • SQOOP-2139: Sqoop2: Change create role API from PUT to POST
  • SQOOP-2144: Sqoop2: Show command for CLI
  • SQOOP-2146: Sqoop2: Sqoop client should return model objects for RBAC methods
  • SQOOP-2148: Sqoop2: Fix wrong input sequence of name and type in function new Principal and Resource
  • SQOOP-2160: Sqoop2: Datatypes: Provide foundation for the exhaustive type checks
  • SQOOP-2166: Sqoop2: Add "Authorization Exception" error code in SecurityError.java
  • SQOOP-2183: Sqoop2: Change resource type, privilege action and principal type from String to Enum.
  • SQOOP-2184: Sqoop2: Add ServerName for authorization handler
  • SQOOP-2198: Sqoop2: Datatypes: Add boolean/char/date support to DerbyTypeList
  • SQOOP-2204: Sanitize issues
  • SQOOP-2205: Cut 1.99.6 branch
  • SQOOP-2206: Update change log with 1.99.6 release
  • SQOOP-2207: Update POM with new version
  • SQOOP-2208: Create release candidate
  • SQOOP-2209: Run vote on RC candidates
  • SQOOP-2210: Release artifacts
  • SQOOP-2211: Finish release
  • SQOOP-2212: Update website
  • SQOOP-2220: Sqoop2: Fix the privilege check on job using the link id not the connector id
  • SQOOP-2226: Sqoop2: Access denied in shell
  • SQOOP-2227: Sqoop2: Clear connector cache in ShowConnectorFunction.java
  • SQOOP-2246: Sqoop2: Use jdbcProperties when creating database connection in GenericJDBCExecutor
  • SQOOP-2248: Sqoop2: ErrorCodes: Move ServerError to server module
  • SQOOP-2249: Sqoop2: ErrorCodes: Move SecurityError to core module
  • SQOOP-2251: Sqoop2: ErrorCodes: Move RepositoryError to core module
  • SQOOP-2255: Sqoop2: Throw Exception when no permission to the resource
  • SQOOP-2256: Sqoop2: Creator should have access to its entities
  • SQOOP-2304: Update license file
  • SQOOP-2313: Update change log with 1.99.6 release for RC-1
  • SQOOP-2314: Create release candidate for RC-1
  • SQOOP-2315: Run vote on RC candidates for RC-1
  • BUG:
  • SQOOP-1439: Sqoop2: Refactor validations
  • SQOOP-1516: Sqoop2: Config Input as a Top Level Entity - RepositoryAPI changes
  • SQOOP-1577: Sqoop2: Refactor repository dump and load tools
  • SQOOP-1579: Sqoop2: Data transfer to load into Hive does not work
  • SQOOP-1738: Sqoop2: HDFS Connector : Check for output directory
  • SQOOP-1746: Sqoop2: Mapreduce job is running in DEBUG mode by default
  • SQOOP-1790: Sqoop2: Upgrade configs with name conflicts
  • SQOOP-1808: Add Integration tests to for Incremental read / write between JDBC and HDFS
  • SQOOP-1841: Sqoop2: Upgrade from 1.99.1 to 1.99.4 onwards is broken
  • SQOOP-1868: Sqoop2: Document MySQL Fast Connector
  • SQOOP-1949: Sqoop2: HDFS append only support
  • SQOOP-2036: Sqoop2: JDBC Connector need to send BigDecimal for decimal type in Object array
  • SQOOP-2089: Sqoop2: Clone configuration object instead of overwriting global
  • SQOOP-2095: Sqoop2: Remove duplicate maven dependency org.mockito:mockito-all
  • SQOOP-2102: Binary tarball missing executable bit on sqoop2-* files
  • SQOOP-2106: Upgrade the version of Derby to 6 and Postgres to 1 in sqoop trunk
  • SQOOP-2117: Sqoop2: Shell command 'show link' should show connector name
  • SQOOP-2119: Sqoop2: Remove remaining imports from junit in PostgreSQL repository tests
  • SQOOP-2120: Sqoop2: Test classes using powermock fails badly when compiling with java target version 1.7
  • SQOOP-2122: Sqoop2: Build documentation only in binary packaging profile
  • SQOOP-2123: Sqoop2: Copy shell libraries only in binary package profile
  • SQOOP-2124: Sqoop2: Pre-Commit hook should report how many tests are executed
  • SQOOP-2129: Sqoop2: SQ_SUBMISSION deletes should cascade
  • SQOOP-2140: Sqoop2: Cast exception when try to cast an integer to Avro's Long type
  • SQOOP-2150: Sqoop2: Kite 1.0 don't allow dashes to be used in dataset names
  • SQOOP-2152: Sqoop2: Kite connector always assumes that FixedPoint is long regardless of the configured size
  • SQOOP-2153: Sqoop2: Ensure creation of valid Avro schema names
  • SQOOP-2155: Sqoop2: KiteDatasetExecutor is not cleaning up merged datasets
  • SQOOP-2156: Sqoop2: HdfsUtils.getOutputMapreduceFiles should ignore hidden files
  • SQOOP-2157: Sqoop2: Kite: Add simple integration test for TO direction
  • SQOOP-2158: Sqoop2: Pre-Commit hook is incorrectly counting number of executed tests
  • SQOOP-2159: Sqoop2: Use BeforeSuite/AfterSuite for startHadoop/stopHadoop in integration tests
  • SQOOP-2162: Sqoop2: InputEditable should be optional in JSON API
  • SQOOP-2163: Sqoop2: Remove jackson dependency
  • SQOOP-2167: Sqoop2: Change getUserName to getShortUserName in AuthorizationEngine.java
  • SQOOP-2169: Sqoop2: Kafka integration test bug
  • SQOOP-2171: Sqoop2: Incorrect basic type boxing with JDK1.7
  • SQOOP-2172: Sqoop2: Move ErrorCode's sub-class from sqoop-core to sqoop-common
  • SQOOP-2188: Sqoop2: Failed to merge Kite datasets
  • SQOOP-2190: Sqoop2: Remove hadoop200 profile from test-patch.py
  • SQOOP-2201: Sqoop2: Add possibility to read Hadoop configuration files to HFDS connector
  • SQOOP-2202: Sqoop2: Add validator to check existence of local directory on Sqoop 2 server
  • SQOOP-2213: Sqoop2: toCSVFixedPoint ClassCastException
  • SQOOP-2214: Sqoop2: Add ability to easily iterate over entries stored in Context
  • SQOOP-2215: Sqoop2: Remember all test logs in pre-commit hook
  • SQOOP-2219: Sqoop2: Skip missing input in DirectoryExistsValidator
  • SQOOP-2225: Sqoop2: Shell client clobbers server exceptions
  • SQOOP-2228: Sqoop2: HDFS Connector: Import data to temporary directory before moving them to target directory
  • SQOOP-2229: Sqoop2: Kite connector should handle doubles
  • SQOOP-2230: Sqoop2: Kite connector can use URIBuilder
  • SQOOP-2231: Sqoop2: Kite connector should use authority in link configuration
  • SQOOP-2232: Sqoop2: Test initializer against local filesystem
  • SQOOP-2233: Sqoop2: Move TestHdfsConnector class under test directory
  • SQOOP-2234: Sqoop2: Move DatasetURIValidator to Kite package as it's not generally usable validator
  • SQOOP-2235: Sqoop2: Move PrefixContext back to mapreduce execution engine
  • SQOOP-2236: Sqoop2: Drop outdated comment from SubmissionCallbacks class
  • SQOOP-2238: Sqoop2: Parquet jars missing
  • SQOOP-2239: Sqoop2: Remove Repository.updateSubmission() call from Mapreduce submission engine
  • SQOOP-2240: Sqoop2: Remove Hadoop 1.x example from catalina.properties
  • SQOOP-2241: Sqoop2: PreCommit hook is incorrectly counting number of executed tests
  • SQOOP-2242: Sqoop2: Add support for full table names into our DatabaseProvider infrastructure
  • SQOOP-2265: Sqoop2: Standardize on methods
  • SQOOP-2268: Sqoop2: Fix a typo in ShowLinkFunction
  • SQOOP-2273: Sqoop2: Move authorizationServlet into v1 package
  • SQOOP-2276: MapReduce execution error in Sqoop job
  • SQOOP-2277: Sqoop2: Add annotation for public usage classes in Sqoop Core
  • SQOOP-2278: Sqoop2: Fix bug for all user could read connector
  • SQOOP-2279: Sqoop2: Clean connector cache after grant and revoke statements
  • SQOOP-2307: Sqoop2: Fix a typo for hdfs-connector-config.properties and kafka-connector-config.properties
  • SQOOP-2308: Sqoop2: Fix a typo for hdfs-connector-config.properties and kafka-connector-config.properties
  • SQOOP-2309: Sqoop2: Add to link id in show job --all command
  • SQOOP-2310: Sqoop2: fix a typo for Hadoop conf directory
  • SQOOP-2327: Sqoop2: Change package name from Authorization to authorization
  • IMPROVEMENT:
  • SQOOP-1805: Sqoop2: GenericJdbcConnector: Delta read support
  • SQOOP-2134: Update Kite dependency to 1.0.0
  • SQOOP-2250: Sqoop2: SQ_LINK_INPUT is missing a foreign key constraint
  • TASK:
  • SQOOP-506: Create Sqoop integration test infrastructure.

New in Apache Sqoop 1.99.5 (Feb 26, 2015)

  • SUB-TASK:
  • SQOOP-1348: Sqoop2: Remove Data class
  • SQOOP-1349: Sqoop2: Use configurable writable to get Intermediate Data Format
  • SQOOP-1526: Sqoop2: Kerberos support when starting service
  • SQOOP-1527: Sqoop2: Kerberos support (SPNEGO) in communication between server and client
  • SQOOP-1549: Simplifying the Configuration class concept in Connector api
  • SQOOP-1552: Revisit if we need a config list for the link/from/to configs in connector api
  • SQOOP-1588: Sqoop2: Kite connector write data to HDFS
  • SQOOP-1589: Sqoop2: Create common constants, error codes, and queries
  • SQOOP-1590: Sqoop2: PostgreSQL repository implementation
  • SQOOP-1591: Sqoop2: PostgreSQL integration tests
  • SQOOP-1592: Sqoop2: Remove Hadoop1 profile
  • SQOOP-1602: Sqoop2: Fix the current balancing across Loaders internal to Sqoop
  • SQOOP-1621: Sqoop2: Allow null as a dummy Schema
  • SQOOP-1624: Sqoop2: Apply repository upgrader api
  • SQOOP-1647: Sqoop2: Read data from HDFS in KiteConnector
  • SQOOP-1651: Sqoop2: Cleanup: Naming and ordering of methods in Repo relates classes
  • SQOOP-1681: DOC: Create an API doc for the Sqoop Execution engine ( since its extensible)
  • SQOOP-1702: Fix typo for SQOOP-1526
  • SQOOP-1703: Sqoop2: rename Type to Column Type + size to length + minor doc fixes
  • SQOOP-1706: Add IDF API doc/wiki for the IDF interface and Schema -> ColumnTypes
  • SQOOP-1707: SQOOP2: Address the validate method in Column class
  • SQOOP-1708: Rename Unsupported Column type to Unknown and add java doc
  • SQOOP-1709: Column Type enhancements for complex types
  • SQOOP-1715: Make name for column required ( fix the corr tests)
  • SQOOP-1716: Add Options as a field in the Enum object ( so it can be used for validation)
  • SQOOP-1717: Sqoop2: Remove Data class from docs
  • SQOOP-1733: Port SQOOP-1728 to sqoop2 branch
  • SQOOP-1734: Port SQOOP-1725 to sqoop2 branch
  • SQOOP-1741: Port SQOOP-1736 to sqoop2 branch
  • SQOOP-1749: Support List Type in CSV IDF
  • SQOOP-1750: Support Map Type in CSV IDF
  • SQOOP-1751: Sqoop2: Rearrange LinkConfig and ToJobConfig of Kite Connector
  • SQOOP-1752: Kite connector cannot submit CSV import job
  • SQOOP-1754: Sqoop2: Remove JarUtil class
  • SQOOP-1755: Sqoop2: Security guide
  • SQOOP-1761: Sqoop2: Unit tests for different Column sub classes Array/Set and Map types
  • SQOOP-1765: Sqoop2: Time/Timestamp format support for CSV IDF
  • SQOOP-1769: Fix Enum to no inherit from list
  • SQOOP-1771: Investigation CSV IDF FORMAT of the Array/NestedArray/ Set/ Map in Postgres and HIVE.
  • SQOOP-1776: Sqoop2: Delegation Token support for Authentication
  • SQOOP-1791: Sqoop2: Add simple authentication parameters to documentation
  • SQOOP-1796: Support Incremental Semantics in command line tool for Job
  • SQOOP-1797: Support Incremental semantics in the REST apis
  • SQOOP-1798: Support a API/ command line utility to retrieve the last value or any JOB_OUTPUT
  • SQOOP-1799: Connector API : Ability for connector to indicate if its FROM and TO support incremental reading/ writing
  • SQOOP-1801: Delta Fetch Merge Config API : Validations for the Delta Fetch/Merge Configs
  • SQOOP-1804: Add editable and override attribute to inputs
  • SQOOP-1811: Sqoop2: IDF API changes
  • SQOOP-1813: Sqoop2: Add SqoopIDFUtils class and unit tests
  • SQOOP-1815: Sqoop2: Date and DateTime is not encoded in Single Quotes
  • SQOOP-1817: Sqoop2: Update CSVIntermediate BIT data type
  • SQOOP-1819: Support Enum in CSVIDF ( + add unit tests)
  • SQOOP-1825: Sqoop2: Handle NULLs for all types in CSV Intermediate Data Format
  • SQOOP-1829: Sqoop2: Define IDF object model
  • SQOOP-1834: Sqoop2: RBAC pluggable framework
  • SQOOP-1842: Sqoop2: Fix timestamp type casting
  • SQOOP-1845: Sqoop2: Make DateTime Column type support datetime with and without timezone
  • SQOOP-1846: Sqoop2: DateTime support in CSV IDF and iso8601
  • SQOOP-1849: Using JODA for datetime means we only have 3 digit millisecond representation for fraction
  • SQOOP-1870: Sqoop2: Merge SPI and connector-sdk packages
  • SQOOP-1876: Update the submission JSON in the doc (restAPI.rst)
  • SQOOP-1879: Sqoop2: Submission Engine does not set all details on SubmissionRecord in Local mode
  • SQOOP-1881: Randomize ports for Kafka integration tests
  • SQOOP-1897: Sqoop2: Submission Engine API change for better performance
  • SQOOP-1898: Change the API of getJars to use SET than LIST to avoid duplicate jars
  • SQOOP-1899: Minor change to CSVIDF to use Set instead of Lists
  • SQOOP-1900: Fix the SqoopWritable read/ write method to delegate the toIDF methods
  • SQOOP-1901: Sqoop2: Support DRY code in IDF impementations and add JSONIDF
  • SQOOP-1902: Sqoop2: Avro IDF class and unit tests
  • SQOOP-1903: Wrap up the design doc once implementation is done
  • SQOOP-1908: Sqoop2: Document external connector support
  • SQOOP-1909: Sqoop2: Connect security guide to main docs
  • SQOOP-1913: Sqoop2: Add jackson as a dependency
  • SQOOP-1915: sqoop2: parseCSV in CSVIDF doesn't correctly parse strings with commas
  • SQOOP-1925: Sqoop2: Clean up duplicate constants in DerbySchemaConstants, use CommonRepoConstants
  • SQOOP-1930: Sqoop2: Enforce a non empty schema name and column names
  • SQOOP-1931: Tech debt : Alternative to isTest flag in the SqoopOutputFormatExecutor
  • SQOOP-1935: Sqoop2: Fix TestSqoopWritable test and make getString and setString package private
  • SQOOP-1936: Sqoop2: Sort by comparing IDF data in shuffle phase
  • SQOOP-1937: Why need SqoopWritable, ? why not just Text?
  • SQOOP-1938: DOC:update the sqoop MR engine implementation details
  • SQOOP-1940: Add hashcode and equals methods to SqoopWritable
  • SQOOP-1950: Sqoop2: Use TestNG
  • SQOOP-1956: Sqoop2: Cleanup IDF implementations
  • SQOOP-1960: Sqoop2: Share auth token across ResourceRequest in SqoopClient
  • SQOOP-1962: Sqoop2: Start sqoop2 minicluster on random port
  • SQOOP-1974: Sqoop2: parseCSVString in SQOOPIDFUtils add more tests
  • SQOOP-1980: Sqoop2: Rule based rest API protection
  • SQOOP-1981: Sqoop2: Default implementation of RBAC in Sqoop
  • SQOOP-1982: Sqoop2: Provide username globally via AuthenticationProvider
  • SQOOP-1984: Sqoop2: Add error summary and details to PostgreSQL repository
  • SQOOP-1985: Sqoop2: On job delete cascade to submission
  • SQOOP-1988: Sqoop2: isNull handling should be moved to CSVIntermediateDataFormat
  • SQOOP-1991: Sqoop2: Define slow category in integration tests
  • SQOOP-1994: Sqoop2: Use common repository constants
  • SQOOP-2002: Sqoop2: Refactor existing security component
  • SQOOP-2004: Sqoop2: Make annotations run always
  • SQOOP-2012: DOC:Explain the connector project structure and relevant modules in sqoop to depend on
  • SQOOP-2013: What is the story for External Connectors to write unit and integration tests?
  • SQOOP-2015: Sqoop2: Create HBase code path in Kite connector
  • SQOOP-2026: Sqoop2: Make getUserName function in RequestContext support Kerberos
  • SQOOP-2028: add license header to TestConnectorManagerUtils
  • SQOOP-2064: Sqoop2: Update POM version on new 1.99.5 branch
  • SQOOP-2072: Sqoop2: Add editable and override attributes for postgres
  • SQOOP-2074: Sqoop2: Add input attributes documentation
  • SQOOP-2084: Sqoop2: Fix Indentation in input attributes documentation
  • SQOOP-2085: Sqoop2: Null handling in AuthorizationEngine
  • SQOOP-2096: Sqoop2: Fix typos in comments in SecurityConstants
  • BUG:
  • SQOOP-1452: Sqoop2: Upgrade to Hadoop 2.5.1
  • SQOOP-1523: Sqoop2: Support for PostgreSQL database for hosting the sqoop entity repository
  • SQOOP-1572: Sqoop2: Duplicate Column Name in Multiple Tables Import
  • SQOOP-1604: Base/ Marker class for Config and Configuration
  • SQOOP-1636: Sqoop2: Add unique constraint to the SQI_INPUT name per type per config
  • SQOOP-1640: Sqoop2: Pull DatabaseProvider into sqoop-common-test module
  • SQOOP-1672: Sqoop2 import inserting null strings('null') for empty values in Oracle
  • SQOOP-1680: DOC: Create an Public API doc for the Sqoop repository.
  • SQOOP-1695: Sqoop2: Removed hardcoded version in pom.xml of connector-sdk
  • SQOOP-1705: SQOOP2: Support AVRO as a an implementation for the IDF interface
  • SQOOP-1713: Sqoop2: Remove SQOOP-1348.patch file
  • SQOOP-1730: Sqoop2: httpcore and httpclient version has not upgrade after hadoop upgraded
  • SQOOP-1740: Sqoop2:The sqoop.properties should say driver.autoupgrade
  • SQOOP-1747: Sqoop2: Provide more information for Kerberos initialization exception
  • SQOOP-1748: Sqoop2: Client throws NPE when misconfigured for Kerberos
  • SQOOP-1773: Sqoop2: ClassAvailable validator null handling
  • SQOOP-1774: Sqoop2: Allow anonymous authentication
  • SQOOP-1775: Sqoop2: Migrate to URL parameter user.name instead of using header
  • SQOOP-1777: Sqoop2: Add integration tests to Pre-Commit build
  • SQOOP-1778: Sqoop2: Increase default memory allocations for tests
  • SQOOP-1782: Sqoop2: Get submissions per job
  • SQOOP-1783: Sqoop2: Create derby integration upgrade tests
  • SQOOP-1784: Sqoop2: Sending POST request to link or job end-point might end up with NPE
  • SQOOP-1785: Sqoop2: Send driver validation results back to client when creating job
  • SQOOP-1786: Sqoop2: Stop using JSONValue.parse method call
  • SQOOP-1787: Sqoop2: Fix dual guava dependency warnings in maven
  • SQOOP-1788: Sqoop2: Add missing version for build-helper-maven-plugin
  • SQOOP-1789: Sqoop2: Add validation for partition column when using query based import
  • SQOOP-1792: Sqoop2: Default to simple authentication type
  • SQOOP-1806: Sqoop2: Remove the duplicate connector-config.properties file in Kite connector
  • SQOOP-1812: Sqoop2: Repository upgrade issue will prevent server startup
  • SQOOP-1816: Sqoop2: Add logging to the test DerbyProvider implementation
  • SQOOP-1818: Sqoop2: Allow transfer between 2 schema less data sources
  • SQOOP-1821: Sqoop2: External connector loading
  • SQOOP-1823: Sqoop2: Fix Derby schema javadocs
  • SQOOP-1827: Sqoop2: Add pattern SQOOP-XYZ-vX.patch to our pre-commit hook
  • SQOOP-1830: GenericJdBcExtractor does not create java date objects when extracting
  • SQOOP-1831: Sqoop2: Log filenames of detected Mapreduce files in HdfsUtils
  • SQOOP-1832: Sqoop2: Enforce that integration test working directory is always in target/ directory
  • SQOOP-1833: Sqoop2: Add elapsed time information to pre-commit hook
  • SQOOP-1836: Sqoop2: Convert the List in Schema Columns to Array for better lookup time
  • SQOOP-1837: Sqoop2: Verify that executing job has been successful in ConnectorTestCase
  • SQOOP-1838: Sqoop2: While running Integration tests in IDE ( we get Derby Start Error)
  • SQOOP-1839: Sqoop2: Mapreduce submission record should propogate failure info as well
  • SQOOP-1840: Sqoop2: Guard test DerbyProvider from infinite waits in Derby implementation
  • SQOOP-1843: Sqoop2: Fix anonymous configuration bug
  • SQOOP-1844: Sqoop2: Start Derby server in DerbyProvider on random port
  • SQOOP-1847: Sqoop2: Move rat configuration from special execution to the main execution
  • SQOOP-1850: Sqoop2: DerbyProvider retry ping
  • SQOOP-1857: Sqoop2: Add rat check to pre-commit hook
  • SQOOP-1858: Sqoop2: Print out warnings from git apply on output of precommit hook
  • SQOOP-1861: Sqoop2: ResourceRequest clobbers uncaught server exceptions
  • SQOOP-1862: Sqoop2: JDBC Connector To side needs to handle converting JODA objects to sql date
  • SQOOP-1871: Sqoop2: Upgrade to Hadoop 2.6.0
  • SQOOP-1872: IDF API should expose a method to add dependent jars
  • SQOOP-1874: Sqoop2: ConfigurableError should be in the configurable package
  • SQOOP-1875: Sqoop2: Sync MSubmission with repository and add tests for the max limit handling for submission fields
  • SQOOP-1882: JobManager currently ignores the TO connector IDF and assumed all IDF use String for the generic T
  • SQOOP-1885: Sqoop2: KafkaConnector pom.xml needs license header
  • SQOOP-1893: add license header to TestSqoopIDFUtils.
  • SQOOP-1894: Sqoop2: Print out files that failed license check in pre-commit hook
  • SQOOP-1895: Sqoop2: Add links to output reports to pre-commit hook
  • SQOOP-1896: Upgrade the Derby repo version to 5 for 1.99.5 release
  • SQOOP-1910: Sqoop2: Add test categories
  • SQOOP-1911: Sqoop2: Derby1_99_4UpgradeTest is constantly failing on pre-commit hook
  • SQOOP-1917: Sqoop2: "show connector --all" fails
  • SQOOP-1927: Sqoop2: Move CSVURIValidator into validators package
  • SQOOP-1929: Sqoop2: Track number of records written in Loader
  • SQOOP-1934: SqoopWritable and MR engine documentation
  • SQOOP-1943: Sqoop2: PreCommit should continue if license check fails
  • SQOOP-1945: Sqoop2: Add table escaping to JDBC Repository
  • SQOOP-1963: Generated files such as .project and .classpath files are included in source distribution
  • SQOOP-1971: Sqoop2: strURL may not be initialized when calling addDelegationToken
  • SQOOP-1972: Sqoop2: PreCommit hook have incorrect link for test reports
  • SQOOP-1973: Sqoop2: Remove maven module spi completely after merging it with connector-sdk
  • SQOOP-1977: Sqoop2: Pre-commit hook does not work for Python 2.6 environment
  • SQOOP-1995: Sqoop2: Allow nulls only if the column for that field has IsNullable to be true
  • SQOOP-1996: Sqoop2: Change getNullable to isNullable
  • SQOOP-2018: GenericJDBCConnector SqlTypes INTEGER should be set 4 bytes
  • SQOOP-2021: Sqoop2: Generic JDBC Connector extractor uses wrong nulls
  • SQOOP-2022: Sqoop2: SqoopIDFUtils uses bit size instead of byteSize to check for INT/LONG/FLOAT/Double
  • SQOOP-2023: GenericJDBCConnector Handling unsigned Ints is broken
  • SQOOP-2027: Sqoop2: SqoopIDFUtils handling of decimal need to be fixed
  • SQOOP-2029: Sqoop2: Get jobs by connector
  • SQOOP-2030: Should we allow creating a job with both from and to to id been the same
  • SQOOP-2032: Command line exception message says class not found
  • SQOOP-2034: Sqoop2: Clean up docstring and names in repository API
  • SQOOP-2041: Sqoop2: Escape identifier names
  • SQOOP-2043: Sqoop2: Server errors don't look informative enough in the client
  • SQOOP-2054: Sqoop2: Client cannot understand server exceptions
  • SQOOP-2077: Sqoop2: Enclose missing/extra lines in output of HdfsAsserts.assertMapreduceOutput
  • SQOOP-2078: Sqoop2: Add ParametrizedUtils.toArrayOfArrays utils method
  • SQOOP-2081: Sqoop2: Provide ability to dump content of testing table to log
  • SQOOP-2086: Sqoop2: Use named constraints
  • SQOOP-2087: Sqoop2: Named constraints should be quoted
  • SQOOP-2088: Sqoop2: Rename constants for named constraints to match convention
  • SQOOP-2099: Sqoop2: Postgresql's detectRepositoryVersion uses wrong casing
  • SQOOP-2100: Sqoop2: Connectors documentation is malformed
  • SQOOP-2109: Sqoop2: Shell module is including test dependencies when building binary artifact
  • SQOOP-2116: Sqoop2: RequestContext should use user short name
  • IMPROVEMENT:
  • SQOOP-1678: Sqoop2: [HDFS Connector] Configurable null values
  • SQOOP-1727: Sqoop2: Log server initialization errors to log4j
  • SQOOP-1767: Sqoop2: commands in 5 minute demo docs are out-of-sync
  • SQOOP-1869: Sqoop2: Expand schema matching to support two schemaless connectors
  • SQOOP-1944: Sqoop2: No (or wrong) log message when Derby server startup/shutdown unexpectedly
  • SQOOP-1966: Turn off maven's source distribution profile by default
  • SQOOP-1968: Optimize schema operation in getMatchingData of NameMatcher
  • SQOOP-1969: Support getColumnSize method for Schema class
  • SQOOP-2006: bad smell: NPE check in SQOOP-1995 is not obvious
  • NEW FEATURE:
  • SQOOP-1763: Sqoop2: Role Based Access Control
  • SQOOP-1926: Sqoop2: Add annotations for visibilty and compatibility guarantees
  • TASK:
  • SQOOP-1350: Sqoop2: Support all supported data types in the CSV Intermediate Data Format implementation
  • SQOOP-1525: Kerberos support on Sqoop 2
  • SQOOP-2033: Technical Guide of the Sqoop Server

New in Apache Sqoop 1.4.5 (Aug 12, 2014)

  • Sub-task:
  • [SQOOP-1194]: Make changes to Sqoop build file to enable Netezza third party tests
  • [SQOOP-1323]: Update HCatalog version to 0.13 in Sqoop builds
  • [SQOOP-1324]: Support new hive datatypes in Sqoop hcatalog integration
  • [SQOOP-1325]: Make hcatalog object names escaped during creation so that reserved words are properly processed
  • [SQOOP-1326]: Support multiple static partition keys for better integration support
  • [SQOOP-1363]: Document Hcatalog integration enhancements introduced in SQOOP-1322
  • Bug:
  • [SQOOP-585]: Bug when sqoop a join of two tables with the same column name with mysql backend
  • [SQOOP-832]: Document --columns argument usage in export tool
  • [SQOOP-1032]: Add the --bulk-load-dir option to support the HBase doBulkLoad function
  • [SQOOP-1107]: Further improve error reporting when exporting malformed data
  • [SQOOP-1117]: when failed to import a non-existing table, the failure information includes NullPointerException
  • [SQOOP-1138]: incremental lastmodified should re-use output directory
  • [SQOOP-1167]: Enhance HCatalog support to allow direct mode connection manager implementations
  • [SQOOP-1170]: Can't import columns with name "public"
  • [SQOOP-1179]: Incorrect warning saying --hive-import was not specified when it was specified
  • [SQOOP-1185]: LobAvroImportTestCase is sensitive to test method order execution
  • [SQOOP-1190]: Class HCatHadoopShims will be removed in HCatalog 0.12
  • [SQOOP-1192]: Add option "--skip-dist-cache" to allow Sqoop not copying jars in %SQOOP_HOME%\lib folder when launched by Oozie and use Oozie share lib
  • [SQOOP-1209]: DirectNetezzaManager fails to find tables from older Netezza system catalogs
  • [SQOOP-1216]: Improve error message on corrupted input while doing export
  • [SQOOP-1224]: Enable use of Oracle Wallets with Oracle Manager
  • [SQOOP-1226]: --password-file option triggers FileSystemClosed exception at end of Oozie action
  • [SQOOP-1227]: Sqoop fails to compile against commons-io higher then 1.4
  • [SQOOP-1228]: Method Configuration#unset is not available on Hadoop < 1.2.0
  • [SQOOP-1239]: Sqoop import code too large error
  • [SQOOP-1246]: HBaseImportJob should add job authtoken only if HBase is secured
  • [SQOOP-1249]: Sqoop HCatalog Import fails with -queries because of validation issues
  • [SQOOP-1250]: Oracle connector is not disabling autoCommit on created connections
  • [SQOOP-1259]: Sqoop on Windows can't run HCatalog/HBase multinode jobs
  • [SQOOP-1260]: HADOOP_MAPRED_HOME should be defaulted correctly
  • [SQOOP-1261]: CompilationManager should add Hadoop 2.x libraries to the classpath under Hadoop 2.x
  • [SQOOP-1268]: Sqoop tarballs do not contain .gitignore and .gitattribute files
  • [SQOOP-1271]: Sqoop hcatalog location should support older bigtop default location also
  • [SQOOP-1273]: Multiple append jobs can easily end up sharing directories
  • [SQOOP-1278]: Allow use of uncommitted isolation for databases that support it as an import option
  • [SQOOP-1279]: Sqoop connection resiliency option breaks older Mysql versions that don't have JDBC 4 methods
  • [SQOOP-1297]: Parameterize the Accumulo version in the build files
  • [SQOOP-1298]: Cannot export to VARBINARY with null value
  • [SQOOP-1302]: Doesn't run the mapper for remaining splits, when split-by ROWNUM
  • [SQOOP-1303]: Can only write to default file system on incremental import
  • [SQOOP-1316]: Example for use of password file in docs is incorrect
  • [SQOOP-1322]: Enhance Sqoop HCatalog Integration to cover features introduced in newer Hive versions
  • [SQOOP-1329]: JDBC connection to Oracle timeout after data import but before hive metadata import
  • [SQOOP-1339]: Synchronize .gitignore files
  • [SQOOP-1353]: Sqoop 1.4.5 release preparation
  • [SQOOP-1358]: Add wallet support for Oracle High performance connector
  • [SQOOP-1359]: Fix avro versions in Sqoop to stop shipping hadoop1 jars with hadoop2
  • [SQOOP-1362]: TestImportJob getContent method doesn't work
  • [SQOOP-1365]: Do not print stack trace when we can't move generated .java file to CWD
  • [SQOOP-1370]: AccumuloUtils can throw NPE when zookeeper or accumulo home is null
  • [SQOOP-1372]: configure-sqoop does not export ZOOKEEPER_HOME
  • [SQOOP-1398]: Upgrade ivy version used to the latest release version
  • [SQOOP-1399]: Fix TestOraOopJdbcUrl test case
  • [SQOOP-1406]: Add license headers
  • [SQOOP-1410]: Update change log for 1.4.5
  • Improvement:
  • [SQOOP-435]: Avro import should write the Schema to a file
  • [SQOOP-1056]: Implement connection resiliency in Sqoop using pluggable failure handlers
  • [SQOOP-1132]: Print out Sqoop version into log during execution
  • [SQOOP-1137]: Put a stress in the user guide that eval tool is meant for evaluation purpose only
  • [SQOOP-1161]: Generated Delimiter Set Field Should be Static
  • [SQOOP-1172]: Make Sqoop compatible with HBase 0.95+
  • [SQOOP-1203]: Add another default case for finding *_HOME when not explicitly defined
  • [SQOOP-1212]: Do not print usage on wrong command line
  • [SQOOP-1213]: Support reading password files from Amazon S3
  • [SQOOP-1223]: Enhance the password file capability to enable plugging-in custom loaders
  • [SQOOP-1282]: Consider avro files even if they carry no extension
  • [SQOOP-1321]: Add ability to serialize SqoopOption into JobConf
  • [SQOOP-1337]: Doc refactoring: Consolidate documentation of --direct
  • [SQOOP-1341]: Sqoop Export Upsert for MySQL lacks batch support
  • [SQOOP-1373]: Sqoop import schema is locked shows NullPointerException
  • New Feature:
  • [SQOOP-767]: Add support for Accumulo
  • [SQOOP-1051]: Support direct mode connection managers in a generalized fashion
  • [SQOOP-1197]: Enable Sqoop to build against Hadoop-2.1.0-beta jar files
  • [SQOOP-1287]: Add high performance Oracle connector into Sqoop
  • Task:
  • [SQOOP-1207]: Allow user to override java source version
  • [SQOOP-1344]: Add documentation for Oracle connector
  • [SQOOP-1408]: Document SQL Server's --non-resilient arg
  • Test:
  • [SQOOP-1057]: Introduce fault injection framework to test connection resiliency

New in Apache Sqoop 1.4.4 (Jul 31, 2013)

  • New Feature:
  • [SQOOP-931] - Integrate HCatalog with Sqoop
  • [SQOOP-999] - Support bulk load from HDFS to PostgreSQL using COPY ... FROM
  • [SQOOP-1038] - Add support for composite keys in HBase import
  • Improvement:
  • [SQOOP-885] - Allow excluding some tables from import-all-tables tools
  • [SQOOP-912] - Add ability to specify --hive-database parameter
  • [SQOOP-914] - Securing passwords in sqoop 1.x
  • [SQOOP-916] - Add an abort validation handler
  • [SQOOP-933] - Properly log SQLExceptions using getNextException() method for chaining exceptions
  • [SQOOP-934] - JDBC Connection can timeout after import but before hive import
  • [SQOOP-937] - Dont generate ORM files for Direct mode connectors
  • [SQOOP-1140] - Enable overriding username, password and database in DB2 test
  • Task:
  • [SQOOP-1010] - Change default Hadoop profile to 2.x
  • [SQOOP-1118] - Move PostgreSQL specific MR codes to org.apache.sqoop.mapreduce.posgresql
  • Sub-task:
  • [SQOOP-838] - Stored procedures: MySQL
  • [SQOOP-839] - Stored procedures: Oracle
  • [SQOOP-847] - Provide direct support
  • [SQOOP-848] - Unit tests for Netezza connector
  • [SQOOP-849] - documentation for netezza connector
  • [SQOOP-949] - Allow Sqoop to build on Windows OS
  • [SQOOP-954] - Create Sqoop runtime scripts to run Sqoop on Windows
  • [SQOOP-955] - Fix Sqoop unit test failures on Windows
  • [SQOOP-962] - Document null string behavior
  • [SQOOP-970] - Explicit delimiter configuration information should be available for exports also
  • [SQOOP-1011] - Allow automatic creation of HCatalog tables
  • [SQOOP-1020] - Fix Hive Unit tests to work in the presence of Hive classes
  • [SQOOP-1050] - Add documentation for HCatalog Integration
  • [SQOOP-1127] - Some database specific types are not properly handled during exports
  • Bug:
  • [SQOOP-846] - Provide a connector for Netezza appliances
  • [SQOOP-856] - PGBulkLoadExportMapper has an infinite waitFor without any logging in the event of Interrupted exceptions
  • [SQOOP-859] - PostgreSQL: Test cases have hard coded username and password
  • [SQOOP-883] - Remove input directory prior Sqoop import
  • [SQOOP-891] - Sqoop export from S3 to MySQL fails when S3 is not default filesystem.
  • [SQOOP-906] - Sqoop is always calling ConnectionManager.datetimeToQueryString with TIMESTAMP column type
  • [SQOOP-907] - Print out queries executed by Sqoop on mapreduce job side
  • [SQOOP-928] - PostgresqlManager should rollback any pending changes instead of commit when closing connections
  • [SQOOP-929] - Add more Netezza direct mode tests
  • [SQOOP-932] - Netezza direct manager is ignoring --null-*-string arguments
  • [SQOOP-936] - Classwriter should check if columnNames generated are null before accessing them
  • [SQOOP-950] - Can't build package from provided binary artifacts
  • [SQOOP-959] - Use CredentialsUtil.fetchPasswordFromFile in BaseSqoopTool
  • [SQOOP-968] - Cannot build when in a directory with spaces
  • [SQOOP-976] - Incorrect SQL when incremental criteria is text column
  • [SQOOP-979] - MySQL direct connector is not working correct after moving password to credential cache
  • [SQOOP-981] - The --hadoop-home command line parameter is no longer accepted
  • [SQOOP-988] - Validation option should provide some output that validation is being done
  • [SQOOP-992] - Allow to set target java version during compilation
  • [SQOOP-1006] - Update year in the notice file to 2013
  • [SQOOP-1008] - Test TestClassWriter.testNoClassGeneration is failing on JDK7
  • [SQOOP-1027] - Incremental import is not properly escaping table name when loading maximal value
  • [SQOOP-1029] - Oracle export using procedures does not work
  • [SQOOP-1030] - Enable column types Binary and Varbinary parsing in Sqoop for export
  • [SQOOP-1031] - Provide default Java type mapping for MS Sql Datetimeoffset type
  • [SQOOP-1033] - CombineFileInputFormat does not work with paths not on default FS like ASV
  • [SQOOP-1034] - Fix SQL Azure database may return table columns in inconsistent order
  • [SQOOP-1052] - Hive import is not compatible with Avro and SequenceFile but no errors are thrown
  • [SQOOP-1053] - Fix sqoop to work with Hadoop 2.0.4 alpha
  • [SQOOP-1071] - Avro import is failing when columns are starting with underscore
  • [SQOOP-1078] - incremental import from database in direct mode
  • [SQOOP-1083] - Split HBaseImportTest into multiple independent tests
  • [SQOOP-1097] - Export using procedures feature needs fixes for working with mysql
  • [SQOOP-1124] - Direct export to PostgreSQL with PGBulkloadManager fails without explicit port number settings
  • [SQOOP-1129] - PostgreSQL direct test are failing on trunk
  • Test:
  • [SQOOP-1035] - Add MS Sqoop Connecter tests

New in Apache Sqoop 1.4.3 (Apr 16, 2013)

  • New Feature:
  • [SQOOP-390] - PostgreSQL connector for direct export with pg_bulkload
  • [SQOOP-540] - Microsoft SQL Connector doesn't support custom schemas
  • [SQOOP-601] - Support custom schemas in PostgreSQL Connector
  • [SQOOP-621] - Requesting support for upsert export with MySQL
  • [SQOOP-724] - Support Table hints in Microsoft SQL Server
  • [SQOOP-749] - Exports Using Stored Procedures (Functions)
  • [SQOOP-767] - Add support for Accumulo
  • Improvement:
  • [SQOOP-350] - Add support for requiring that a connector be used, otherwise the job should fail
  • [SQOOP-467] - Add support for Oracle BINARY_FLOAT and BINARY_DOUBLE data types
  • [SQOOP-579] - Simple refactoring of org.apache.sqoop.manager.DefaultManagerFactory - extract methods for reuse
  • [SQOOP-580] - Add Open-ended "teardown" routine that is called after the job execution for cleanup
  • [SQOOP-598] - Document NULL substitution string changes for HIVE import
  • [SQOOP-603] - Support small intervals in IntegerSplitter implementation
  • [SQOOP-604] - Easy throttling feature for MySQL exports
  • [SQOOP-635] - Explaining why it's important to specify weird urandom path in Troubleshooting guide
  • [SQOOP-638] - Add an optional, simple and extensible validation framework for sqoop
  • [SQOOP-650] - Add ability to name mapreduce job generated by Sqoop
  • [SQOOP-653] - Add splitter support for N(LONG)(VAR)CHAR Sql type
  • [SQOOP-720] - Improve error handling when exporting malformed text data
  • [SQOOP-726] - Put debugging information into generated SqoopRecord class
  • [SQOOP-804] - Warn if hive special arguments will be used without --hive-import
  • [SQOOP-820] - Escape table name in export job only if it's required by connector
  • [SQOOP-833] - add *.eml "ant eclipse" (IntelliJ import) generated file, to .gitignore
  • Task:
  • [SQOOP-535] - Support splitting metadata to connector and framework specific
  • [SQOOP-731] - Move HBase tests to manual tests as they depend on unreleased artifacts
  • Sub-task:
  • [SQOOP-592] - Change repository location on Sqoop website
  • [SQOOP-595] - Change repository location in source code
  • [SQOOP-683] - Documenting sqoop.mysql.export.sleep.ms - easy throttling feature for direct MySQL exports
  • Bug:
  • [SQOOP-382] - Connection parameters should be used on the mapper
  • [SQOOP-437] - Preparing release requires manual changes to LICENSE.txt file
  • [SQOOP-462] - Sqoop hbase Test Compilation failure.
  • [SQOOP-526] - Unable to resolve maven-scm-providers-standard
  • [SQOOP-578] - sqoop script calls com.cloudera.sqoop.Sqoop instead of org.apache.sqoop.Sqoop
  • [SQOOP-582] - Create a template method for job submission in Export/Import JobBase
  • [SQOOP-585] - Bug when sqoop a join of two tables with the same column name with mysql backend
  • [SQOOP-586] - ant checkstyle has 2 errors
  • [SQOOP-597] - PostgreSQL Manager is incorrectly escaping table names
  • [SQOOP-599] - Import to HBase is not working on secure cluster
  • [SQOOP-600] - HBase table family presence is not working correctly on new HBase versions
  • [SQOOP-607] - In-process Hive CLI driver does not work
  • [SQOOP-609] - 3rd party LobAvroImportTest failures in trunk
  • [SQOOP-615] - Error in Avro generation
  • [SQOOP-616] - HBase import/export is not working on non secure cluster where security is available
  • [SQOOP-617] - User guide is missing argument --fetch-size
  • [SQOOP-628] - Postgres direct import fails on non-lowercase column names
  • [SQOOP-636] - ExportJobTestCase.runExport method does not reuse the existing Configuration and SqoopOptions
  • [SQOOP-652] - Improve error message generated in DataDrivenDBInputFormat when no splitter is available
  • [SQOOP-654] - PostgreSQL direct connector is ignoring --null(-input)string and --null(-input)-non-string arguments
  • [SQOOP-657] - ant checkstyle results in errors
  • [SQOOP-687] - org.apache.sqoop.tool.ExportTool.applyOptions invokes applyOutputFormatOptions(..) twice in succession
  • [SQOOP-706] - Sqoop won't compile on Java 7 due to new methods in ResultSet interface
  • [SQOOP-721] - Duplicating rows on export when exporting from compressed files.
  • [SQOOP-725] - Enable HBase tests in sqoop 1
  • [SQOOP-730] - User Guide Update: Section 3 - Sqoop Releases
  • [SQOOP-741] - The OracleConnect getTables() implementation needs to restrict tables to current user
  • [SQOOP-796] - Unable to use new Hadoop environment variables
  • [SQOOP-808] - SQLExceptions From Batched Exports Aren't Very Helpful
  • [SQOOP-818] - Missing method setConnManager(Lcom/cloudera/sqoop/manager/ConnManager;)V in ExportJobContContext breaks
  • MS SQL Connector:
  • [SQOOP-821] - Hadoop has changed logic for job id in LocalJobRunner that breaks Lob* tests
  • [SQOOP-824] - Sqoop code generation in 'update' export mode incompatible with '--columns' option
  • [SQOOP-825] - Warning for incorrect usage of hive parameters is not checking HIVE_HOME correctly
  • [SQOOP-829] - Error Messages For --map-column-java Should Be More Useful
  • [SQOOP-830] - HBase import formatting BigDecimal inconsistently
  • [SQOOP-840] - Inconsistent java generics declaration for exportwriter classes
  • [SQOOP-846] - Provide a connector for Netezza appliances
  • [SQOOP-853] - sqoop list-tables isn't returning the tables that have custom schemas
  • [SQOOP-854] - pg_bulkload: Return value of pg_bulkload utility is being ignored
  • [SQOOP-855] - pg_bulkload: NullPointerException will be thrown if user specified invalid path the binary
  • [SQOOP-860] - NullPointerException when running procedure export against old database
  • [SQOOP-862] - Hbase import fails if there is a row where all columns are null
  • [SQOOP-870] - Allow Sqoop to import row key column into HBase
  • [SQOOP-901] - Allow user to override hardcoded boolean strings in direct PostgreSQL connector
  • Test:
  • [SQOOP-606] - Increase unit test timeout