Hadoop Changelog

What's new in Hadoop 2.7.2

Feb 23, 2016
  • YARN-4434. Minor bug reported by Takashi Ohnishi and fixed by Weiwei Yang (documentation , nodemanager) NodeManager Disk Checker parameter documentation is not correct
  • YARN-4424. Blocker bug reported by Yesha Vora and fixed by Jian He Fix deadlock in RMAppImpl
  • YARN-4365. Major bug reported by Jason Lowe and fixed by Kuhu Shukla (resourcemanager) FileSystemNodeLabelStore should check for root dir existence on startup
  • YARN-4354. Blocker bug reported by Jason Lowe and fixed by Jason Lowe (nodemanager) Public resource localization fails with NPE
  • YARN-4348. Blocker bug reported by Tsuyoshi Ozawa and fixed by Tsuyoshi Ozawa ZKRMStateStore.syncInternal shouldn't wait for sync completion for avoiding blocking ZK's event thread
  • YARN-4344. Critical bug reported by Varun Vasudev and fixed by Varun Vasudev (resourcemanager) NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations
  • YARN-4326. Major bug reported by MENG DING and fixed by MENG DING Fix TestDistributedShell timeout as AHS in MiniYarnCluster no longer binds to default port 8188
  • YARN-4321. Major bug reported by Varun Saxena and fixed by Varun Saxena (resourcemanager) Incessant retries if NoAuthException is thrown by Zookeeper in non HA mode
  • YARN-4320. Major bug reported by Varun Saxena and fixed by Varun Saxena TestJobHistoryEventHandler fails as AHS in MiniYarnCluster no longer binds to default port 8188
  • YARN-4313. Major bug reported by Jian He and fixed by Jian He Race condition in MiniMRYarnCluster when getting history server address
  • YARN-4312. Major bug reported by Varun Saxena and fixed by Varun Saxena TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 as some of the test cases time out
  • YARN-4281. Blocker bug reported by Chang Li and fixed by Chang Li 2.7 RM app page is broken
  • YARN-4241. Major bug reported by Anthony Rojas and fixed by Anthony Rojas (documentation) Fix typo of property name in
  • YARN-default.xml
  • YARN-4209. Critical bug reported by zhihai xu and fixed by zhihai xu (resourcemanager) RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition
  • YARN-4180. Critical bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot (resourcemanager) AMLauncher does not retry on failures when talking to NM
  • YARN-4158. Minor improvement reported by zhihai xu and fixed by zhihai xu (nodemanager) Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainers
  • YARN-4127. Major bug reported by Jian He and fixed by Varun Saxena (resourcemanager) RM fail with noAuth error if switched from failover mode to non-failover mode
  • YARN-4105. Major bug reported by Chang Li and fixed by Chang Li (capacityscheduler) Capacity Scheduler headroom for DRF is wrong
  • YARN-4103. Major bug reported by Jonathan Eagles and fixed by Jonathan Eagles RM WebServices missing scheme for appattempts logLinks
  • YARN-4101. Critical sub-task reported by Yesha Vora and fixed by Xuan Gong (yarn) RM should print alert messages if Zookeeper and Resourcemanager gets connection issue
  • YARN-4096. Major bug reported by Jason Lowe and fixed by Jason Lowe (log-aggregation , nodemanager) App local logs are leaked if log aggregation fails to initialize for the app
  • YARN-4092. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager) RM HA UI redirection needs to be fixed when both RMs are in standby mode
  • YARN-4087. Major bug reported by Jian He and fixed by Jian He Followup fixes after
  • YARN-2019 regarding RM behavior when state-store error occurs
  • YARN-4047. Major bug reported by Jason Lowe and fixed by Jason Lowe (resourcemanager) ClientRMService getApplications has high scheduler lock contention
  • YARN-4041. Major bug reported by Jason Lowe and fixed by Sunil G (resourcemanager) Slow delegation token renewal can severely prolong RM recovery
  • YARN-4009. Major bug reported by Prakash Ramachandran and fixed by Varun Vasudev CORS support for ResourceManager REST API
  • YARN-4005. Major bug reported by Jun Gong and fixed by Jun Gong Completed container whose app is finished is not removed from NMStateStore
  • YARN-4000. Major bug reported by Jason Lowe and fixed by Varun Saxena (capacityscheduler , resourcemanager) RM crashes with NPE if leaf queue becomes parent queue during restart
  • YARN-3999. Major bug reported by Jian He and fixed by Jian He RM hangs on draining events
  • YARN-3990. Critical bug reported by Rohith Sharma K S and fixed by Bibin A Chundatt (resourcemanager) AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected
  • YARN-3978. Major improvement reported by Eric Payne and fixed by Eric Payne (timelineserver , yarn) Configurably turn off the saving of container info in Generic AHS
  • YARN-3975. Major bug reported by Mit Desai and fixed by Mit Desai WebAppProxyServlet should not redirect to RM page if AHS is enabled
  • YARN-3969. Major sub-task reported by Subru Krishnan and fixed by Subru Krishnan (capacityscheduler , fairscheduler , resourcemanager) Allow jobs to be submitted to reservation that is active but does not have any allocations
  • YARN-3967. Major bug reported by Mit Desai and fixed by Mit Desai Fetch the application report from the AHS if the RM does not know about it
  • YARN-3925. Critical bug reported by zhihai xu and fixed by zhihai xu (nodemanager) ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks.
  • YARN-3905. Major bug reported by Eric Payne and fixed by Eric Payne (timelineserver) Application History Server UI NPEs when accessing apps run after RM restart
  • YARN-3896. Major bug reported by Jun Gong and fixed by Jun Gong (resourcemanager) RMNode transitioned from RUNNING to REBOOTED because its response id had not been reset synchronously
  • YARN-3893. Critical sub-task reported by Bibin A Chundatt and fixed by Bibin A Chundatt (resourcemanager) Both RM in active state when Admin#transitionToActive failure from refeshAll()
  • YARN-3878. Critical bug reported by Varun Saxena and fixed by Varun Saxena AsyncDispatcher can hang while stopping if it is configured for draining events on stop
  • YARN-3857. Critical bug reported by mujunchao and fixed by mujunchao (resourcemanager) Memory leak in ResourceManager with SIMPLE mode
  • YARN-3802. Major bug reported by zhihai xu and fixed by zhihai xu (resourcemanager) Two RMNodes for the same NodeId are used in RM sometimes after NM is reconnected.
  • YARN-3798. Blocker bug reported by Bibin A Chundatt and fixed by Varun Saxena (resourcemanager) ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED
  • YARN-3793. Major bug reported by Karthik Kambatla and fixed by Varun Saxena (nodemanager) Several NPEs when deleting local files on NM recovery
  • YARN-3780. Minor bug reported by zhihai xu and fixed by zhihai xu (resourcemanager) Should use equals when compare Resource in RMNodeImpl#ReconnectNodeTransition
  • YARN-3740. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager , webapp , yarn) Fixed the typo with the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS
  • YARN-3727. Major improvement reported by zhihai xu and fixed by zhihai xu (nodemanager) For better error recovery, check if the directory exists before using it for localization.
  • YARN-3700. Major sub-task reported by Xuan Gong and fixed by Xuan Gong (resourcemanager , webapp , yarn) ATS Web Performance issue at load time when large number of jobs
  • YARN-3697. Critical bug reported by zhihai xu and fixed by zhihai xu (fairscheduler) FairScheduler: ContinuousSchedulingThread can fail to shutdown
  • YARN-3690. Major bug reported by Akira AJISAKA and fixed by Brahma Reddy Battula (api , site) [JDK8] 'mvn site' fails
  • YARN-3624. Major bug reported by Mit Desai and fixed by Mit Desai (timelineserver) ApplicationHistoryServer reverses the order of the filters it gets
  • YARN-3619. Major bug reported by Jason Lowe and fixed by zhihai xu (nodemanager) ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException
  • YARN-3580. Major test reported by Robert Kanter and fixed by Robert Kanter (test) [JDK 8] TestClientRMService.testGetLabelsToNodes fails
  • YARN-3535. Critical bug reported by Peng Zhang and fixed by Peng Zhang (capacityscheduler , fairscheduler , resourcemanager) Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED
  • YARN-3508. Major bug reported by Jason Lowe and fixed by Varun Saxena (resourcemanager , scheduler) Prevent processing preemption events on the main RM dispatcher
  • YARN-3248. Major improvement reported by Varun Vasudev and fixed by Varun Vasudev (capacityscheduler , resourcemanager) Display count of nodes blacklisted by apps in the web UI
  • YARN-3170. Major improvement reported by Allen Wittenauer and fixed by Brahma Reddy Battula (documentation) YARN architecture document needs updating
  • YARN-3136. Major sub-task reported by Jason Lowe and fixed by Sunil G (scheduler) getTransferredContainers can be a bottleneck during AM registration
  • YARN-2902. Major sub-task reported by Jason Lowe and fixed by Varun Saxena (nodemanager) Killing a container that is localizing can orphan resources in the DOWNLOADING state
  • YARN-2890. Major bug reported by Mit Desai and fixed by Mit Desai MiniYarnCluster should turn on timeline service if configured to do so
  • YARN-2859. Critical bug reported by Hitesh Shah and fixed by Vinod Kumar Vavilapalli (timelineserver) ApplicationHistoryServer binds to default port 8188 in MiniYARNCluster
  • YARN-2801. Major sub-task reported by Gururaj Shetty and fixed by Wangda Tan (documentation) Add documentation for node labels feature
  • YARN-2513. Major sub-task reported by Jonathan Eagles and fixed by Jonathan Eagles (timelineserver) Host framework UIs in YARN for use with the ATS
  • YARN-2019. Critical bug reported by Junping Du and fixed by Jian He Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore
  • MAPREDUCE-6549. Major bug reported by Dustin Cote and fixed by Wilfred Spiegelenburg (mrv1 , mrv2) multibyte delimiters with LineRecordReader cause duplicate records
  • MAPREDUCE-6540. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (test) TestMRTimelineEventHandling fails
  • MAPREDUCE-6528. Critical bug reported by Junping Du and fixed by Junping Du (jobhistoryserver) Memory leak for HistoryFileManager.getJobSummary()
  • MAPREDUCE-6518. Major bug reported by Nathan Roberts and fixed by Chang Li (mrv2 , nodemanager) Set SO_KEEPALIVE on shuffle connections
  • MAPREDUCE-6497. Major bug reported by Shinichi Yamashita and fixed by Shinichi Yamashita Fix wrong value of JOB_FINISHED event in JobHistoryEventHandler
  • MAPREDUCE-6492. Critical bug reported by Bibin A Chundatt and fixed by Bibin A Chundatt AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask
  • MAPREDUCE-6481. Critical bug reported by zhihai xu and fixed by zhihai xu (mrv2) LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes.
  • MAPREDUCE-6474. Major bug reported by Nathan Roberts and fixed by Kuhu Shukla (mrv2 , nodemanager) ShuffleHandler can possibly exhaust nodemanager file descriptors
  • MAPREDUCE-6472. Major bug reported by Jason Lowe and fixed by Naganarasimha G R (mr-am) MapReduce AM should have java.io.tmpdir=./tmp to be consistent with tasks
  • MAPREDUCE-6454. Critical bug reported by Junping Du and fixed by Junping Du MapReduce doesn't set the HADOOP_CLASSPATH for jar lib in distributed cache.
  • MAPREDUCE-6451. Major bug reported by Kuhu Shukla and fixed by Kuhu Shukla (distcp) DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic
  • MAPREDUCE-6442. Major bug reported by Chang Li and fixed by Chang Li (client) Stack trace is missing when error occurs in client protocol provider's constructor
  • MAPREDUCE-6439. Critical bug reported by Anubhav Dhoot and fixed by Anubhav Dhoot AM may fail instead of retrying if RM shuts down during the allocate call
  • MAPREDUCE-6426. Major bug reported by Devaraj K and fixed by zhihai xu (test) TestShuffleHandler#testGetMapOutputInfo is failing
  • MAPREDUCE-6425. Major bug reported by zhihai xu and fixed by zhihai xu (mrv2 , nodemanager) ShuffleHandler passes wrong "base" parameter to getMapOutputInfo if mapId is not in the cache.
  • MAPREDUCE-6377. Minor bug reported by Bibin A Chundatt and fixed by zhihai xu (jobhistoryserver) JHS sorting on state column not working in webUi
  • MAPREDUCE-6273. Minor bug reported by zhihai xu and fixed by zhihai xu (jobhistoryserver) HistoryFileManager should check whether summaryFile exists to avoid FileNotFoundException causing HistoryFileInfo into MOVE_FAILED state
  • MAPREDUCE-5982. Major bug reported by Jason Lowe and fixed by Chang Li (mr-am) Task attempts that fail from the ASSIGNED state can disappear
  • MAPREDUCE-5948. Critical bug reported by Kris Geusebroek and fixed by Akira AJISAKA org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well
  • MAPREDUCE-5883. Minor bug reported by Nathan Roberts and fixed by Nathan Roberts "Total megabyte-seconds" in job counters is slightly misleading
  • MAPREDUCE-5649. Major bug reported by stanley shi and fixed by Gera Shegalov (mrv2) Reduce cannot use more than 2G memory for the final merge
  • HDFS-9574. Major bug reported by Kihwal Lee and fixed by Kihwal Lee Reduce client failures during datanode restart
  • HDFS-9470. Critical bug reported by Xiao Chen and fixed by Xiao Chen Encryption zone on root not loaded from fsimage after NN restart
  • HDFS-9445. Blocker bug reported by Kihwal Lee and fixed by Walter Su Datanode may deadlock while handling a bad volume
  • HDFS-9434. Major improvement reported by Tsz Wo Nicholas Sze and fixed by Tsz Wo Nicholas Sze (namenode) Recommission a datanode with 500k blocks may pause NN for 30 seconds
  • HDFS-9431. Major bug reported by Kazuho Fujii and fixed by Kazuho Fujii (
  • HDFS-client) DistributedFileSystem#concat fails if the target path is relative.
  • HDFS-9426. Blocker bug reported by Kihwal Lee and fixed by Kihwal Lee Rollingupgrade finalization is not backward compatible
  • HDFS-9413. Critical bug reported by Brahma Reddy Battula and fixed by Brahma Reddy Battula getContentSummary() on standby should throw StandbyException
  • HDFS-9317. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA (documentation) Document fsck -blockId and -storagepolicy options in branch-2.7
  • HDFS-9305. Major bug reported by Chris Nauroth and fixed by Arpit Agarwal (datanode) Delayed heartbeat processing causes storm of subsequent heartbeats
  • HDFS-9294. Blocker bug reported by DENG FEI and fixed by Brahma Reddy Battula (
  • HDFS-client) DFSClient deadlock when close file and failed to renew lease
  • HDFS-9290. Blocker bug reported by Tony Wu and fixed by Tony Wu DFSClient#callAppend() is not backward compatible for slightly older NameNodes
  • HDFS-9289. Critical bug reported by Chang Li and fixed by Chang Li Make DataStreamer#block thread safe and verify genStamp in commitBlock
  • HDFS-9273. Critical bug reported by Xiao Chen and fixed by Xiao Chen (namenode) ACLs on root directory may be lost after NN restart
  • HDFS-9221. Major improvement reported by Staffan Friberg and fixed by Staffan Friberg (performance) HdfsServerConstants#ReplicaState#getState should avoid calling values() since it creates a temporary array
  • HDFS-9220. Blocker bug reported by Bogdan Raducanu and fixed by Jing Zhao Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum
  • HDFS-9178. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee Slow datanode I/O can cause a wrong node to be marked bad
  • HDFS-9106. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee Transfer failure during pipeline recovery causes permanent write failures
  • HDFS-9083. Blocker bug reported by Rushabh S Shah and fixed by Rushabh S Shah (namenode) Replication violates block placement policy.
  • HDFS-9043. Minor bug reported by J.Andreina and fixed by J.Andreina (documentation) Doc updation for commands in HDFS Federation
  • HDFS-9042. Minor bug reported by J.Andreina and fixed by J.Andreina (documentation) Update document for the Storage policy name
  • HDFS-9033. Major bug reported by Archana T and fixed by Brahma Reddy Battula dfsadmin -metasave prints "NaN" for cache used%
  • HDFS-8995. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee Flaw in registration bookeeping can make DN die on reconnect
  • HDFS-8950. Major bug reported by Daniel Templeton and fixed by Daniel Templeton (datanode , namenode) NameNode refresh doesn't remove DataNodes that are no longer in the allowed list
  • HDFS-8891. Blocker bug reported by Yong Zhang and fixed by Yong Zhang HDFS concat should keep srcs order
  • HDFS-8879. Major bug reported by Kihwal Lee and fixed by Xiaoyu Yao (namenode) Quota by storage type usage incorrectly initialized upon namenode restart
  • HDFS-8867. Major bug reported by Rushabh S Shah and fixed by Daryn Sharp Enable optimized block reports
  • HDFS-8863. Critical bug reported by Kihwal Lee and fixed by Kihwal Lee The remaining space check in BlockPlacementPolicyDefault is flawed
  • HDFS-8852. Major bug reported by Hong Dai Thanh and fixed by Ajith S (documentation) HDFS architecture documentation of version 2.x is outdated about append write support
  • HDFS-8850. Major bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe (datanode) VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks
  • HDFS-8846. Major bug reported by Zhe Zhang and fixed by Zhe Zhang (namenode) Add a unit test for INotify functionality across a layout version upgrade
  • HDFS-8806. Major bug reported by Zhe Zhang and fixed by Zhe Zhang Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared
  • HDFS-8767. Critical bug reported by Haohui Mai and fixed by Kanaka Kumar Avvaru RawLocalFileSystem.listStatus() returns null for UNIX pipefile
  • HDFS-8722. Critical improvement reported by Kihwal Lee and fixed by Kihwal Lee Optimize datanode writes for small writes and flushes
  • HDFS-8676. Critical bug reported by Kihwal Lee and fixed by Walter Su Delayed rolling upgrade finalization can cause heartbeat expiration and write failures
  • HDFS-8659. Major improvement reported by Yongjun Zhang and fixed by Yongjun Zhang (datanode) Block scanner INFO message is spamming logs
  • HDFS-8656. Critical bug reported by Andrew Wang and fixed by Andrew Wang (rolling upgrades) Preserve compatibility of ClientProtocol#rollingUpgrade after finalization
  • HDFS-8615. Major bug reported by Akira AJISAKA and fixed by Brahma Reddy Battula (documentation) Correct HTTP method in WebHDFS document
  • HDFS-8431. Critical bug reported by Sumana Sathish and fixed by Anu Engineer (scripts) hdfs crypto class not found in Windows
  • HDFS-8384. Minor improvement reported by Tsz Wo Nicholas Sze and fixed by Jing Zhao (namenode) Allow NN to startup if there are files having a lease but are not under construction
  • HDFS-8219. Major bug reported by Peter Shi and fixed by Surendra Singh Lilhore setStoragePolicy with folder behavior is different after cluster restart
  • HDFS-8099. Minor bug reported by Charles Lamb and fixed by Charles Lamb (
  • HDFS-client) Change "DFSInputStream has been closed already" message to debug log level
  • HDFS-8046. Major bug reported by Kihwal Lee and fixed by Kihwal Lee Allow better control of getContentSummary
  • HDFS-7725. Major bug reported by Ming Ma and fixed by Ming Ma Incorrect "nodes in service" metrics caused all writes to fail
  • HDFS-7645. Major improvement reported by Nathan Roberts and fixed by Keisuke Ogiwara (datanode) Rolling upgrade is restoring blocks from trash multiple times
  • HDFS-7609. Critical bug reported by Carrey Zhan and fixed by Ming Ma (namenode) Avoid retry cache collision when Standby NameNode loading edits
  • HDFS-7314. Major improvement reported by Ming Ma and fixed by Ming Ma When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient
  • HDFS-6945. Critical bug reported by Akira AJISAKA and fixed by Akira AJISAKA (namenode) BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed
  • HADOOP-12577. Blocker bug reported by Wei-Chiu Chuang and fixed by Wei-Chiu Chuang (build , security) Bump up commons-collections version to 3.2.2 to address a security flaw
  • HADOOP-12526. Major bug reported by Sangjin Lee and fixed by Sangjin Lee (build) [Branch-2] there are duplicate dependency definitions in pom's
  • HADOOP-12465. Minor bug reported by Martin Petricek and fixed by Jagadesh Kiran N (documentation) Incorrect javadoc in WritableUtils.java
  • HADOOP-12464. Major bug reported by Kihwal Lee and fixed by Kihwal Lee (ipc) Interrupted client may try to fail-over and retry
  • HADOOP-12451. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (scripts) [Branch-2] Setting HADOOP_HOME explicitly should be allowed
  • HADOOP-12415. Major bug reported by Konstantin Boudnik and fixed by Tom Zeng (nfs) hdfs and nfs builds broken on -missing compile-time dependency on netty
  • HADOOP-12413. Major improvement reported by zhihai xu and fixed by zhihai xu (security) AccessControlList should avoid calling getGroupNames in isUserInList with empty groups.
  • HADOOP-12359. Major bug reported by Daniel Templeton and fixed by Jagadesh Kiran N (documentation) hadoop fs -getmerge doc is wrong
  • HADOOP-12304. Blocker bug reported by Chris Nauroth and fixed by Chris Nauroth (fs) Applications using FileContext fail with the default file system configured to be wasb/s3/etc.
  • HADOOP-12280. Minor improvement reported by Masatake Iwasaki and fixed by Masatake Iwasaki (test) Skip unit tests based on maven profile rather than NativeCodeLoader.isNativeCodeLoaded
  • HADOOP-12232. Major improvement reported by Chris Nauroth and fixed by Chris Nauroth (build) Upgrade Tomcat dependency to 6.0.44.
  • HADOOP-12230. Minor bug reported by Steve Loughran and fixed by Rakesh R (build)
  • HADOOP-project declares duplicate, conflicting curator dependencies
  • HADOOP-12213. Minor bug reported by Oleg Zhurakousky and fixed by Kuhu Shukla Interrupted exception can occur when Client#stop is called
  • HADOOP-12191. Major bug reported by Jason Lowe and fixed by Brahma Reddy Battula (io) Bzip2Factory is not thread safe
  • HADOOP-12186. Major bug reported by zhihai xu and fixed by zhihai xu (ha) ActiveStandbyElector shouldn't call monitorLockNodeAsync multiple times
  • HADOOP-12006. Major bug reported by Akira AJISAKA and fixed by Akira AJISAKA Remove unimplemented option for `hadoop fs -ls` from document in branch-2.7
  • HADOOP-11932. Critical bug reported by Jian He and fixed by Brahma Reddy Battula MetricsSinkAdapter hangs when being stopped
  • HADOOP-11812. Blocker improvement reported by Gera Shegalov and fixed by Gera Shegalov (fs) Implement listLocatedStatus for ViewFileSystem to speed up split calculation
  • HADOOP-11491. Critical bug reported by Gera Shegalov and fixed by Brahma Reddy Battula (fs) HarFs incorrectly declared as requiring an authority
  • HADOOP-10365. Minor bug reported by Ted Yu and fixed by Kiran Kumar M R (util) BufferedOutputStream in FileUtil#unpackEntries() should be closed in finally block
  • HADOOP-9242. Major bug reported by Andrey Klochkov and fixed by Andrey Klochkov (test) Duplicate surefire plugin config in
  • HADOOP-common
  • HADOOP-8151. Major bug reported by Todd Lipcon and fixed by Matt Foley (io , native) Error handling in snappy decompressor throws invalid exceptions
  • HADOOP-7139. Major improvement reported by Stephen Rose and fixed by Kanaka Kumar Avvaru (io) Allow appending to existing SequenceFiles Existing sequence files can be appended.
  • HADOOP-5323. Minor improvement reported by Suman Sehgal and fixed by Weiwei Yang (documentation) Trash documentation should describe its directory structure and configurations

New in Hadoop 2.7.1 (Jul 7, 2015)

  • Bug HADOOP-12103: Small refactoring of DelegationTokenAuthenticationFilter to allow code sharing
  • Bug HADOOP-12100: ImmutableFsPermission should not override applyUmask since that method doesn't modify the FsPermission
  • Bug HADOOP-12078: The default retry policy does not handle RetriableException correctly
  • Bug HADOOP-12058: Fix dead links to DistCp and Hadoop Archives pages.
  • Bug HADOOP-11934: Use of JavaKeyStoreProvider in LdapGroupsMapping causes infinite loop
  • Bug HADOOP-11973: Ensure ZkDelegationTokenSecretManager namespace znodes get created with ACLs
  • Bug HADOOP-11966: Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh.
  • Bug HADOOP-11663: Remove description about Java 6 from docs
  • Sub-task HADOOP-7468: HADOOP-7466 hadoop-core JAR contains a log4j.properties file
  • Improvement HADOOP-9384: Update S3 native fs implementation to use AWS SDK to support authorization through roles
  • Bug HADOOP-9658: SnappyCodec#checkNativeCodeLoaded may unexpectedly fail when native code is not loaded
  • Bug HADOOP-11891: OsSecureRandom should lazily fill its reservoir
  • Bug HADOOP-11872: "hadoop dfs" command prints message about using "yarn jar" on Windows(branch-2 only)
  • Bug HADOOP-11730: Regression: s3n read failure recovery broken
  • Bug HADOOP-11802: DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm
  • Improvement HADOOP-11851: s3n to swallow IOEs on inner stream close
  • Bug HADOOP-11868: Invalid user logins trigger large backtraces in server log

New in Hadoop 2.7.0 (Apr 23, 2015)

  • Common:
  • Authentication improvements when using an HTTP proxy server. This is useful when accessing WebHDFS via a proxy server.
  • A new Hadoop metrics sink that allows writing directly to Graphite.
  • Specification work related to the Hadoop Compatible Filesystem (HCFS) effort.
  • HDFS:
  • Support for POSIX-style filesystem extended attributes. See the user documentation for more details.
  • Using the OfflineImageViewer, clients can now browse an fsimage via the WebHDFS API.
  • The NFS gateway received a number of supportability improvements and bug fixes. The Hadoop portmapper is no longer required to run the gateway, and the gateway is now able to reject connections from unprivileged ports.
  • The SecondaryNameNode, JournalNode, and DataNode web UIs have been modernized with HTML5 and Javascript.
  • YARN:
  • YARN’s REST APIs now support write/modify operations. Users can submit and kill applications through REST APIs.
  • The timeline store in YARN, used for storing generic and application-specific information for applications, supports authentication through Kerberos.
  • The Fair Scheduler supports dynamic hierarchical user queues, user queues are created dynamically at runtime under any specified parent-queue.

New in Hadoop 2.6.0 (Dec 27, 2014)

  • Common:
  • Authentication improvements when using an HTTP proxy server. This is useful when accessing WebHDFS via a proxy server.
  • A new Hadoop metrics sink that allows writing directly to Graphite.
  • Specification work related to the Hadoop Compatible Filesystem (HCFS) effort.
  • HDFS:
  • Support for POSIX-style filesystem extended attributes. See the user documentation for more details.
  • Using the OfflineImageViewer, clients can now browse an fsimage via the WebHDFS API.
  • The NFS gateway received a number of supportability improvements and bug fixes. The Hadoop portmapper is no longer required to run the gateway, and the gateway is now able to reject connections from unprivileged ports.
  • The SecondaryNameNode, JournalNode, and DataNode web UIs have been modernized with HTML5 and Javascript.
  • YARN:
  • YARN's REST APIs now support write/modify operations. Users can submit and kill applications through REST APIs.
  • The timeline store in YARN, used for storing generic and application-specific information for applications, supports authentication through Kerberos.
  • The Fair Scheduler supports dynamic hierarchical user queues, user queues are created dynamically at runtime under any specified parent-queue.

New in Hadoop 2.5.1 (Sep 15, 2014)

  • Changes since Hadoop 2.5.0:
  • MAPREDUCE-6033. Major bug reported by Yu Gao and fixed by Yu Gao. Users are not allowed to view their own jobs, denied by JobACLsManager.
  • HADOOP-11065. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla. Rat check should exclude **/build/**.
  • HADOOP-11001. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (scripts). Fix test-patch to work with the git repo.
  • HADOOP-10957. Blocker bug reported by Colin Patrick McCabe and fixed by Colin Patrick McCabe. The globber will sometimes erroneously return a permission denied exception when there is a non-terminal wildcard.
  • HADOOP-10956. Blocker bug reported by Karthik Kambatla and fixed by Karthik Kambatla (scripts). Fix create-release script to include docs and necessary txt files.

New in Hadoop 2.5.0 (Sep 12, 2014)

  • Major features and improvements:
  • Authentication improvements when using an HTTP proxy server.
  • A new Hadoop Metrics sink that allows writing directly to Graphite.
  • Specification for Hadoop Compatible Filesystem effort.
  • Support for POSIX-style filesystem extended attributes.
  • OfflineImageViewer to browse an fsimage via the WebHDFS API.
  • Supportability improvements and bug fixes to the NFS gateway.
  • Modernized web UIs (HTML5 and Javascript) for HDFS daemons.
  • YARN's REST APIs support submitting and killing applications.
  • Kerberos integration for the YARN's timeline store.
  • FairScheduler allows creating user queues at runtime under any specified parent queue.

New in Hadoop 2.4.1 (Jul 9, 2014)

  • Highlights:
  • CVE-2014-0229: Add privilege checks to HDFS admin sub-commands refreshNamenodes, deleteBlockPool and shutdownDatanode.

New in Hadoop 2.4.0 (Apr 23, 2014)

  • Significant enhancements:
  • Support for Access Control Lists in HDFS
  • Native support for Rolling Upgrades in HDFS
  • Usage of protocol-buffers for HDFS FSImage for smooth operational upgrades
  • Complete HTTPS support in HDFS
  • Support for Automatic Failover of the YARN ResourceManager
  • Enhanced support for new applications on YARN with Application History Server and Application Timeline Server
  • Support for strong SLAs in YARN CapacityScheduler via Preemption

New in Hadoop 2.3.0 (Feb 28, 2014)

  • Highlights:
  • Support for Heterogeneous Storage hierarchy in HDFS.
  • In-memory cache for HDFS data with centralized administration and management.
  • Simplified distribution of MapReduce binaries via HDFS in YARN Distributed Cache

New in Hadoop 2.2.0 (Oct 22, 2013)

  • This release has a number of significant highlights compared to Hadoop 1.x:
  • YARN - A general purpose resource management system for Hadoop to allow MapReduce and other other data processing frameworks and services
  • High Availability for HDFS
  • HDFS Federation
  • HDFS Snapshots
  • NFSv3 access to data in HDFS
  • Support for running Hadoop on Microsoft Windows
  • Binary Compatibility for MapReduce applications built on hadoop-1.x
  • Substantial amount of integration testing with rest of projects in the ecosystem
  • A couple of important points to note while upgrading to hadoop-2.2.0:
  • HDFS - The HDFS community decided to push the symlinks feature out to a future 2.3.0 release and is currently disabled.
  • YARN/MapReduce - Users need to change ShuffleHandler service name from mapreduce.shuffle to mapreduce_shuffle.

New in Hadoop 2.1.0 Beta (Sep 2, 2013)

  • HDFS Snapshots
  • Support for running Hadoop on Microsoft Windows
  • YARN API stabilization
  • Binary Compatibility for MapReduce applications built on hadoop-1.x
  • Substantial amount of integration testing with rest of projects in the ecosystem

New in Hadoop 1.2.1 (Aug 5, 2013)

  • CapacityScheduler incorrectly utilizes extra-resources of queue for high-memory jobs
  • MetricsDynamicMBeanBase has concurrency issues in createMBeanInfo
  • BlockDecompressorStream#decompress will throw EOFException instead of return -1 when EOF
  • fix hadoop.spec to add task-log4j.properties
  • TestBalancerWithNodeGroup times out
  • DataNode#checkDiskError should not be called on network errors
  • TestPipelinesFailover#testPipelineRecoveryStress fails sporadically
  • Diagnostic logging while loading name/edits files
  • Add extra info to JH files
  • Syslog missing from Map/Reduce tasks
  • JT can show the same job multiple times in Retired Jobs section
  • CombineInputFormat isn't thread safe affecting HiveServer
  • Job failed because of JvmManager running into inconsistent state
  • Ampersand in JSPUtil.java is not escaped
  • JobTracker memory leak caused by CleanupQueue reopening FileSystem
  • Deadlock between RenewalTimerTask methods cancel() and run()
  • Save memory by set capacity, load factor and concurrency level for ConcurrentHashMap in TaskInProgress
  • Delegation Token renewal exception in jobtracker logs

New in Hadoop 2.0.5 Alpha (Jun 10, 2013)

  • This release delivers a number of critical bug-fixes for hadoop-2.x uncovered during integration testing of previous release.

New in Hadoop 1.2.0 (Jun 10, 2013)

  • DistCp v2 backported
  • Web services for JobTracker
  • WebHDFS enhancements
  • Extensions of task placement and replica placement policy interfaces
  • Offline Image Viewer backported
  • Namenode more robust in case of edit log corruption
  • Add NodeGroups level to NetworkTopology
  • Add "unset" to Configuration API

New in Hadoop 2.0.3 Alpha (Feb 18, 2013)

  • QJM for HDFS HA for NameNode
  • Multi-resource scheduling (CPU and memory) for YARN
  • YARN ResourceManager Restart
  • Significant stability at scale for YARN (over 30,000 nodes and 14 million applications so far, at time of release)

New in Hadoop 1.1.1 (Dec 4, 2012)

  • Bug fixes and improvements

New in Hadoop 2.0.2 Alpha (Oct 18, 2012)

  • This delivers significant enhancements to HDFS HA. Also it has a significantly more stable version of YARN which, at the time of release, has already been deployed on a 2000 node cluster.

New in Hadoop 1.0.4 (Oct 18, 2012)

  • Security issue CVE-2012-4449: Hadoop tokens use a 20-bit secret
  • HADOOP-7154 - set MALLOC_ARENA_MAX in hadoop-config.sh to resolve problems with glibc in RHEL-6
  • HDFS-3652 - FSEditLog failure removes the wrong edit stream when storage dirs have same name
  • MAPREDUCE-4399 - Fix (up to 3x) performance regression in shuffle