[hadoop2.7.1]I/O之tfile

tenght

浏览: 50427 次

最近访客更多访客>>

jxqc_job

汽车城路

极品拖拉机

aubdiy

博主相关

博客

微博

相册

留言

关于我

文章分类

社区版块

存档分类

在这儿做个记录吧

org.apache.hadoop.io.file.tfile.TFile

@Public
@Evolving
ATFileisacontainerofkey-valuepairs.Bothkeysandvaluesaretype-lessbytes.Keysarerestrictedto64KB,valuelengthisnotrestricted(practicallylimitedtotheavailablediskstorage).TFilefurtherprovidesthefollowingfeatures:
BlockCompression.
Namedmetadatablocks.
Sortedorunsortedkeys.
Seekbykeyorbyfileoffset.
ThememoryfootprintofaTFileincludesthefollowing:
Someconstantoverheadofreadingorwritingacompressedblock.
Eachcompressedblockrequiresonecompression/decompressioncodecforI/O.
Temporaryspacetobufferthekey.
Temporaryspacetobufferthevalue(forTFile.Writeronly).Valuesarechunkencoded,sothatwebufferatmostonechunkofuserdata.Bydefault,thechunkbufferis1MB.Readingchunkedvaluedoesnotrequireadditionalmemory.
TFileindex,whichisproportionaltothetotalnumberofDataBlocks.Thetotalamountofmemoryneededtoholdtheindexcanbeestimatedas(56+AvgKeySize)*NumBlocks.MetaBlockindex,whichisproportionaltothetotalnumberofMetaBlocks.ThetotalamountofmemoryneededtoholdtheindexforMetaBlockscanbeestimatedas(40+AvgMetaBlockName)*NumMetaBlock.
ThebehaviorofTFilecanbecustomizedbythefollowingvariablesthroughConfiguration:
tfile.io.chunk.size:Valuechunksize.Integer(inbytes).Defaultto1MB.Valuesofthelengthlessthanthechunksizeisguaranteedtohaveknownvaluelengthinreadtime(SeeTFile.Reader.Scanner.Entry.isValueLengthKnown()).
tfile.fs.output.buffer.size:BuffersizeusedforFSDataOutputStream.Integer(inbytes).Defaultto256KB.
tfile.fs.input.buffer.size:BuffersizeusedforFSDataInputStream.Integer(inbytes).Defaultto256KB.
Suggestionsonperformanceoptimization.
Minimumblocksize.Werecommendasettingofminimumblocksizebetween256KBto1MBforgeneralusage.Largerblocksizeispreferrediffilesareprimarilyforsequentialaccess.However,itwouldleadtoinefficientrandomaccess(becausetherearemoredatatodecompress).Smallerblocksaregoodforrandomaccess,butrequiremorememorytoholdtheblockindex,andmaybeslowertocreate(becausewemustflushthecompressorstreamattheconclusionofeachdatablock,whichleadstoanFSI/Oflush).Further,duetotheinternalcachinginCompressioncodec,thesmallestpossibleblocksizewouldbearound20KB-30KB.
Thecurrentimplementationdoesnotoffertruemulti-threadingforreading.TheimplementationusesFSDataInputStreamseek()+read(),whichisshowntobemuchfasterthanpositioned-readcallinsinglethreadmode.However,italsomeansthatifmultiplethreadsattempttoaccessthesameTFile(usingmultiplescanners)simultaneously,theactualI/OiscarriedoutsequentiallyeveniftheyaccessdifferentDFSblocks.
Compressioncodec.Use"none"ifthedataisnotverycompressable(bycompressable,Imeanacompressionratioatleast2:1).Generally,use"lzo"asthestartingpointforexperimenting."gz"oversslightlybettercompressionratioover"lzo"butrequires4xCPUtocompressand2xCPUtodecompress,comparingto"lzo".
Filesystembuffering,iftheunderlyingFSDataInputStreamandFSDataOutputStreamisalreadyadequatelybuffered;orifapplicationsreads/writeskeysandvaluesinlargebuffers,wecanreducethesizesofinput/outputbufferinginTFilelayerbysettingtheconfigurationparameters"tfile.fs.input.buffer.size"and"tfile.fs.output.buffer.size".

分享到：