Lucene 3.0.2索引文件官方文档（二）

Deletable File

A writer dynamically computes the files that are deletable, instead, so no file is written.

Compound Files

Starting with Lucene 1.4 the compound file format became default. This is simply a container for all files described in the next section (except for the .del file).

Compound (.cfs) --> FileCount, <DataOffset, FileName> ^FileCount , FileData ^FileCount

FileCount --> VInt

DataOffset --> Long

FileName --> String

FileData --> raw file data

The raw file data is the data from the individual files named above.

Starting with Lucene 2.3, doc store files (stored field values and term vectors) can be shared in a single set of files for more than one segment. When compound file is enabled, these shared files will be added into a single compound file (same format as above) but with the extension .cfx.

Per-Segment Files

The remaining files are all per-segment, and are thus defined by suffix.

Fields

Field Info

Field names are stored in the field info file, with suffix .fnm.

FieldInfos (.fnm) --> FNMVersion,FieldsCount, <FieldName, FieldBits> ^FieldsCount

FNMVersion, FieldsCount --> VInt

FieldName --> String

FieldBits --> Byte

 

• The low-order bit is one for indexed fields, and zero for non-indexed fields.
• The second lowest-order bit is one for fields that have term vectors stored, and zero for fields without term vectors.
• If the third lowest-order bit is set (0x04), term positions are stored with the term vectors.
• If the fourth lowest-order bit is set (0x08), term offsets are stored with the term vectors.
• If the fifth lowest-order bit is set (0x10), norms are omitted for the indexed field.
• If the sixth lowest-order bit is set (0x20), payloads are stored for the indexed field.

 

FNMVersion (added in 2.9) is always -2.

Fields are numbered by their order in this file. Thus field zero is the first field in the file, field one the next, and so on. Note that, like document numbers, field numbers are segment relative.

Stored Fields

Stored fields are represented by two files:

1. The field index, or .fdx file.

   This contains, for each document, a pointer to its field data, as follows:

   FieldIndex (.fdx) --> <FieldValuesPosition> ^SegSize

   FieldValuesPosition --> Uint64

   This is used to find the location within the field data file of the fields of a particular document. Because it contains fixed-length data, this file may be easily randomly accessed. The position of document n 's field data is the Uint64 at n*8 in this file.

2.  The field data, or .fdt file.

   This contains the stored fields of each document, as follows:

   FieldData (.fdt) --> <DocFieldData> ^SegSize

   DocFieldData --> FieldCount, <FieldNum, Bits, Value> ^FieldCount

   FieldCount --> VInt

   FieldNum --> VInt

   Bits --> Byte

    

   • low order bit is one for tokenized fields
   • second bit is one for fields containing binary data
   • third bit is one for fields with compression option enabled (if compression is enabled, the algorithm used is ZLIB), only available for indexes until Lucene version 2.9.x

    

   Value --> String | BinaryValue (depending on Bits)

   BinaryValue --> ValueSize, <Byte>^ValueSize

   ValueSize --> VInt

Term Dictionary

The term dictionary is represented as two files:

1. The term infos, or tis file.

   TermInfoFile (.tis)--> TIVersion, TermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermInfos

   TIVersion --> UInt32

   TermCount --> UInt64

   IndexInterval --> UInt32

   SkipInterval --> UInt32

   MaxSkipLevels --> UInt32

   TermInfos --> <TermInfo> ^TermCount

   TermInfo --> <Term, DocFreq, FreqDelta, ProxDelta, SkipDelta>

   Term --> <PrefixLength, Suffix, FieldNum>

   Suffix --> String

   PrefixLength, DocFreq, FreqDelta, ProxDelta, SkipDelta 
   --> VInt

   This file is sorted by Term. Terms are ordered first lexicographically (by UTF16 character code) by the term's field name, and within that lexicographically (by UTF16 character code) by the term's text.

   TIVersion names the version of the format of this file and is equal to TermInfosWriter.FORMAT_CURRENT.

   Term text prefixes are shared. The PrefixLength is the number of initial characters from the previous term which must be pre-pended to a term's suffix in order to form the term's text. Thus, if the previous term's text was "bone" and the term is "boy", the PrefixLength is two and the suffix is "y".

   FieldNumber determines the term's field, whose name is stored in the .fdt file.

   DocFreq is the count of documents which contain the term.

   FreqDelta determines the position of this term's TermFreqs within the .frq file. In particular, it is the difference between the position of this term's data in that file and the position of the previous term's data (or zero, for the first term in the file).

   ProxDelta determines the position of this term's TermPositions within the .prx file. In particular, it is the difference between the position of this term's data in that file and the position of the previous term's data (or zero, for the first term in the file. For fields with omitTf true, this will be 0 since prox information is not stored.

   SkipDelta determines the position of this term's SkipData within the .frq file. In particular, it is the number of bytes after TermFreqs that the SkipData starts. In other words, it is the length of the TermFreq data. SkipDelta is only stored if DocFreq is not smaller than SkipInterval.

2.  The term info index, or .tii file.

   This contains every IndexInterval ^th entry from the .tis file, along with its location in the "tis" file. This is designed to be read entirely into memory and used to provide random access to the "tis" file.

   The structure of this file is very similar to the .tis file, with the addition of one item per record, the IndexDelta.

   TermInfoIndex (.tii)--> TIVersion, IndexTermCount, IndexInterval, SkipInterval, MaxSkipLevels, TermIndices

   TIVersion --> UInt32

   IndexTermCount --> UInt64

   IndexInterval --> UInt32

   SkipInterval --> UInt32

   TermIndices --> <TermInfo, IndexDelta> ^{IndexTermCount}

   IndexDelta --> VLong

   IndexDelta determines the position of this term's TermInfo within the .tis file. In particular, it is the difference between the position of this term's entry in that file and the position of the previous term's entry.

   SkipInterval is the fraction of TermDocs stored in skip tables. It is used to accelerate TermDocs.skipTo(int). Larger values result in smaller indexes, greater acceleration, but fewer accelerable cases, while smaller values result in bigger indexes, less acceleration (in case of a small value for MaxSkipLevels) and more accelerable cases.

   MaxSkipLevels is the max. number of skip levels stored for each term in the .frq file. A low value results in smaller indexes but less acceleration, a larger value results in slighly larger indexes but greater acceleration. See format of .frq file for more information about skip levels.

Frequencies

The .frq file contains the lists of documents which contain each term, along with the frequency of the term in that document (if omitTf is false).

FreqFile (.frq) --> <TermFreqs, SkipData> ^TermCount

TermFreqs --> <TermFreq> ^DocFreq

TermFreq --> DocDelta[, Freq?]

SkipData --> <<SkipLevelLength, SkipLevel> ^{NumSkipLevels-1}, SkipLevel> <SkipDatum>

SkipLevel --> <SkipDatum> ^{DocFreq/(SkipInterval^(Level + 1))}

SkipDatum --> DocSkip,PayloadLength?,FreqSkip,ProxSkip,SkipChildLevelPointer?

DocDelta,Freq,DocSkip,PayloadLength,FreqSkip,ProxSkip --> VInt

SkipChildLevelPointer --> VLong

TermFreqs are ordered by term (the term is implicit, from the .tis file).

TermFreq entries are ordered by increasing document number.

DocDelta: if omitTf is false, this determines both the document number and the frequency. In particular, DocDelta/2 is the difference between this document number and the previous document number (or zero when this is the first document in a TermFreqs). When DocDelta is odd, the frequency is one. When DocDelta is even, the frequency is read as another VInt. If omitTf is true, DocDelta contains the gap (not multiplied by 2) between document numbers and no frequency information is stored.

For example, the TermFreqs for a term which occurs once in document seven and three times in document eleven, with omitTf false, would be the following sequence of VInts:

15, 8, 3

If omitTf were true it would be this sequence of VInts instead:

7,4

DocSkip records the document number before every SkipInterval ^th document in TermFreqs. If payloads are disabled for the term's field, then DocSkip represents the difference from the previous value in the sequence. If payloads are enabled for the term's field, then DocSkip/2 represents the difference from the previous value in the sequence. If payloads are enabled and DocSkip is odd, then PayloadLength is stored indicating the length of the last payload before the SkipInterval^th document in TermPositions. FreqSkip and ProxSkip record the position of every SkipInterval ^th entry in FreqFile and ProxFile, respectively. File positions are relative to the start of TermFreqs and Positions, to the previous SkipDatum in the sequence.

For example, if DocFreq=35 and SkipInterval=16, then there are two SkipData entries, containing the 15 ^th and 31 ^st document numbers in TermFreqs. The first FreqSkip names the number of bytes after the beginning of TermFreqs that the 16 ^th SkipDatum starts, and the second the number of bytes after that that the 32 ^nd starts. The first ProxSkip names the number of bytes after the beginning of Positions that the 16 ^th SkipDatum starts, and the second the number of bytes after that that the 32 ^nd starts.

Each term can have multiple skip levels. The amount of skip levels for a term is NumSkipLevels = Min(MaxSkipLevels, floor(log(DocFreq/log(SkipInterval)))). The number of SkipData entries for a skip level is DocFreq/(SkipInterval^(Level + 1)), whereas the lowest skip level is Level=0. 
Example: SkipInterval = 4, MaxSkipLevels = 2, DocFreq = 35. Then skip level 0 has 8 SkipData entries, containing the 3^rd, 7^th, 11^th, 15^th, 19^th, 23^rd, 27^th, and 31^st document numbers in TermFreqs. Skip level 1 has 2 SkipData entries, containing the 15^th and 31^st document numbers in TermFreqs. 
The SkipData entries on all upper levels > 0 contain a SkipChildLevelPointer referencing the corresponding SkipData entry in level-1. In the example has entry 15 on level 1 a pointer to entry 15 on level 0 and entry 31 on level 1 a pointer to entry 31 on level 0.

Positions

The .prx file contains the lists of positions that each term occurs at within documents. Note that fields with omitTf true do not store anything into this file, and if all fields in the index have omitTf true then the .prx file will not exist.

ProxFile (.prx) --> <TermPositions> ^TermCount

TermPositions --> <Positions> ^DocFreq

Positions --> <PositionDelta,Payload?> ^Freq

Payload --> <PayloadLength?,PayloadData>

PositionDelta --> VInt

PayloadLength --> VInt

PayloadData --> byte^{PayloadLength}

TermPositions are ordered by term (the term is implicit, from the .tis file).

Positions entries are ordered by increasing document number (the document number is implicit from the .frq file).

PositionDelta is, if payloads are disabled for the term's field, the difference between the position of the current occurrence in the document and the previous occurrence (or zero, if this is the first occurrence in this document). If payloads are enabled for the term's field, then PositionDelta/2 is the difference between the current and the previous position. If payloads are enabled and PositionDelta is odd, then PayloadLength is stored, indicating the length of the payload at the current term position.

For example, the TermPositions for a term which occurs as the fourth term in one document, and as the fifth and ninth term in a subsequent document, would be the following sequence of VInts (payloads disabled):

4, 5, 4

PayloadData is metadata associated with the current term position. If PayloadLength is stored at the current position, then it indicates the length of this Payload. If PayloadLength is not stored, then this Payload has the same length as the Payload at the previous position.

Normalization Factors

There's a single .nrm file containing all norms:

AllNorms (.nrm) --> NormsHeader,<Norms> ^{NumFieldsWithNorms}

Norms --> <Byte> ^SegSize

NormsHeader --> 'N','R','M',Version

Version --> Byte

NormsHeader has 4 bytes, last of which is the format version for this file, currently -1.

Each byte encodes a floating point value. Bits 0-2 contain the 3-bit mantissa, and bits 3-8 contain the 5-bit exponent.

These are converted to an IEEE single float value as follows:

1. If the byte is zero, use a zero float.

2. Otherwise, set the sign bit of the float to zero;

3. add 48 to the exponent and use this as the float's exponent;

4. map the mantissa to the high-order 3 bits of the float's mantissa; and

5. set the low-order 21 bits of the float's mantissa to zero.

A separate norm file is created when the norm values of an existing segment are modified. When field N is modified, a separate norm file .sN is created, to maintain the norm values for that field.

Separate norm files are created (when adequate) for both compound and non compound segments.

Term Vectors

Term Vector support is an optional on a field by field basis. It consists of 3 files.

1. The Document Index or .tvx file.

   For each document, this stores the offset into the document data (.tvd) and field data (.tvf) files.

   DocumentIndex (.tvx) --> TVXVersion<DocumentPosition,FieldPosition> ^NumDocs

   TVXVersion --> Int (TermVectorsReader.CURRENT)

   DocumentPosition --> UInt64 (offset in the .tvd file)

   FieldPosition --> UInt64 (offset in the .tvf file)

2. The Document or .tvd file.

   This contains, for each document, the number of fields, a list of the fields with term vector info and finally a list of pointers to the field information in the .tvf (Term Vector Fields) file.

   Document (.tvd) --> TVDVersion<NumFields, FieldNums, FieldPositions> ^NumDocs

   TVDVersion --> Int (TermVectorsReader.FORMAT_CURRENT)

   NumFields --> VInt

   FieldNums --> <FieldNumDelta> ^NumFields

   FieldNumDelta --> VInt

   FieldPositions --> <FieldPositionDelta> ^NumFields-1

   FieldPositionDelta --> VLong

   The .tvd file is used to map out the fields that have term vectors stored and where the field information is in the .tvf file.

3. The Field or .tvf file.

   This file contains, for each field that has a term vector stored, a list of the terms, their frequencies and, optionally, position and offest information.

   Field (.tvf) --> TVFVersion<NumTerms, Position/Offset, TermFreqs> ^NumFields

   TVFVersion --> Int (TermVectorsReader.FORMAT_CURRENT)

   NumTerms --> VInt

   Position/Offset --> Byte

   TermFreqs --> <TermText, TermFreq, Positions?, Offsets?> ^NumTerms

   TermText --> <PrefixLength, Suffix>

   PrefixLength --> VInt

   Suffix --> String

   TermFreq --> VInt

   Positions --> <VInt>^TermFreq

   Offsets --> <VInt, VInt>^TermFreq

   
   

   Notes:

   • Position/Offset byte stores whether this term vector has position or offset information stored.
   • Term text prefixes are shared. The PrefixLength is the number of initial characters from the previous term which must be pre-pended to a term's suffix in order to form the term's text. Thus, if the previous term's text was "bone" and the term is "boy", the PrefixLength is two and the suffix is "y".
   • Positions are stored as delta encoded VInts. This means we only store the difference of the current position from the last position
   • Offsets are stored as delta encoded VInts. The first VInt is the startOffset, the second is the endOffset.

Deleted Documents

The .del file is optional, and only exists when a segment contains deletions.

Although per-segment, this file is maintained exterior to compound segment files.

Deletions (.del) --> [Format],ByteCount,BitCount, Bits | DGaps (depending on Format)

Format,ByteSize,BitCount --> Uint32

Bits --> <Byte> ^ByteCount

DGaps --> <DGap,NonzeroByte> ^{NonzeroBytesCount}

DGap --> VInt

NonzeroByte --> Byte

Format is Optional. -1 indicates DGaps. Non-negative value indicates Bits, and that Format is excluded.

ByteCount indicates the number of bytes in Bits. It is typically (SegSize/8)+1.

BitCount indicates the number of bits that are currently set in Bits.

Bits contains one bit for each document indexed. When the bit corresponding to a document number is set, that document is marked as deleted. Bit ordering is from least to most significant. Thus, if Bits contains two bytes, 0x00 and 0x02, then document 9 is marked as deleted.

DGaps represents sparse bit-vectors more efficiently than Bits. It is made of DGaps on indexes of nonzero bytes in Bits, and the nonzero bytes themselves. The number of nonzero bytes in Bits (NonzeroBytesCount) is not stored.

For example, if there are 8000 bits and only bits 10,12,32 are set, DGaps would be used:

(VInt) 1 , (byte) 20 , (VInt) 3 , (Byte) 1

Limitations

When referring to term numbers, Lucene's current implementation uses a Java int to hold the term index, which means the maximum number of unique terms in any single index segment is ~2.1 billion times the term index interval (default 128) = ~274 billion. This is technically not a limitation of the index file format, just of Lucene's current implementation.

Similarly, Lucene uses a Java int to refer to document numbers, and the index file format uses an Int32 on-disk to store document numbers. This is a limitation of both the index file format and the current implementation. Eventually these should be replaced with either UInt64 values, or better yet,VInt values which have no limit.