Solr: Indexing 1

At a high level, the Solr indexing process distills down to three key tasks: 

1. Convert a document from its native format into a format supported by Solr,such as XML or JSON.
2. Add the document to Solr using one of several well-defined interfaces, typically HTTP POST.
3. Configure Solr to apply transformations to the text in the document during indexing.

Solr supports several formats for indexing your document, including XML, JSON,and CSV.

 

 

------------------------------------------------------------------------------------------------------------------------------------

Designing your schema

Specifically, you’ll learn to answer the following key questions about your search application:

• What is a document in your index?
• How is each document uniquely identified?
• What fields in your documents can be searched by users?
• Which fields should be displayed to users in the search results?

 

Document granularity

Determining what a document should represent in your Solr index drives the entire schema-design process. In some cases it’s obvious, such as with our tweet example; the text content is typically short, so each tweet will be a document. But if the content you want to index is large, such as a technical computer book, you may want to treat subsections of a large document as the indexed unit. The key is to think about what your users will want to see in the search results.

 

 

Unique key

Solr doesn’t require a unique identifier for each document, but if one is supplied, Solr uses it to avoid duplicating documents in your index. If a document with the same unique key is added to the index, Solr overwrites the existing record with the latest document.

 

Indexed fields
The best way to think about indexed fields is to ask whether a typical user could develop a meaningful query using that field. Another way to decide if a field should be indexed is to determine if your users would miss it if you did not provide it as a queryable option in your search form.

In addition to enabling searching, you will also need to mark your field as indexed if you need to sort, facet, group by, provide query suggestions for, or execute function queries on values within a field.

Determining which fields to include in the index is specific to every search application.

Stored fields
Although users probably won’t search by editor name to find a book to read, we may  still want to display the editor’s name in the search results. In general, your documents may contain fields that aren’t useful from a search perspective but are still useful for displaying search results. In Solr, these are called stored fields.
Of course, a field may be indexed and stored, which can be searched and displayed in results.

----------------------------------------------------------------------------------------------------------------------------

schema.xml

 

 ------------------------------------------------------------------------------------------------------------------------------------

 Defining fields in schema.xml

 Required field attributes

 

<field name="screen_name"
type="string"
indexed="true"
stored="true" />

 

 

 Multivalued fields

In Solr, fields that can have more than one value per document are called multivalued fields.

 

<field name="link"
type="string"
indexed="true"
stored="true"
multiValued="true"/>

 

 

 

Dynamic fields

In Solr, dynamic fields allow you to apply the same definition to any fields in your documents whose names match either a prefix or suffix pattern, such as s_* or *_s. Dynamic fields use a special naming scheme to apply the same field definition to any fields that match this kind of glob-style pattern. Dynamic fields help address common problems that occur when building search applications, including

• Modeling documents with many fields
• Supporting documents from diverse sources
• Adding new document sources

 

Copy fields

In Solr, copy fields allow you to populate one field from one or more other fields. Specifically, copy fields support two use cases that are common in most search applications:

1. Populate a single catch-all field with the contents of multiple fields.   In most search applications, users are presented with a single search box in which to enter a query. The intent of this approach is to help your users quickly find documents without having to fill out a complicated form; think about how successful a simple search box has been for Google.
   
    
   
    
2. Apply different text analysis to the same field content to create a new searchable field.     Solr copy fields give you the flexibility to enable or disable certain text-analysis features like stemming without having to duplicate storage in your index.

 

Unique key field

If you provide a unique identifier field for each of your documents, Solr will avoid creating duplicates during indexing. In addition, if you plan to distribute your Solr index across multiple servers, you must provide a unique identifier for your documents.

<uniqueKey>id</uniqueKey>

One thing to note is that it’s best to use a primitive field type, such as string or long, for the field you indicate as being the <uniqueKey/> as that ensures Solr doesn’t make any changes to the value during indexing.