ElasticSearch(2)Version Upgrade 6.2.4 and Cluster

ElasticSearch(2)Version Upgrade 6.2.4 and Cluster

Try to get Latest Documents here
https://github.com/elastic/elasticsearch

Beats, Logstash, ElasticSearch, Kibana

More documents
https://www.elastic.co/guide/index.html

First of All, Install Elastic Search
https://www.elastic.co/downloads/elasticsearch
Download the tar file from there https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.2.4.tar.gz
> tar zxvf elasticsearch-6.2.4.tar.gz
Put it in the working directory
> sudo ln -s /Users/hluo/tool/elasticsearch-6.2.4 /opt/elasticsearch-6.2.4
> sudo ln -s /opt/elasticsearch-6.2.4 /opt/elasticsearch
Add to the Path

Install X-Pack into ElasticSearch
> bin/elasticsearch-plugin install x-pack

Start ElasticSearch
> bin/elasticsearch

Set up password
> bin/x-pack/setup-passwords auto
Changed password for user kibana
PASSWORD kibana = sBqGvem69d2jeal6rf3j
Changed password for user logstash_system
PASSWORD logstash_system = xn9ElgHqJTbZFcmDPQFy
Changed password for user elastic
PASSWORD elastic = Y2r19FsAXIEtjp26HTcE

Visit
http://localhost:9200/

Username elastic and password

Install Kibana
Go here and download the MAC version for myself https://www.elastic.co/downloads/kibana
> tar zxvf kibana-6.2.4-darwin-x86_64.tar.gz
> mv kibana-6.2.4-darwin-x86_64 ~/tool/kibana-6.2.4
> sudo ln -s /Users/hluo/tool/kibana-6.2.4 /opt/kibana-6.2.4
> sudo ln -s /opt/kibana-6.2.4 /opt/kibana
Add to the Path

Install Kibana Plugin
> bin/kibana-plugin install x-path

Config the username/pasword Kibana connect to ElasticSearch
> vi config/kibana.yml
elasticsearch.username: "kibana"
elasticsearch.password: "sBqGvem69d2jeal6rf3j"

Start Kibana
> bin/kibana

Visit the Page
http://localhost:5601/

Use the elastic username and password.

Try the things in DEV Tools
POST /inspections/report
{
    "business_address": "660 Sacramento St",
    "business_city": "San Francisco",
    "business_id": "2228",
    "business_latitude": "37.793698",
    "business_location": {
      "type": "Point",
      "coordinates": [
        -122.403984,
        37.793698
      ]
    },
    "business_longitude": "-122.403984",
    "business_name": "Tokyo Express",
    "business_postal_code": "94111",
    "business_state": "CA",
    "inspection_date": "2016-02-04T00:00:00.000",
    "inspection_id": "2228_20160204",
    "inspection_type": "Routine",
    "inspection_score": 96,
    "risk_category": "Low Risk",
    "violation_description": "Unclean nonfood contact surfaces",
    "violation_id": "2228_20160204_103142"
}

Search for the Content
GET inspections/report/_search

Update Something with an ID
PUT inspections/report/9527
{
    "business_address": "660 Sacramento St",
    "business_city": "San Francisco",
    "business_id": "2228",
    "business_latitude": "37.793698",
    "business_location": {
      "type": "Point",
      "coordinates": [
        -122.403984,
        37.793698
      ]
    },
    "business_longitude": "-122.403984",
    "business_name": "Tokyo Express",
    "business_postal_code": "94111",
    "business_state": "CA",
    "inspection_date": "2016-02-04T00:00:00.000",
    "inspection_id": "2228_20160204",
    "inspection_type": "Routine",
    "inspection_score": 96,
    "risk_category": "Low Risk",
    "violation_description": "Unclean nonfood contact surfaces",
    "violation_id": "2228_20160204_103142"
}

Delete Index
DELETE inspections

Create the index ahead
PUT /inspections
{
  "settings": {
    "index.number_of_shards": 1,
    "index.number_of_replicas": 0
  }
}

Bulk API
One Simple Example
POST _bulk
{ "index" : { "_index" : "test", "_type" : "_doc", "_id" : "1" } }
{ "field1" : "value1" }
{ "delete" : { "_index" : "test", "_type" : "_doc", "_id" : "2" } }
{ "create" : { "_index" : "test", "_type" : "_doc", "_id" : "3" } }
{ "field1" : "value3" }
{ "update" : {"_id" : "1", "_type" : "_doc", "_index" : "test"} }
{ "doc" : {"field2" : "value2"} }

POST inspections/report/_bulk
{ "index": { "_id": 1 }}
{ "business_address": "315 California St", "business_city": "San Francisco", "business_id":"24936","business_latitude":"37.793199","business_location":{"type":"Point","coordinates":[-122.400152,37.793199]},"business_logitude":"-122.400152","business_name":"San Francisco Soap Company","business_postal_code":"94104","business_state":"CA","inspection_date":"2016-06-09T00:00:00.000","inspection_id":"24936_20160609","inspection_score":77,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violcation_description":"Imporoper food labeling or menu misrepresentation","violation_id":"24936_20160609_103141"}
{ "index": { "_id":2 }}
{"business_address":"10 Mason St","business_city":"San Francisco","business_id":"60354","business_latitude":"37.783527","business_location":{"type":"Point","coordinates":[-122.409061,37.783527]},"business_longitude":"-122.409061","business_name":"Soup Unlimited","business_postal_code":"94102","business_state":"CA","inspection_date":"2016-11-23T00:00:00.000","inspection_id":"60354_20161123","inspection_type":"Routine","inspection_score":95}
{ "index": { "_id":3 }}
{"business_address":"2872 24th St","business_city":"San Francisco","business_id":"1797","business_latitude":"37.752807","business_location":{"type":"Point","coordinates":[-122.409752,37.752807]},"business_longitude":"-122.409752","business_name":"TIO CHILOS GRILL","business_postal_code":"94110","business_state":"CA","inspection_date":"2016-07-05T00:00:00.000","inspection_id":"1797_20160705","inspection_score":90,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unclean nonfood contact surfaces","violation_id":"1797_20160705_103142"}
{ "index": { "_id": 4 }}
{"business_address":"1661 Tennessee St Suite 38","business_city":"San Francisco Whard Restaurant","business_id":"66198","business_latitude":"37.75072","business_location":{"type":"Point","coordinates":[-122.388478,37.75072]},"business_longitude":"-122.388478","business_name":"San Francisco Restaurant","business_postal_code":"94107","business_state":"CA","inspection_date":"2016-05-27T00:00:00.000","inspection_id":"66198_20160527","inspection_type":"Routine","inspection_score":56}
{ "index": { "_id": 5 }}
{"business_address":"2162 24th Ave","business_city":"San Francisco","business_id":"5794","business_latitude":"37.747228","business_location":{"type":"Point","coordinates":[-122.481299,37.747228]},"business_longitude":"-122.481299","business_name":"Soup House","business_phone_number":"[phone]","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09-07T00:00:00.000","inspection_id":"5794_20160907","inspection_score":96,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"}
{ "index":{ "_id": 6 }}
{"business_address":"2162 24th Ave","business_city":"San Francisco","business_id":"5794","business_latitude":"37.747228","business_location":{"type":"Point","coordinates":[-122.481299,37.747228]},"business_longitude":"-122.481299","business_name":"Soup-or-Salad","business_phone_number":"[phone]","business_postal_code":"94116","business_state":"CA","inspection_date":"2016-09-07T00:00:00.000","inspection_id":"5794_20160907","inspection_score":96,"inspection_type":"Routine - Unscheduled","risk_category":"Low Risk","violation_description":"Unapproved or unmaintained equipment or utensils","violation_id":"5794_20160907_103144"}

Search on the API
GET inspections/report/_search
These will return:
{
  "took": 0,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 6,
    "max_score": 1,
    "hits": [
      {
        "_index": "inspections",
        "_type": "report",
        "_id": "1",
        "_score": 1,
        "_source": {
          "business_address": "315 California St",
          "business_city": "San Francisco",
          "business_id": "24936",
          "business_latitude": "37.793199",
          "business_location": {
            "type": "Point",
            "coordinates": [
              -122.400152,
              37.793199

Search with One Property
GET inspections/report/_search
{
  "query":{
    "match":{
      "business_name":"soup"
    }
  }
}

Match Phrase
GET inspections/report/_search
{
  "query":{
    "match_phrase": {
      "business_name": "san francisco"
    }
  }
}

Search 2 properties with AND
//bool - must, should, must_not, filter
GET inspections/report/_search
{
  "query":{
    "bool":{
      "must”:[
        {
          "match":{
            "business_name":"soup"
          }
        },
        {
          "match":{
            "violation_description": "Unapproved"
          }
        }
      ]
    }
  }
}

Must Not
GET inspections/report/_search
{
  "query": {
    "bool":{
      "must_not":[
        {
          "match":{
            "business_name":"soup"
          }
        }
      ]
    }
  }
}

Should OR Search with Boost
GET inspections/report/_search
{
  "query":{
    "bool":{
      "should":[
        {
          "match_phrase":{
            "business_name":{
              "query":"soup",
              "boost":3
            }
          }
        },
        {
          "match":{
            "inspection_type":{
              "query":"routine"
            }
          }
        }
      ]
    }
  }
}

Search with High Lights
GET inspections/report/_search
{
  "query":{
    "match":{
      "business_name":"soup"
    }
  },
  "highlight": {
    "fields": {
      "business_name":{}
    }
  }
}

Aggregations with Search
GET inspections/report/_search
{
  "query": {
    "match": {
      "business_name": "soup"
    }
  },
  "aggregations": {
    "healthscore":{
      "range": {
        "field": "inspection_score",
        "ranges": [
          {
            "key": "0-80",
            "from": 0,
            "to": 80
          },
          {
            "key" : "81-90",
            "from": 81,
            "to": 90
          },
          {
            "key": "91-100",
            "from" : 91,
            "to" : 100
          }
        ]
      }
    }
  }
}

Check the Index Mapping
GET inspections/_mapping/report


ElasticSearch
https://www.elastic.co/webinars/get-started-with-elasticsearch

Kibana
https://www.elastic.co/webinars/getting-started-kibana?baymax=default&elektra=docs&storm=top-video

Logstash
https://www.elastic.co/webinars/getting-started-logstash?baymax=default&elektra=docs&storm=top-video


References:
http://www.cnblogs.com/dennisit/p/4132269.html
http://guoze.me/2015/04/04/elasticsearch-optimization/

redis logstash, elastic search kibana
http://nkcoder.github.io/blog/20141031/elkr-log-platform-deploy/
http://nkcoder.github.io/blog/20141106/elkr-log-platform-deploy-ha/

kibana
http://kibana.logstash.es/content/

http://sillycat.iteye.com/blog/2153824