ElasticSearch(4)Manually Elasticsearch Cluster

ElasticSearch(4)Manually Elasticsearch Cluster

Follow previous link, start Kibana on ubuntu-master as well.
>bin/elasticsearch -Ecluster.name=laprocluster -Enode.name=elastic1
> bin/kibana

Open Kibana page and go to DEV tool
http://ubuntu-master:5601/app/kibana#/dev_tools/console?_g=()v

In the console, put command
>GET /_cat/health?v

It will give us the cluster information

Node Information Command
>GET /_cat/nodes?v

List the indexes, indices
>GET /_cat/indices?v

Create an index and have a try, create a sample index named customer, just pretty to give the nice JSON response
>PUT /customer?pretty
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "customer"
}

Check the status of the index
>GET /_cat/indices?v
health status index                uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   customer             Y9ieCh28SuSrVLekAMMrng   1   1          0            0       230b           230b

We create an index named customer with 1 primary and 1 replica, there is only 1 node, so it is yellow.

Index and Query a Document
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-query-document.html

>PUT /customer/_doc/1?pretty
{
  "name": "Carl Luo"
}

PUT /customer/_doc/2?pretty
{
  "name": "Leo Luo"
}

PUT /customer/_doc/3?pretty
{
  "name": "Angela Yuqi Luo"
}

We can fetch the Records
>GET /customer/_doc/1?pretty

Delete an index
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-delete-index.html
>DELETE /customer?pretty

Delete a document
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-delete-documents.html
>DELETE /customer/_doc/3?pretty

We can do batch operation as well
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-batch-processing.html

Here is some links to load the sample data and do some search around that
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-explore-data.html

Manually Set Up Cluster
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/setup.html

Start the ElasticSearch in the daemon
> bin/elasticsearch -d -Ecluster.name=laprocluster -Enode.name=elastic1

https://www.elastic.co/guide/en/elasticsearch/reference/7.0/targz.html

We can save the pid as well
>bin/elasticsearch -d -Ecluster.name=laprocluster -Enode.name=elastic1 -p pid

We can do this later to stop that
>pkill -F pid

Some configuration
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/settings.html
Config files Location
elasticsearch.yml for elasticsearch
jvm.options for JVM
Log4j2.properties for logging

We can change the configuration location if we need
>ES_PATH_CONF=/path/to/my/config ./bin/elasticsearch

Some important configurations
cluster.name: sillycatcluster
node.name: elastic1
path.data: /opt/elasticsearch/data
path.logs: /opt/elasticsearch/logs
transport.host: localhost
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0


JVM Option is in
> vi config/jvm.options
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/jvm-options.html

Adding Nodes to Cluster
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/add-elasticsearch-nodes.html

Setting Up 2 Other Nodes : ubuntu-dev2 and ubuntu-dev4
> java -version
java version "1.8.0_171"

Download the Oracle JDK from here https://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Copy the file to my virtual machines.
> scp jdk-8u211-linux-x64.tar.gz ubuntu-dev2:/home/carl/install/

Unzip and place in the working directory
> sudo ln -s /home/carl/tool/jdk1.8.0_211 /opt/jdk1.8.0
> sudo ln -s /opt/jdk1.8.0 /opt/jdk

Add this to .profile
export JAVA_HOME="/opt/jdk"
PATH="/opt/jdk/bin:$PATH"

Install Elasticsearch on Node
> wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.0.1-linux-x86_64.tar.gz

> sudo ln -s /home/carl/tool/elasticsearch-7.0.1 /opt/elasticsearch-7.0.1
> sudo ln -s /opt/elasticsearch-7.0.1 /opt/elasticsearch

I am using virtual box with host only adapter, so I can not make it working with 9300
[2019-05-20T14:35:08,876][INFO ][o.e.t.TransportService   ] [elastic1] publish_address {192.168.56.101:9300}, bound_addresses {192.168.56.101:9300}
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

The vm.max_map_count is the issue, not the IP bind.
http://kael-aiur.com/elk/ES%E9%85%8D%E7%BD%AE%E7%BB%99%E5%A4%96%E9%83%A8%E6%9C%BA%E5%99%A8%E9%80%9A%E8%BF%87ip%E8%AE%BF%E9%97%AE.html

> sudo vi /etc/security/limits.conf
#ftp             -       chroot          /ftp
#@student        -       maxlogins       4
carl            -      nofile          65536
> sudo sysctl -w vm.max_map_count=262144
vm.max_map_count = 262144

The configurations are as follow:
On ubuntu-master:
transport.host: ubuntu-master
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0

discovery.seed_hosts: ["ubuntu-master", "ubuntu-dev2"]

On ubuntu-dev2:
transport.host: ubuntu-dev2
transport.tcp.port: 9300
http.port: 9200
network.host: 0.0.0.0
discovery.seed_hosts: ["ubuntu-master", "ubuntu-dev2"]

We can see the nodes after that
>GET /_cat/nodes?v
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.56.3              7          86   0    0.19    0.24     0.10 mdi       -      elastic2
192.168.56.101           33          45   1    0.27    0.25     0.16 mdi       *      elastic1

Similar to all the configurations, we can add another node as we want.
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.56.3             13          87   0    0.00    0.05     0.06 mdi       -      elastic2
192.168.56.103            9          96  30    0.39    0.10     0.03 mdi       -      elastic3
192.168.56.101           11          45   0    0.08    0.09     0.11 mdi       *      elastic1

Once the cluster is ready, we can access the data like this
http://ubuntu-dev4:9200/customer/_doc/1?pretty
{"_index": "customer","_type": "_doc","_id": "1","_version": 3,"_seq_no": 4,"_primary_term": 1,"found": true,"_source": {"name": "Karl Luo","gender": "male"}}


References:
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-install.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-create-index.html
http://kael-aiur.com/elk/ES%E9%85%8D%E7%BD%AE%E7%BB%99%E5%A4%96%E9%83%A8%E6%9C%BA%E5%99%A8%E9%80%9A%E8%BF%87ip%E8%AE%BF%E9%97%AE.html