badtrash@badbook:/data/badtrash/work/es_spark/sandbox$ ./cluster_setup.sh
Delegating to setup/setup.sh
0. HOST SYSTEM SANITY CHECKS
===============================================================================\n
- This is debian based distro
- Checking is virtualbox is installed
- Checking is vagrant is installed
1. DELETING PREVIOUS VMS (if exist)
===============================================================================\n
2. BUILDING ELK-MS CLUSTER VMs
===============================================================================\n
1.1 BUILDING mes_master VM
Bringing machine 'mes_master' up with 'virtualbox' provider...
==> mes_master: Box 'niceideas-debian-stretch64.box' could not be found. Attempting to find and install...
mes_master: Box Provider: virtualbox
mes_master: Box Version: >= 0
==> mes_master: Box file was not detected as metadata. Adding it directly...
==> mes_master: Adding box 'niceideas-debian-stretch64.box' (v0) for provider: virtualbox
mes_master: Unpacking necessary files from: file:///data/badtrash/work/es_spark/sandbox/setup/vagrant/niceideas-debian-stretch64.box
==> mes_master: Successfully added box 'niceideas-debian-stretch64.box' (v0) for 'virtualbox'!
==> mes_master: Importing base box 'niceideas-debian-stretch64.box'...
==> mes_master: Matching MAC address for NAT networking...
==> mes_master: Setting the name of the VM: mes_master
==> mes_master: Clearing any previously set network interfaces...
==> mes_master: Preparing network interfaces based on configuration...
mes_master: Adapter 1: nat
mes_master: Adapter 2: hostonly
==> mes_master: Forwarding ports...
mes_master: 22 (guest) => 2222 (host) (adapter 1)
==> mes_master: Running 'pre-boot' VM customizations...
==> mes_master: Booting VM...
==> mes_master: Waiting for machine to boot. This may take a few minutes...
mes_master: SSH address: 127.0.0.1:2222
mes_master: SSH username: vagrant
mes_master: SSH auth method: private key
==> mes_master: Machine booted and ready!
==> mes_master: Checking for guest additions in VM...
==> mes_master: Configuring and enabling network interfaces...
==> mes_master: Mounting shared folders...
mes_master: /setup => /data/badtrash/work/es_spark/sandbox/setup/provisionning
mes_master: /vagrant => /data/badtrash/work/es_spark/sandbox/setup/vagrant
==> mes_master: Running provisioner: shell...
mes_master: Running: /tmp/vagrant-shell20170809-17563-1d2bvvw.sh
==> mes_master: mesg:
==> mes_master: ttyname failed
==> mes_master: :
==> mes_master: Inappropriate ioctl for device
==> mes_master: -- PROVISIONNING INFORMATION ---------------------------------------------------
==> mes_master: VM name : mes_master
==> mes_master: VM is master
==> mes_master: IP address is 192.168.10.10
==> mes_master: -- BASIC SETUP -----------------------------------------------------------------
==> mes_master: - Updating the packages
==> mes_master: - Upgrading the appliance
==> mes_master: - Installing a few utility tools
==> mes_master: - Installing the latest OpenJDK
==> mes_master: - Installing build tools
==> mes_master: - Installing swapspace
==> mes_master: - Installing python
==> mes_master: - Installing other Mesos dependencies
==> mes_master: - Installing Spark Dependency : Scala
==> mes_master: - Installing Zookeeper with apt (using plain debian package)
==> mes_master: - Installing ntp
==> mes_master: - Disabling IPv6
==> mes_master: - Creating test user: mes
==> mes_master: - Adding user mes to sudoers
==> mes_master: -- INSTALLING ELASTICSEARCH ----------------------------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading elasticsearch-5.5.0
==> mes_master: - Extracting elasticsearch-5.5.0
==> mes_master: - Installing ElasticSearch
==> mes_master: - Registering test cleaning traps
==> mes_master: - Starting ElasticSearch
==> mes_master: - Checking Elasticsearch startup
==> mes_master: - Stopping Elasticsearch
==> mes_master: -- INSTALLING LOGSTASH ---------------------------------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading logstash-5.5.0
==> mes_master: - Extracting logstash-5.5.0
==> mes_master: - Installing LogStash
==> mes_master: -- INSTALLING KIBANA -----------------------------------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading kibana-5.5.0
==> mes_master: - Extracting kibana-5.5.0
==> mes_master: - Installing Kibana
==> mes_master: - Registering test cleaning traps
==> mes_master: - Starting ElasticSearch
==> mes_master: - Checking Elasticsearch startup
==> mes_master: - Starting Kibana
==> mes_master: - Checking Kibana startup
==> mes_master: - Stopping Kibana
==> mes_master: - Stopping Elasticsearch
==> mes_master: -- INSTALLING CEREBRO ---------------------------------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading cerebro-0.6.5
==> mes_master: - Extracting cerebro-0.6.5
==> mes_master: - Installing Cerebro
==> mes_master: - Registering test cleaning traps
==> mes_master: - Starting ElasticSearch
==> mes_master: - Checking Elasticsearch startup
==> mes_master: - Starting Cerebro
==> mes_master: - Checking Cerebro startup
==> mes_master: - Stopping Cerebro
==> mes_master: - Stopping Elasticsearch
==> mes_master: -- INSTALLING MESOS ------------------------------------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading mesos-1.3.0 (pre-built version for Debian Stretch from niceideas.ch)
==> mes_master: - Extracting mesos-1.3.0
==> mes_master: - Installing mesos
==> mes_master: - Basic environment setup
==> mes_master: - Registering test cleaning traps
==> mes_master: - Starting Mesos Master
==> mes_master: - Checking Mesos master startup
==> mes_master: - Starting Mesos Agent
==> mes_master: - Checking Mesos Agent startup
==> mes_master: - Stopping Mesos Agent
==> mes_master: - Stopping Mesos Master
==> mes_master: - Cleaning build folder
==> mes_master: -- INSTALLING SPARK -----------------------------------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading spark-2.2.0
==> mes_master: - Extracting spark-2.2.0
==> mes_master: - Installing spark
==> mes_master: - Checking Spark Installation
==> mes_master: -- INSTALLING ES-HADOOP (Spark to ES connector) -------------------------------
==> mes_master: - Changing to temp directory
==> mes_master: - Downloading elasticsearch-hadoop-5.5.1
==> mes_master: - Extracting elasticsearch-hadoop-5.5.1
==> mes_master: - Installing eshadoop spark (to spark jar folder)
==> mes_master: -- SETTING UP NTP --------------------------------------------------------------
==> mes_master: - Configuring master NTP server
==> mes_master: - restarting ntp
==> mes_master: -- SETTING UP ZOOKEEPER --------------------------------------------------------
==> mes_master: - Adapting configuration in file zoo.cfg
==> mes_master: - Generating myid
==> mes_master: - restarting zookeeper
==> mes_master: -- SETTING UP ELASTICSEARCH ----------------------------------------------------
==> mes_master: - Increasing system vm.max_map_count setting
==> mes_master: - Creating elasticsearch user (if not exist)
==> mes_master: - Creating elasticsearch data directory in /var/lib/elasticsearch
==> mes_master: - Creating elasticsearch PID file directory
==> mes_master: - Opening rights to logs folder to user elasticsearch
==> mes_master: - Linking elasticsearch logs to /var/log/
==> mes_master: - Simlinking ES config to /usr/local/etc/elasticsearch
==> mes_master: - Simlinking ES binaries and scripts to /usr/local/sbin
==> mes_master: - Copying elasticsearch systemd file
==> mes_master: - Checking Systemd file
==> mes_master: - Adapting configuration in file elasticsearch.yml
==> mes_master: - Addressing issue with multiple interfaces but only one global
==> mes_master: - Adapting configuration in file jvm.options
==> mes_master: - Testing systemd startup - starting Elasticsearch
==> mes_master: - Testing systemd startup - Checking startup
==> mes_master: - Testing systemd startup - stopping elasticsearch
==> mes_master: - Enabling elasticsearch on startup
==> mes_master: -- SETUPING LOGSTASH -----------------------------------------------------------
==> mes_master: - Simlinking logstash binaries and scripts to /usr/local/bin
==> mes_master: !! TODO !!
==> mes_master: - Changing permissions of logstash working directories
==> mes_master: -- SETTING UP KIBANA -----------------------------------------------------------
==> mes_master: - Creating elasticsearch user (if not exist)
==> mes_master: - Simlinking Kibana config to /usr/local/etc/kibana
==> mes_master: - Simlinking Kibana binaries and scripts to /usr/local/sbin
==> mes_master: - Adapting kibana config
==> mes_master: - Copying kibana systemd file
==> mes_master: - Checking Systemd file
==> mes_master: - Testing systemd startup - starting Elasticsearch (required as dependency)
==> mes_master: - Testing systemd startup - starting Kibana
==> mes_master: - Testing systemd startup - Checking startup
==> mes_master: - Testing systemd startup - stopping Kibana
==> mes_master: - Enabling Kibana on startup
==> mes_master: -- SETTING UP CEREBRO ----------------------------------------------------------
==> mes_master: - Creating elasticsearch user (if not exist)
==> mes_master: - Simlinking cerebro config to /usr/local/etc/kibana
==> mes_master: - Simlinking Kibana binaries and scripts to /usr/local/sbin
==> mes_master: - Copying cerebro systemd file
==> mes_master: - Checking Systemd file
==> mes_master: - Testing systemd startup - starting Elasticsearch (required as dependency)
==> mes_master: - Testing systemd startup - starting cerebro
==> mes_master: - Testing systemd startup - Checking startup
==> mes_master: - Testing systemd startup - stopping cerebro
==> mes_master: - Enabling cerebro on startup
==> mes_master: -- SETTING UP MESOS ------------------------------------------------------------
==> mes_master: - Simlinking mesos config to /usr/local/etc/mesos
==> mes_master: - Simlinking mesos binaries to /usr/local/bin
==> mes_master: - Simlinking mesos system binaries to /usr/local/sbin
==> mes_master: - Creating mesos master working directory in /var/lib/mesos/master
==> mes_master: - Creating mesos slave working directory in /var/lib/mesos/slave
==> mes_master: - Creating mesos log directory in /var/log/mesos
==> mes_master: - Copying mesos-init-wrapper script
==> mes_master: - Copying mesos-master systemd file
==> mes_master: - Copying mesos-slave systemd file
==> mes_master: - Creating common mesos environment file
==> mes_master: - Creating mesos master environment file
==> mes_master: - Creating mesos ACLs file to forbid root from submitting tasks.
==> mes_master: - Creating mesos slave environment file
==> mes_master: - Checking mesos slave Systemd file
==> mes_master: - Checking mesos master Systemd file
==> mes_master: - Testing systemd startup - starting mesos-master
==> mes_master: - Testing systemd startup - Checking startup of master
==> mes_master: - Testing systemd startup - stopping mesos-master
==> mes_master: - Enabling mesos-master on startup
==> mes_master: - Testing systemd startup - starting mesos-master
==> mes_master: - Testing systemd startup - starting mesos-slave
==> mes_master: - Testing systemd startup - Checking startup of slave
==> mes_master: - Testing systemd startup - stopping mesos-slave
==> mes_master: - Enabling mesos-slave on startup
==> mes_master: -- SETTING UP SPARK ------------------------------------------------------------
==> mes_master: - Creating spark user (if not exist)
==> mes_master: - Simlinking spark config to /usr/local/etc/spark
==> mes_master: - Simlinking spark binaries to /usr/local/bin
==> mes_master: - Simlinking spark system binaries to /usr/local/sbin
==> mes_master: - Creating spark event log directory in /var/lib/spark/eventlog
==> mes_master: - Opening rights to logs folder to user spark
==> mes_master: - Linking spark logs to /var/log/
==> mes_master: - Copying spark-mesos-dispatcher systemd file
==> mes_master: - Checking Systemd file
==> mes_master: - Defining SPARK_HOME
==> mes_master: - Creating spark environment file
==> mes_master: - Creating spark default configuration file
==> mes_master: - Copying log4j.properties from template
==> mes_master: - Creating start-mesos-dispatcher.sh wrapper
==> mes_master: - Testing systemd startup - starting spark-mesos-dispatcher
==> mes_master: - Testing systemd startup - Checking startup
==> mes_master: - Testing systemd startup - stopping spark-mesos-dispatcher
==> mes_master: - Enabling spark-mesos-dispatcher on startup
==> mes_master: - Generating /etc/hosts for proper DNS resolution
==> mes_master: - !! TODO : remove hardcoding of IPs and make them configurable at Provisionning level !!
==> mes_master: - Changing hostname - THIS REQUIRES REBOOT
==> mes_master: -> Hostname changed successfully => PLEASE REBOOT VM ASAP !
==> mes_master: Running provisioner: reload...
==> mes_master: Attempting graceful shutdown of VM...
==> mes_master: Clearing any previously set forwarded ports...
==> mes_master: Clearing any previously set network interfaces...
==> mes_master: Preparing network interfaces based on configuration...
mes_master: Adapter 1: nat
mes_master: Adapter 2: hostonly
==> mes_master: Forwarding ports...
mes_master: 22 (guest) => 2222 (host) (adapter 1)
==> mes_master: Running 'pre-boot' VM customizations...
==> mes_master: Booting VM...
==> mes_master: Waiting for machine to boot. This may take a few minutes...
mes_master: SSH address: 127.0.0.1:2222
mes_master: SSH username: vagrant
mes_master: SSH auth method: private key
==> mes_master: Machine booted and ready!
==> mes_master: Checking for guest additions in VM...
==> mes_master: Configuring and enabling network interfaces...
==> mes_master: Mounting shared folders...
mes_master: /setup => /data/badtrash/work/es_spark/sandbox/setup/provisionning
mes_master: /vagrant => /data/badtrash/work/es_spark/sandbox/setup/vagrant
==> mes_master: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> mes_master: flag to force provisioning. Provisioners marked to run always will still run.
1.2 BUILDING mes_node1 VM
Bringing machine 'mes_node1' up with 'virtualbox' provider...
==> mes_node1: Importing base box 'niceideas-debian-stretch64.box'...
==> mes_node1: Matching MAC address for NAT networking...
==> mes_node1: Setting the name of the VM: mes_node1
==> mes_node1: Fixed port collision for 22 => 2222. Now on port 2200.
==> mes_node1: Clearing any previously set network interfaces...
==> mes_node1: Preparing network interfaces based on configuration...
mes_node1: Adapter 1: nat
mes_node1: Adapter 2: hostonly
==> mes_node1: Forwarding ports...
mes_node1: 22 (guest) => 2200 (host) (adapter 1)
==> mes_node1: Running 'pre-boot' VM customizations...
==> mes_node1: Booting VM...
==> mes_node1: Waiting for machine to boot. This may take a few minutes...
mes_node1: SSH address: 127.0.0.1:2200
mes_node1: SSH username: vagrant
mes_node1: SSH auth method: private key
==> mes_node1: Machine booted and ready!
==> mes_node1: Checking for guest additions in VM...
==> mes_node1: Configuring and enabling network interfaces...
==> mes_node1: Mounting shared folders...
mes_node1: /setup => /data/badtrash/work/es_spark/sandbox/setup/provisionning
mes_node1: /vagrant => /data/badtrash/work/es_spark/sandbox/setup/vagrant
==> mes_node1: Running provisioner: shell...
mes_node1: Running: /tmp/vagrant-shell20170809-22033-18hbkum.sh
==> mes_node1: mesg: ttyname failed: Inappropriate ioctl for device
==> mes_node1: -- PROVISIONNING INFORMATION ---------------------------------------------------
==> mes_node1: VM name : mes_node1
==> mes_node1: VM is slave
==> mes_node1: Master IP is 192.168.10.10
==> mes_node1: IP address is 192.168.10.11
==> mes_node1: -- BASIC SETUP -----------------------------------------------------------------
==> mes_node1: - Updating the packages
==> mes_node1: - Upgrading the appliance
==> mes_node1: - Installing a few utility tools
==> mes_node1: - Installing the latest OpenJDK
==> mes_node1: - Installing build tools
==> mes_node1: - Installing swapspace
==> mes_node1: - Installing python
==> mes_node1: - Installing other Mesos dependencies
==> mes_node1: - Installing Spark Dependency : Scala
==> mes_node1: - Installing ntp
==> mes_node1: - Disabling IPv6
==> mes_node1: - Creating test user: mes
==> mes_node1: - Adding user mes to sudoers
==> mes_node1: -- INSTALLING ELASTICSEARCH ----------------------------------------------------
==> mes_node1: - Changing to temp directory
==> mes_node1: - Downloading elasticsearch-5.5.0
==> mes_node1: - Extracting elasticsearch-5.5.0
==> mes_node1: - Installing ElasticSearch
==> mes_node1: - Registering test cleaning traps
==> mes_node1: - Starting ElasticSearch
==> mes_node1: - Checking Elasticsearch startup
==> mes_node1: - Stopping Elasticsearch
==> mes_node1: -- INSTALLING MESOS ------------------------------------------------------------
==> mes_node1: - Changing to temp directory
==> mes_node1: - Downloading mesos-1.3.0 (pre-built version for Debian Stretch from niceideas.ch)
==> mes_node1: - Extracting mesos-1.3.0
==> mes_node1: - Installing mesos
==> mes_node1: - Basic environment setup
==> mes_node1: - Registering test cleaning traps
==> mes_node1: - Starting Mesos Master
==> mes_node1: - Checking Mesos master startup
==> mes_node1: - Starting Mesos Agent
==> mes_node1: - Checking Mesos Agent startup
==> mes_node1: - Stopping Mesos Agent
==> mes_node1: - Stopping Mesos Master
==> mes_node1: - Cleaning build folder
==> mes_node1: -- INSTALLING SPARK -----------------------------------------------------------
==> mes_node1: - Changing to temp directory
==> mes_node1: - Downloading spark-2.2.0
==> mes_node1: - Extracting spark-2.2.0
==> mes_node1: - Installing spark
==> mes_node1: - Checking Spark Installation
==> mes_node1: -- INSTALLING ES-HADOOP (Spark to ES connector) -------------------------------
==> mes_node1: - Changing to temp directory
==> mes_node1: - Downloading elasticsearch-hadoop-5.5.1
==> mes_node1: - Extracting elasticsearch-hadoop-5.5.1
==> mes_node1: - Installing eshadoop spark (to spark jar folder)
==> mes_node1: -- SETTING UP NTP --------------------------------------------------------------
==> mes_node1: - Configuring slave NTP server
==> mes_node1: - restarting ntp
==> mes_node1: -- SETTING UP ELASTICSEARCH ----------------------------------------------------
==> mes_node1: - Increasing system vm.max_map_count setting
==> mes_node1: - Creating elasticsearch user (if not exist)
==> mes_node1: - Creating elasticsearch data directory in /var/lib/elasticsearch
==> mes_node1: - Creating elasticsearch PID file directory
==> mes_node1: - Opening rights to logs folder to user elasticsearch
==> mes_node1: - Linking elasticsearch logs to /var/log/
==> mes_node1: - Simlinking ES config to /usr/local/etc/elasticsearch
==> mes_node1: - Simlinking ES binaries and scripts to /usr/local/sbin
==> mes_node1: - Copying elasticsearch systemd file
==> mes_node1: - Checking Systemd file
==> mes_node1: - Adapting configuration in file elasticsearch.yml
==> mes_node1: - Adapting configuration in file elasticsearch.yml - enabling discovery of master
==> mes_node1: - Addressing issue with multiple interfaces but only one global
==> mes_node1: - Adapting configuration in file jvm.options
==> mes_node1: - Testing systemd startup - starting Elasticsearch
==> mes_node1: - Testing systemd startup - Checking startup
==> mes_node1: - Testing systemd startup - stopping elasticsearch
==> mes_node1: - Enabling elasticsearch on startup
==> mes_node1: -- SETTING UP MESOS ------------------------------------------------------------
==> mes_node1: - Simlinking mesos config to /usr/local/etc/mesos
==> mes_node1: - Simlinking mesos binaries to /usr/local/bin
==> mes_node1: - Simlinking mesos system binaries to /usr/local/sbin
==> mes_node1: - Creating mesos slave working directory in /var/lib/mesos/slave
==> mes_node1: - Creating mesos log directory in /var/log/mesos
==> mes_node1: - Copying mesos-init-wrapper script
==> mes_node1: - Copying mesos-slave systemd file
==> mes_node1: - Creating common mesos environment file
==> mes_node1: - Creating mesos slave environment file
==> mes_node1: - Checking mesos slave Systemd file
==> mes_node1: - Testing systemd startup - starting mesos-slave
==> mes_node1: - Testing systemd startup - Checking startup of slave
==> mes_node1: - Testing systemd startup - stopping mesos-slave
==> mes_node1: - Enabling mesos-slave on startup
==> mes_node1: -- SETTING UP SPARK ------------------------------------------------------------
==> mes_node1: - Creating spark user (if not exist)
==> mes_node1: - Simlinking spark config to /usr/local/etc/spark
==> mes_node1: - Simlinking spark binaries to /usr/local/bin
==> mes_node1: - Simlinking spark system binaries to /usr/local/sbin
==> mes_node1: - Creating spark event log directory in /var/lib/spark/eventlog
==> mes_node1: - Opening rights to logs folder to user spark
==> mes_node1: - Linking spark logs to /var/log/
==> mes_node1: - Defining SPARK_HOME
==> mes_node1: - Creating spark environment file
==> mes_node1: - Creating spark default configuration file
==> mes_node1: - Copying log4j.properties from template
==> mes_node1: - Generating /etc/hosts for proper DNS resolution
==> mes_node1: - !! TODO : remove hardcoding of IPs and make them configurable at Provisionning level !!
==> mes_node1: - Changing hostname - THIS REQUIRES REBOOT
==> mes_node1: -> Hostname changed successfully => PLEASE REBOOT VM ASAP !
==> mes_node1: Running provisioner: reload...
==> mes_node1: Attempting graceful shutdown of VM...
==> mes_node1: Clearing any previously set forwarded ports...
==> mes_node1: Clearing any previously set network interfaces...
==> mes_node1: Preparing network interfaces based on configuration...
mes_node1: Adapter 1: nat
mes_node1: Adapter 2: hostonly
==> mes_node1: Forwarding ports...
mes_node1: 22 (guest) => 2200 (host) (adapter 1)
==> mes_node1: Running 'pre-boot' VM customizations...
==> mes_node1: Booting VM...
==> mes_node1: Waiting for machine to boot. This may take a few minutes...
mes_node1: SSH address: 127.0.0.1:2200
mes_node1: SSH username: vagrant
mes_node1: SSH auth method: private key
==> mes_node1: Machine booted and ready!
==> mes_node1: Checking for guest additions in VM...
==> mes_node1: Configuring and enabling network interfaces...
==> mes_node1: Mounting shared folders...
mes_node1: /setup => /data/badtrash/work/es_spark/sandbox/setup/provisionning
mes_node1: /vagrant => /data/badtrash/work/es_spark/sandbox/setup/vagrant
==> mes_node1: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> mes_node1: flag to force provisioning. Provisioners marked to run always will still run.
1.3 BUILDING mes_node2 VM
Bringing machine 'mes_node2' up with 'virtualbox' provider...
==> mes_node2: Importing base box 'niceideas-debian-stretch64.box'...
==> mes_node2: Matching MAC address for NAT networking...
==> mes_node2: Setting the name of the VM: mes_node2
==> mes_node2: Fixed port collision for 22 => 2222. Now on port 2201.
==> mes_node2: Clearing any previously set network interfaces...
==> mes_node2: Preparing network interfaces based on configuration...
mes_node2: Adapter 1: nat
mes_node2: Adapter 2: hostonly
==> mes_node2: Forwarding ports...
mes_node2: 22 (guest) => 2201 (host) (adapter 1)
==> mes_node2: Running 'pre-boot' VM customizations...
==> mes_node2: Booting VM...
==> mes_node2: Waiting for machine to boot. This may take a few minutes...
mes_node2: SSH address: 127.0.0.1:2201
mes_node2: SSH username: vagrant
mes_node2: SSH auth method: private key
==> mes_node2: Machine booted and ready!
==> mes_node2: Checking for guest additions in VM...
==> mes_node2: Configuring and enabling network interfaces...
==> mes_node2: Mounting shared folders...
mes_node2: /setup => /data/badtrash/work/es_spark/sandbox/setup/provisionning
mes_node2: /vagrant => /data/badtrash/work/es_spark/sandbox/setup/vagrant
==> mes_node2: Running provisioner: shell...
mes_node2: Running: /tmp/vagrant-shell20170809-26262-1njec1.sh
==> mes_node2: mesg:
==> mes_node2: ttyname failed
==> mes_node2: :
==> mes_node2: Inappropriate ioctl for device
==> mes_node2: -- PROVISIONNING INFORMATION ---------------------------------------------------
==> mes_node2: VM name : mes_node2
==> mes_node2: VM is slave
==> mes_node2: Master IP is 192.168.10.10
==> mes_node2: IP address is 192.168.10.12
==> mes_node2: -- BASIC SETUP -----------------------------------------------------------------
==> mes_node2: - Updating the packages
==> mes_node2: - Upgrading the appliance
==> mes_node2: - Installing a few utility tools
==> mes_node2: - Installing the latest OpenJDK
==> mes_node2: - Installing build tools
==> mes_node2: - Installing swapspace
==> mes_node2: - Installing python
==> mes_node2: - Installing other Mesos dependencies
==> mes_node2: - Installing Spark Dependency : Scala
==> mes_node2: - Installing ntp
==> mes_node2: - Disabling IPv6
==> mes_node2: - Creating test user: mes
==> mes_node2: - Adding user mes to sudoers
==> mes_node2: -- INSTALLING ELASTICSEARCH ----------------------------------------------------
==> mes_node2: - Changing to temp directory
==> mes_node2: - Downloading elasticsearch-5.5.0
==> mes_node2: - Extracting elasticsearch-5.5.0
==> mes_node2: - Installing ElasticSearch
==> mes_node2: - Registering test cleaning traps
==> mes_node2: - Starting ElasticSearch
==> mes_node2: - Checking Elasticsearch startup
==> mes_node2: - Stopping Elasticsearch
==> mes_node2: -- INSTALLING MESOS ------------------------------------------------------------
==> mes_node2: - Changing to temp directory
==> mes_node2: - Downloading mesos-1.3.0 (pre-built version for Debian Stretch from niceideas.ch)
==> mes_node2: - Extracting mesos-1.3.0
==> mes_node2: - Installing mesos
==> mes_node2: - Basic environment setup
==> mes_node2: - Registering test cleaning traps
==> mes_node2: - Starting Mesos Master
==> mes_node2: - Checking Mesos master startup
==> mes_node2: - Starting Mesos Agent
==> mes_node2: - Checking Mesos Agent startup
==> mes_node2: - Stopping Mesos Agent
==> mes_node2: - Stopping Mesos Master
==> mes_node2: - Cleaning build folder
==> mes_node2: -- INSTALLING SPARK -----------------------------------------------------------
==> mes_node2: - Changing to temp directory
==> mes_node2: - Downloading spark-2.2.0
==> mes_node2: - Extracting spark-2.2.0
==> mes_node2: - Installing spark
==> mes_node2: - Checking Spark Installation
==> mes_node2: -- INSTALLING ES-HADOOP (Spark to ES connector) -------------------------------
==> mes_node2: - Changing to temp directory
==> mes_node2: - Downloading elasticsearch-hadoop-5.5.1
==> mes_node2: - Extracting elasticsearch-hadoop-5.5.1
==> mes_node2: - Installing eshadoop spark (to spark jar folder)
==> mes_node2: -- SETTING UP NTP --------------------------------------------------------------
==> mes_node2: - Configuring slave NTP server
==> mes_node2: - restarting ntp
==> mes_node2: -- SETTING UP ELASTICSEARCH ----------------------------------------------------
==> mes_node2: - Increasing system vm.max_map_count setting
==> mes_node2: - Creating elasticsearch user (if not exist)
==> mes_node2: - Creating elasticsearch data directory in /var/lib/elasticsearch
==> mes_node2: - Creating elasticsearch PID file directory
==> mes_node2: - Opening rights to logs folder to user elasticsearch
==> mes_node2: - Linking elasticsearch logs to /var/log/
==> mes_node2: - Simlinking ES config to /usr/local/etc/elasticsearch
==> mes_node2: - Simlinking ES binaries and scripts to /usr/local/sbin
==> mes_node2: - Copying elasticsearch systemd file
==> mes_node2: - Checking Systemd file
==> mes_node2: - Adapting configuration in file elasticsearch.yml
==> mes_node2: - Adapting configuration in file elasticsearch.yml - enabling discovery of master
==> mes_node2: - Addressing issue with multiple interfaces but only one global
==> mes_node2: - Adapting configuration in file jvm.options
==> mes_node2: - Testing systemd startup - starting Elasticsearch
==> mes_node2: - Testing systemd startup - Checking startup
==> mes_node2: - Testing systemd startup - stopping elasticsearch
==> mes_node2: - Enabling elasticsearch on startup
==> mes_node2: -- SETTING UP MESOS ------------------------------------------------------------
==> mes_node2: - Simlinking mesos config to /usr/local/etc/mesos
==> mes_node2: - Simlinking mesos binaries to /usr/local/bin
==> mes_node2: - Simlinking mesos system binaries to /usr/local/sbin
==> mes_node2: - Creating mesos slave working directory in /var/lib/mesos/slave
==> mes_node2: - Creating mesos log directory in /var/log/mesos
==> mes_node2: - Copying mesos-init-wrapper script
==> mes_node2: - Copying mesos-slave systemd file
==> mes_node2: - Creating common mesos environment file
==> mes_node2: - Creating mesos slave environment file
==> mes_node2: - Checking mesos slave Systemd file
==> mes_node2: - Testing systemd startup - starting mesos-slave
==> mes_node2: - Testing systemd startup - Checking startup of slave
==> mes_node2: - Testing systemd startup - stopping mesos-slave
==> mes_node2: - Enabling mesos-slave on startup
==> mes_node2: -- SETTING UP SPARK ------------------------------------------------------------
==> mes_node2: - Creating spark user (if not exist)
==> mes_node2: - Simlinking spark config to /usr/local/etc/mesos
==> mes_node2: - Simlinking spark binaries to /usr/local/bin
==> mes_node2: - Simlinking spark system binaries to /usr/local/sbin
==> mes_node2: - Creating spark event log directory in /var/lib/spark/eventlog
==> mes_node2: - Opening rights to logs folder to user spark
==> mes_node2: - Linking spark logs to /var/log/
==> mes_node2: - Defining SPARK_HOME
==> mes_node2: - Creating spark environment file
==> mes_node2: - Creating spark default configuration file
==> mes_node2: - Copying log4j.properties from template
==> mes_node2: - Generating /etc/hosts for proper DNS resolution
==> mes_node2: - !! TODO : remove hardcoding of IPs and make them configurable at Provisionning level !!
==> mes_node2: - Changing hostname - THIS REQUIRES REBOOT
==> mes_node2: -> Hostname changed successfully => PLEASE REBOOT VM ASAP !
==> mes_node2: Running provisioner: reload...
==> mes_node2: Attempting graceful shutdown of VM...
==> mes_node2: Clearing any previously set forwarded ports...
==> mes_node2: Clearing any previously set network interfaces...
==> mes_node2: Preparing network interfaces based on configuration...
mes_node2: Adapter 1: nat
mes_node2: Adapter 2: hostonly
==> mes_node2: Forwarding ports...
mes_node2: 22 (guest) => 2201 (host) (adapter 1)
==> mes_node2: Running 'pre-boot' VM customizations...
==> mes_node2: Booting VM...
==> mes_node2: Waiting for machine to boot. This may take a few minutes...
mes_node2: SSH address: 127.0.0.1:2201
mes_node2: SSH username: vagrant
mes_node2: SSH auth method: private key
==> mes_node2: Machine booted and ready!
==> mes_node2: Checking for guest additions in VM...
==> mes_node2: Configuring and enabling network interfaces...
==> mes_node2: Mounting shared folders...
mes_node2: /setup => /data/badtrash/work/es_spark/sandbox/setup/provisionning
mes_node2: /vagrant => /data/badtrash/work/es_spark/sandbox/setup/vagrant
==> mes_node2: Machine already provisioned. Run `vagrant provision` or use the `--provision`
==> mes_node2: flag to force provisioning. Provisioners marked to run always will still run.
3. ELK-MS CLUSTER SANITY CHECKS
===============================================================================
(TODO To Be Implemented)