Storm 0.9.5 cluster installation

Storm 0.9.5 cluster installation

Preparation before installation

Turn off the firewall
chkconfig iptables off && setenforce 0


Create user
groupadd realtime && useradd realtime && usermod -a -G realtime realtime


Create a working directory and empower
mkdir/export
mkdir/export/servers
chmod 755 -R/export


Switch to realtime user
su realtime

1. Install a zookeeper cluster and start the zookeeper cluster.

2. Upload the installation package of storm and decompress it

3. Modify the storm configuration file storm.yaml

Required configuration

#The ip address of the zookeeper cluster host used
storm.zookeeper.servers:
-"weekend01"
-"weekend02"
-"weekend03"
#storm the host name where nimbus is located
nimbus.host: "weekend04"

Optional

#Specify a supervisor can start several work processes, each process occupies a port, if not configured, the default is 4 work processes.
supervisor.slots.ports
-6701
-6702
-6703
-6704
-6705
#Specify the maximum available memory size for nimbus to start JVM
nimbus.childopts:"-Xmx1024m"
#Specify the maximum available memory size of the supervisor to start the JVM
supervisor.childopts:"-Xmx1024m"
#Specify the maximum available memory size for each worker to start JVM on the supervisor node
worker.childopts:"-Xmx1024m"
#Specify the maximum available memory size of the ui startup JVM, the ui service is generally on the same node as the nimbus
ui.childopts:"-Xmx1024m"
and so on....

Sina Weibo: intsmaze Liu Yangyang brother

Start storm

In the storm/bin directory

        First start nimbus on the nimbus host

        ./storm nimbus 

        ./storm ui

In this way, if you close the session window, the nimbus service will be closed.

Enter the following command to make nimbus process in the background; run.

        nohup ./storm nimbus 1>/dev/null 2>&1 &

        You can see from the print station that the command is actually executed to start a java class: Running:/export/servers/jdk/bin/java -server backtype.storm.daemon.nimbus

        nohup ./storm ui 1>/dev/null 2>&1 &

On the supervisor host

        nohup ./storm supervisor 1>/dev/null 2>&1 &

        Running:/export/servers/jdk/bin/java -server backtype.storm.daemon.supervisor

After launching ./storm ui, you can access the graphical interface.

Visit nimbus.host:/8080, you can see the ui interface of storm.

4. Storm common operation commands

 There are many simple and useful commands to manage the topology. They can submit, kill, disable, and rebalance the topology.

Submit task command format: storm jar [jar path] [topology package name. topology class name] [topology name]

bin/storm jar examples/storm-starter/storm-starter-topologies-0.10.0.jar storm.starter.WordCountTopology wordcount Kill task command format: storm kill [topology name] -w 10 (can pass-when executing the kill command w [Waiting seconds] Specify the waiting time after the topology is deactivated)

storm kill topology-name -w 10 Command format for deactivating tasks: storm deactivte [topology name]

storm deactivte topology-name

We can suspend or deactivate the running topology. When the topology is deactivated, all distributed tuples will be processed, but the nextTuple method of spouts will not be called. To destroy a topology, you can use the kill command. It will destroy a topology in a safe way, first deactivate the topology, and allow the topology to complete the current data flow while waiting for the topology message.

Enable task command format: storm activate [topology name]

        storm activate topology-name redeployment task command format: storm rebalance [topology name]

        storm rebalance topology-name rebalance allows you to redistribute cluster tasks. This is a very powerful command. For example, you add nodes to a running cluster. The rebalance command will deactivate the topology, then redistribute workers after the corresponding timeout period, and restart the topology.

5. Familiar with the process and logs of Storm cluster

5.1, view the log information of nimbus

On the nimbus server cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/nimbus.log

5.2, view ui operation log information

On the ui server, usually the same server as nimbus cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/ui.log

5.3, view supervisor running log information

On the supervisor service cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/supervisor.log

5.4. View the worker running log information on the supervisor

On the supervisor service cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/worker-6702.log