Preparation before installation
Turn off the firewall chkconfig iptables off && setenforce 0 Create user groupadd realtime && useradd realtime && usermod -a -G realtime realtime Create a working directory and empower mkdir/export mkdir/export/servers chmod 755 -R/export Switch to realtime user su realtime
1. Install a zookeeper cluster and start the zookeeper cluster.
2. Upload the installation package of storm and decompress it
3. Modify the storm configuration file storm.yaml
#The ip address of the zookeeper cluster host used storm.zookeeper.servers: -"weekend01" -"weekend02" -"weekend03" #storm the host name where nimbus is located nimbus.host: "weekend04"
#Specify a supervisor can start several work processes, each process occupies a port, if not configured, the default is 4 work processes. supervisor.slots.ports -6701 -6702 -6703 -6704 -6705 #Specify the maximum available memory size for nimbus to start JVM nimbus.childopts:"-Xmx1024m" #Specify the maximum available memory size of the supervisor to start the JVM supervisor.childopts:"-Xmx1024m" #Specify the maximum available memory size for each worker to start JVM on the supervisor node worker.childopts:"-Xmx1024m" #Specify the maximum available memory size of the ui startup JVM, the ui service is generally on the same node as the nimbus ui.childopts:"-Xmx1024m" and so on....
Sina Weibo: intsmaze Liu Yangyang brother
In the storm/bin directory
First start nimbus on the nimbus host
In this way, if you close the session window, the nimbus service will be closed.
Enter the following command to make nimbus process in the background; run.
nohup ./storm nimbus 1>/dev/null 2>&1 &
You can see from the print station that the command is actually executed to start a java class: Running:/export/servers/jdk/bin/java -server backtype.storm.daemon.nimbus
nohup ./storm ui 1>/dev/null 2>&1 &
On the supervisor host
nohup ./storm supervisor 1>/dev/null 2>&1 &
Running:/export/servers/jdk/bin/java -server backtype.storm.daemon.supervisor
After launching ./storm ui, you can access the graphical interface.
Visit nimbus.host:/8080, you can see the ui interface of storm.
4. Storm common operation commands
There are many simple and useful commands to manage the topology. They can submit, kill, disable, and rebalance the topology.
Submit task command format: storm jar [jar path] [topology package name. topology class name] [topology name]
bin/storm jar examples/storm-starter/storm-starter-topologies-0.10.0.jar storm.starter.WordCountTopology wordcount Kill task command format: storm kill [topology name] -w 10 (can pass-when executing the kill command w [Waiting seconds] Specify the waiting time after the topology is deactivated)
storm kill topology-name -w 10 Command format for deactivating tasks: storm deactivte [topology name]
storm deactivte topology-name
We can suspend or deactivate the running topology. When the topology is deactivated, all distributed tuples will be processed, but the nextTuple method of spouts will not be called. To destroy a topology, you can use the kill command. It will destroy a topology in a safe way, first deactivate the topology, and allow the topology to complete the current data flow while waiting for the topology message.
Enable task command format: storm activate [topology name]
storm activate topology-name redeployment task command format: storm rebalance [topology name]
storm rebalance topology-name rebalance allows you to redistribute cluster tasks. This is a very powerful command. For example, you add nodes to a running cluster. The rebalance command will deactivate the topology, then redistribute workers after the corresponding timeout period, and restart the topology.
5. Familiar with the process and logs of Storm cluster
5.1, view the log information of nimbus
On the nimbus server cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/nimbus.log
5.2, view ui operation log information
On the ui server, usually the same server as nimbus cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/ui.log
5.3, view supervisor running log information
On the supervisor service cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/supervisor.log
5.4. View the worker running log information on the supervisor
On the supervisor service cd/export/servers/storm/logs tail -100f/export/servers/storm/logs/worker-6702.log