The exploration of the principle of jar triggered by the submission of the storm project jar package

The exploration of the principle of jar triggered by the submission of the storm project jar package

Preface: When developing the storm project, submit the project jar package. When all the dependent third-party jar packages were entered and submitted to the storm cluster, it was reported that multiple file errors with the same name were found. This started a deep understanding of the jar package.

java.lang.RuntimeException: Found multiple defaults.yaml resources.

You're probably bundling the Storm jars with your topology jar.

[jar:file:/home/hadoop/app/storm/lib/storm-core-0.9.6.jar!/defaults.yaml,

jar:file:/home/hadoop/stormApi-wordcount-1.0-SNAPSHOT-jar-with-dependencies.jar!/defaults.yaml]

This shows that there is the storm jar package in the stom cluster environment, and the jar package we submitted also contains the storm jar package. When reading the configuration file, we found that the same file conflicts caused the startup error.

eclipse jar package

code show as below:

package cn.intsmaze;
public class A {
public static void main(String[] args) {
System.out.println(args[0]);
System.out.println("jar package for java project");
}
}
 
package cn.intsmaze;
import java.util.HashMap;
import java.util.Map;
import redis.clients.jedis.Jedis;
public class demo {
public static void main(String[] args)
{
System.out.println("Java project calls a third-party jar package");
Jedis jedis = new Jedis("localhost", 6379);
Map map=new HashMap();
map.put("11", "1");
d.jedis.hmset("33", map);
d.jedis.close();
}
}

The java project of the above code is as follows:

Use eclipse to package the project into a jar package:

Select the entry class of this jar package

After packaging the above code as A.jar, eclipse will automatically generate the following file for us located in META-INF:

MANIFEST.MF file

Manifest-Version: 1.0

Main-Class: cn.intsmaze.A Specify the entry class here

Package the above code as B.jar

MANIFEST.MF file

Manifest-Version: 1.0

Main-Class: cn.intsmaze.demo Specify the entry class here

Because the third-party jar package under the project lib is quoted here, but the jar package is not under the classpath path, the class cannot be found.

Just increase the calsspath value in the MANIFEST.MF file.

Manifest-Version: 1.0

Main-Class: cn.intsmaze.demo

Class-Path: lib/jedis-2.8.0.jar 

Then we put jedis-2.8.0.jar in the same level directory of our B.jar.

It ran successfully here.

note:

The manifest.mf file definition is as follows:

Manifest-Version: 1.0

Main-Class: com.Task

Class-Path: lib/dom4j-1.6.1.jar lib/jaxen-1.1-beta-7.jar

Note:

<1> The last line of the manifest.mf file must be a blank line.
<2> A space separates lib/dom4j-1.6.1.jar and lib/jaxen-1.1-beta-7.jar.
<3> There is a space after each colon.

Maven packaging:

code show as below:

package cn.intsmaze;
public class A {
public static void main(String[] args) {
System.out.println(args[0]);
System.out.println("jar package for java project");
}
}
 
package cn.intsmaze;
import java.util.HashMap;
import java.util.Map;
import redis.clients.jedis.Jedis;
public class demo {
public static void main(String[] args)
{
System.out.println("Java project calls a third-party jar package");
Jedis jedis = new Jedis("localhost", 6379);
Map map=new HashMap();
map.put("11", "1");
d.jedis.hmset("33", map);
d.jedis.close();
}
}

The above code when only the following attributes are configured in their pom file:

 <dependencies>
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>2.7.3</version>
</dependency>
</dependencies>

At this time, use maven to package. Observe his directory structure.

This jar package does not contain the dependent jedis jar package, and the entry class and Class-Path are not specified in the manifest.mf file (where does the program load the dependent jedis.jar package).

If you want to run this jar package successfully, we have to set Main-Class and Class-Path in manifest.mf.

Method 2: We can set in the pom file to include the jar packages it depends on when the project is jar packaged, and specify Main-Class at the same time.

 <build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>cn.intsmaze.demo.RedisDemo</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>

Observe his directory structure.

At the same time, it will also open a jar package without dependencies (the effect is the same as without adding plug-in settings)

Problems when packaging the storm project:

 <dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.5</version>
</dependency>
</dependencies>

This project will only rely on the jdk jar package and the storm jar package, and does not rely on other third-party jar packages. We typed this project into the jar package. According to the above, it is obvious that we know that the jar package does not contain the dependent storm jar package, and The Main-Class and Class-Path are also not specified in the manifest.mf file.

But submit it to the storm cluster, it will run, this is because the path of the class-path of the stom cluster has jdk and storm jar packages (we use the java -jar command is jdk or something.). About Class-Path I can understand, but the entry class is not specified. How to start it, I have to study slowly.

When the above storm project needs to rely on a third-party mysql package, we must require the dependent jar package to be entered in the pom file, otherwise we have to specify in manifest.mf where to load the dependent mysql package, and also Upload the mysql package to the specified directory on the machine that the jar package we are playing will run on so that it can be loaded according to the Class-Path.

<dependencies>
<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<version>0.9.5</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.27</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>cn.intsmaze.helloworld.WordCountTopologyMain</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<configuration>
<source>1.7</source>
<target>1.7</target>
</configuration>
</plugin>
</plugins>
</build>

Here, all the dependent third-party jar packages are entered.

But when we submitted the jar package to the storm cluster, the following error was reported:

java.lang.RuntimeException: Found multiple defaults.yaml resources.
You're probably bundling the Storm jars with your topology jar.
[jar:file:/home/hadoop/app/storm/lib/storm-core-0.9.6.jar!/defaults.yaml,
jar:file:/home/hadoop/stormApi-wordcount-1.0-SNAPSHOT-jar-with-dependencies.jar!/defaults.yaml]

This shows that there is the storm jar package in the stom cluster environment, and the jar package we submitted also contains the storm jar package. When reading the configuration file, we found that the same file conflicts caused the startup error.

At this time we set as follows:

<dependency>
<groupId>org.apache.storm</groupId>
<artifactId>storm-core</artifactId>
<scope>provided</scope> expect JDK, container or user to provide this dependency
<version>0.9.5</version>
</dependency>

At this time, the dependent storm packages will not be entered into the project, only the dependent mysql packages will be entered.