Eclipse Maven 非本地Spark
maven
Maven: Apache Maven是一个开源的项目管理和构建工具,主要用于Java项目。适合需要自动化项目构建和依赖管理的开发者。特点包括约定优于配置、易于使用和社区驱动。
项目地址:https://gitcode.com/gh_mirrors/ma/maven
·
本地Spark跑不会出现什么问题,但是切换到服务器上各种问题。
异常
java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.f$3 of type org.apache.spark.api.java.function.FlatMapFunction in instance of org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1
修改pom.xml文件在project中添加下方xml,然后右击项目执行-> Run as -> maven claen && maven install
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.1</version>
<configuration>
<source>1.8</source>
<target>1.8</target>
</configuration>
</plugin>
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
添加jars路径
String jarPath = "打包成功之后的项目jar文件"
JavaSparkContext jsc = new JavaSparkContext(new SparkConf()
.setMaster("spark://服务器地址:端口")
.setJars(new String[] {jarPath})
.setAppName("Classification"));
然后在执行
END:
Maven: Apache Maven是一个开源的项目管理和构建工具,主要用于Java项目。适合需要自动化项目构建和依赖管理的开发者。特点包括约定优于配置、易于使用和社区驱动。
最近提交(Master分支:2 个月前 )
19c7fa2d
Bumps [org.codehaus.plexus:plexus-testing](https://github.com/codehaus-plexus/plexus-testing) from 2.0.1 to 2.0.2.
- [Release notes](https://github.com/codehaus-plexus/plexus-testing/releases)
- [Commits](https://github.com/codehaus-plexus/plexus-testing/compare/plexus-testing-2.0.1...plexus-testing-2.0.2)
---
updated-dependencies:
- dependency-name: org.codehaus.plexus:plexus-testing
dependency-version: 2.0.2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 21 小时前
0516e652
Bumps [actions/checkout](https://github.com/actions/checkout) from 6.0.0 to 6.0.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/1af3b93b6815bc44a9784bd300feb67ff0d1eeb3...8e8c483db84b4bee98b60c0593521ed34d9990e8)
---
updated-dependencies:
- dependency-name: actions/checkout
dependency-version: 6.0.1
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 21 小时前
AtomGit 是由开放原子开源基金会联合 CSDN 等生态伙伴共同推出的新一代开源与人工智能协作平台。平台坚持“开放、中立、公益”的理念,把代码托管、模型共享、数据集托管、智能体开发体验和算力服务整合在一起,为开发者提供从开发、训练到部署的一站式体验。
更多推荐



所有评论(0)