本地Spark跑不会出现什么问题,但是切换到服务器上各种问题。

异常

java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1.f$3 of type org.apache.spark.api.java.function.FlatMapFunction in instance of org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$1$1

修改pom.xml文件在project中添加下方xml,然后右击项目执行-> Run as -> maven claen && maven install

<build> 
  <plugins> 
    <plugin> 
      <groupId>org.apache.maven.plugins</groupId>  
      <artifactId>maven-compiler-plugin</artifactId>  
      <version>3.1</version>  
      <configuration> 
        <source>1.8</source>  
        <target>1.8</target> 
      </configuration> 
    </plugin>  
    <plugin> 
      <artifactId>maven-assembly-plugin</artifactId>  
      <configuration> 
        <descriptorRefs> 
          <descriptorRef>jar-with-dependencies</descriptorRef> 
        </descriptorRefs> 
      </configuration>  
      <executions> 
        <execution> 
          <id>make-assembly</id>  
          <phase>package</phase>  
          <goals> 
            <goal>single</goal> 
          </goals> 
        </execution> 
      </executions> 
    </plugin> 
  </plugins> 
</build>

添加jars路径

String jarPath = "打包成功之后的项目jar文件"
JavaSparkContext jsc = new JavaSparkContext(new SparkConf()
				.setMaster("spark://服务器地址:端口")
				.setJars(new String[] {jarPath})
				.setAppName("Classification"));

然后在执行

END:

 

GitHub 加速计划 / ma / maven
36
2
下载
Maven: Apache Maven是一个开源的项目管理和构建工具,主要用于Java项目。适合需要自动化项目构建和依赖管理的开发者。特点包括约定优于配置、易于使用和社区驱动。
最近提交(Master分支:1 个月前 )
001c86bb Bumps [ch.qos.logback:logback-classic](https://github.com/qos-ch/logback) from 1.5.20 to 1.5.21. - [Release notes](https://github.com/qos-ch/logback/releases) - [Commits](https://github.com/qos-ch/logback/compare/v_1.5.20...v_1.5.21) --- updated-dependencies: - dependency-name: ch.qos.logback:logback-classic dependency-version: 1.5.21 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 8 小时前
f2e9432a The exception was observed in the integration tests of Maven Compiler Plugin when Maven is executed with the `-T4` option. 17 小时前
Logo

新一代开源开发者平台 GitCode,通过集成代码托管服务、代码仓库以及可信赖的开源组件库,让开发者可以在云端进行代码托管和开发。旨在为数千万中国开发者提供一个无缝且高效的云端环境,以支持学习、使用和贡献开源项目。

更多推荐