Scala signature package has wrong version expected: 5.0 found: 45.0 in scala.package #9539

scabug · 2015-10-28T13:33:21Z

Hi,

I created a small program to use Spark/Hadoop with JDBC and it ran fine on Windows 7 with JDK 8 from Oracle and JDK 7.1 from IBM.
I ran the scala program with java -cp myscalapgm.jar;spark-assembly-1.4.1-hadoop2.6.0.jar;jccdriver.jar myscala.Test
Maybe thats not a good approach but it works on Windows. Now I have moved all the jars to z/OS (Bigendian platform) and get the exception below. A hello world scala works with no problem, but the Spark seams more complex in terms of scala usage.

During execution on z/OS I get the following stacktrace:

 java.lang.ExceptionInInitializerError
  at java.lang.J9VMInternals.ensureError(J9VMInternals.java:134)
  at java.lang.J9VMInternals.recordInitializationFailure(J9VMInternals.java:123)
  at org.apache.spark.sql.jdbc.JDBCRDD$.org$apache$spark$sql$jdbc$JDBCRDD$$getCatalystType(JDBCRDD.scala:62)
  at org.apache.spark.sql.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:137)
  at org.apache.spark.sql.jdbc.JDBCRDD$$anonfun$1.apply(JDBCRDD.scala:137)
  at scala.Option.getOrElse(Option.scala:120)
  at org.apache.spark.sql.jdbc.JDBCRDD$.resolveTable(JDBCRDD.scala:136)
  at org.apache.spark.sql.jdbc.JDBCRelation.<init>(JDBCRelation.scala:128)
  at org.apache.spark.sql.jdbc.DefaultSource.createRelation(JDBCRelation.scala:113)
  at org.apache.spark.sql.sources.ResolvedDataSource$.apply(ddl.scala:269)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114)
  at com.ibm.imstest.IMSAccessSample$delayedInit$body.apply(IMSAccessSample.scala:26)
  at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
  at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
  at scala.App$$anonfun$main$1.apply(App.scala:71)
  at scala.App$$anonfun$main$1.apply(App.scala:71)
  at scala.collection.immutable.List.foreach(List.scala:318)
  at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
  at scala.App$class.main(App.scala:71)
  at com.ibm.imstest.IMSAccessSample$.main(IMSAccessSample.scala:8)
  at com.ibm.imstest.IMSAccessSample.main(IMSAccessSample.scala)
 Caused by: scala.reflect.internal.MissingRequirementError: error while loading package, Scala signature package has
 wrong version
  expected: 5.0
  found: 45.0 in scala.package
  at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.handleError$1(JavaMirrors.scala:535)
  at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:584)
  at scala.reflect.runtime.SymbolLoaders$TopClassCompleter.complete(SymbolLoaders.scala:32)
  at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
  at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:244)
  at scala.reflect.internal.SymbolTable.openPackageModule(SymbolTable.scala:300)
  at scala.reflect.runtime.SymbolLoaders$LazyPackageType.complete(SymbolLoaders.scala:89)
  at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:1231)
  at scala.reflect.internal.Definitions$DefinitionsClass.AnyValClass$lzycompute(Definitions.scala:275)
  at scala.reflect.internal.Definitions$DefinitionsClass.AnyValClass(Definitions.scala:275)
  at scala.reflect.runtime.JavaMirrors$class.init(JavaMirrors.scala:50)
  at scala.reflect.runtime.JavaUniverse.init(JavaUniverse.scala:12)
  at scala.reflect.runtime.JavaUniverse.<init>(JavaUniverse.scala:26)
  at scala.reflect.runtime.package$.universe$lzycompute(package.scala:16)
  at scala.reflect.runtime.package$.universe(package.scala:16)
  at org.apache.spark.sql.types.AtomicType.<init>(DataType.scala:95)
  at org.apache.spark.sql.types.StringType.<init>(StringType.scala:33)
  at org.apache.spark.sql.types.StringType$.<init>(StringType.scala:49)
  at org.apache.spark.sql.types.StringType$.<clinit>(StringType.scala)
  ... 19 more

This is the source code, the crash happens at sqlContext.read...

import org.apache.spark.SparkConf
import org.apache.spark.SparkContext

import org.apache.spark.sql.SQLContext

object IMSAccessSample extends App {
val conf = new SparkConf()
.setMaster("local[1]")
.setAppName("GetStokStat")
.set("spark.executor.memory", "1g")

val sc = new SparkContext(conf)

val sqlContext = new SQLContext(sc)

val optionsStokStat = scala.collection.mutable.Map[String, String]();
optionsStokStat.put("driver", "com.ibm.ims.jdbc.IMSDriver");
optionsStokStat.put("url", "jdbc:ims://172.16.36.226:5559/class://com.ibm.ims.db.databaseviews.DFSSAM09DatabaseView:user=gaebler;password=password;");
optionsStokStat.put("dbtable", "STOKSTAT");

val stokStat = sqlContext.read.format("jdbc").options(optionsStokStat).load();

I have opened a problem for the IBM JDK on z/OS and they state that the problem happens in the unpickleClass method:

  at scala.reflect.runtime.JavaMirrors$JavaMirror.unpickleClass(JavaMirrors.scala:584)

For whatever reason value 45.0 is returned, although the javap -verbose output shows the following:

public final class scala.package                                                
  SourceFile: "package.scala"                                                   
  InnerClasses:                                                                 
       public static #10 of #2; //class scala/package$$anon$1 of class scala/pac
kage                                                                            
       public static #15= #12 of #14; //$hash$colon$colon$=class scala/collectio
n/immutable/Stream$$hash$colon$colon$ of class scala/collection/immutable/Stream
  RuntimeVisibleAnnotations:                                                    
    0: #6(#7=s#8)                                                               
    ScalaSig: length = 0x3                                                      
     05 00 00                                                                   
  minor version: 0                                                              
  major version: 50                                                             
  flags: ACC_PUBLIC, ACC_FINAL, ACC_SUPER                                       
Constant pool:                                                                  
    #1 = Utf8               scala/package                                       
    #2 = Class              #1            //  scala/package                     
    #3 = Utf8               java/lang/Object                                    
    #4 = Class              #3            //  java/lang/Object                  
    #5 = Utf8               package.scala                                       
    #6 = Utf8               Lscala/reflect/ScalaSignature;                      
    #7 = Utf8               bytes

...
So I don't really think its an endianess problem, but it seams only to occur on z/OS and I have no idea how to fix it, if its a JVM bug on z/OS I need something that I can give the IBM JDK support to work with, e.g. a Bytearrayinputstream with 2 more bytes in it or something like that. The must be a reason, that the unpickleClass does not return the 5 as its shown by the javap command.

Thanks for helping, it would be great to get scala to work on z/OS.
Denis.

The text was updated successfully, but these errors were encountered:

scabug · 2015-10-28T13:33:21Z

Imported From: https://issues.scala-lang.org/browse/SI-9539?orig=1
Reporter: Denis Gaebler (gaebler)
Affected Versions: 2.11.6
See #6407

scabug · 2015-10-28T18:02:44Z

@soc said:
Thanks for your bug report! The whole thing is weird, because "5.0" refers to scala-reflects format, unrelated to the class file format, correct me if I'm wrong [~xeno.by]?

scabug · 2015-10-28T21:01:38Z

@xeno-by said:
Yes, 5.0 is the version that scala.reflect expects to find in the @ScalaSignature annotation of the Scala-compiled class in question.

Here we raise the exception: https://github.com/scala/scala/blob/1fbce4612c21a4d0c553ea489b4765494828c09f/src/reflect/scala/reflect/internal/pickling/UnPickler.scala#L55. And this is how we end up there: https://github.com/scala/scala/blob/1fbce4612c21a4d0c553ea489b4765494828c09f/src/reflect/scala/reflect/runtime/JavaMirrors.scala#L619.

scabug · 2015-10-29T02:00:40Z

@retronym said (edited on Oct 29, 2015 2:00:51 AM UTC):
I'm pretty sure this is a character encoding bug.

object Test {
  def main(args: Array[String]): Unit = {
    import scala.reflect.runtime.universe._
    try { typeOf[List[String]] } catch { case ex: Throwable => sys.exit(1) }
  }
}

% scalac sandbox/test.scala && scala -Dfile.encoding=UTF-8 Test; echo $?
0
% scalac sandbox/test.scala && scala -Dfile.encoding=Cp1252 Test; echo $?
0
% scalac sandbox/test.scala && scala -Dfile.encoding=Cp1047 Test; echo $? # IBM EBDIC
1

The culpable code in Scala's runtime reflection is:

https://github.com/scala/scala/blob/v2.11.7/src/reflect/scala/reflect/runtime/JavaMirrors.scala#L616
https://github.com/scala/scala/blob/v2.11.7/src/reflect/scala/reflect/runtime/JavaMirrors.scala#L625

It calls String#getBytes which uses the platform default encoding.

Until we fix this, a workaround is to specify -Dfile.encoding=UTF-8 in the JVM command line. You of course then must be careful if your code reads and writes files, and perhaps change it to use overloads of methods like String#getBytes, InputStreamReader#<init> that explicitly pass in an encoding.

scabug · 2015-10-29T10:09:10Z

Denis Gaebler (gaebler) said:
Thank you, that fixes the issue.
And I am aware that -Dfile.encoding migth introduce other problems.

scabug · 2015-10-29T10:09:29Z

Denis Gaebler (gaebler) said:
-Dfile.encoding= fixes the issue.

scabug · 2016-02-17T07:54:05Z

Ulrich Seelbach (ulrich.seelbach) said:
Playing around with Scala / Spark on z/OS myself and I was getting the same issue.

The -Dfile.encoding switch does work around the issue, but does not fix the underlying problem. The root of all evil is the fact that the original Java designers chose String.getBytes() to use the platform encoding rather than a well-known, fixed one.
I don't fully understand what is going on in the code -- that is, what exactly happens in the "unpickling" operation -- but that operation currently assumes that the encoding is an ASCII variant, and that is not true on z/OS unless the encoding had been overridden with -Dfile.encoding.

So, the clean, no-workaround solution would be to change the two no-argument getBytes invocations in JavaMirrors.scala to the one-argument getBytes with the encoding explicitly specified, e.g., getBytes("ISO8859-1") or getBytes("UTF8").

scabug · 2016-03-15T19:57:09Z

@soc said:
Hi Denis, hi Ulrich, sorry for the delay.
The PR is here: scala/scala#5042

Can you recommend a good way to trigger the bug on non-z/OS platforms?
(So we can be sure that this change fixes it and prevents future regressions.)

scabug · 2016-03-15T20:09:03Z

Ulrich Seelbach (ulrich.seelbach) said:
Hi Simon,
a good way to trigger the bug on non-z/OS platforms would be to set the JVM default encoding to an EBCDIC variant, using, well, the -Dfile.encoding switch. Maybe -Dfile.encoding=IBM-1141 would do the trick. (1141 is the German EBCDIC codepage with Euro support.) At least, it does on Linux with IBM JRE 1.8.0:

[de103252@] $ java -fullversion
java full version JRE 1.8.0 IBM Linux build pxa6480sr2fp10-20160108_01(SR2 FP10)
[de103252@] $ java -Dfile.encoding=IBM-1141 -fullversion
[garbled output in EBCDIC-displayed-as-ASCII snipped]

Thanks!

scabug · 2016-03-16T15:28:34Z

@soc said:
Mhh, I tried that with scala but it didn't trigger an obvious failures ...

scabug · 2016-03-16T19:53:25Z

Ulrich Seelbach (ulrich.seelbach) said:
Phew, guess it could be tricky to reproduce the case on non-z/OS. When you invoked the scala script (that eventually starts the JVM), what did console output look like? With the encoding set to an EBCDIC variant, you should see garbled output. If you don't, maybe the scala script changes the encoding, or does not pass the -D switch to the JVM.
Was the changed code in ScalaMirror actually executed? Could it matter that z/OS is a big-endian platform?
In any event, I am pretty confident that your change does fix the problem on z/OS and does not introduce new problems on non-ASCII platforms. When I have some time, I can try it again on z/OS. Thanks very much Simon!

scabug · 2016-05-12T10:16:34Z

Velkan Rajkumar KS (velkanrajkumar_karunkanshanmugam-at-bmc.com) said:
Hello,

I am a mainframe guy trying to install Apache spark for z/OS 1.5.42 and facing some problem on it. I am new to all this.

When I run the ./bin/spark-shell I am getting the following error with a
"Exception in thread "main" scala.reflect.internal.MissingRequirementError: error while loading package, Scala signature package has wrong version expected: 5.0 found: 45.0 in scala.package" and get some lines of trace.

I tried the options of -Dfile.encoding=Cp1047 and -Dfile.encoding=UTF8 on the spark-env.sh conf parms.
(eg: SPARK_DAEMON_JAVA_OPTS="-Dfile.encoding=Cp1047") .

But still getting the errors. Can you help me get over this issue?

Regards,
Velkan

scabug · 2016-05-12T11:55:44Z

Denis Gaebler (gaebler) said:
Hi Velkan,

you should open a support ticket with IBM.

Regards,
Denis.

scabug · 2016-05-12T11:58:14Z

Velkan Rajkumar KS (velkanrajkumar_karunkanshanmugam-at-bmc.com) said:
Hi Denis,

Yeah I have that open already. the support guy pointed me to this discussion forum.
Anyways I have updated my IBM ticket as well with info I pasted here.

Thanks for your support.

Regards,
Velkan

scabug · 2016-05-12T16:37:40Z

Ulrich Seelbach (ulrich.seelbach) said:
Hi Velkan,
you should try ISO8859-1 rather than Cp1047. It's precisely because Cp1047 is the default on z/OS that it does not work.
Also, setting the options for the Spark daemon is not going to make a difference if you are using the Spark shell.
It's better to configure the code page in spark-defaults.conf. Add the following lines:

spark.driver.extraJavaOptions -Dfile.encoding=ISO8859-1
spark.executor.extraJavaOptions -Dfile.encoding=ISO8859-1

scabug · 2016-05-12T17:45:49Z

Velkan Rajkumar KS (velkanrajkumar_karunkanshanmugam-at-bmc.com) said:
oh that worked. Thank you so much.
I am past that issue now. but the spark.master is not connecting.
spark.master spark://xxx.yyy.com:7077
this is what is coded on the default.conf but its failing to connect.

Warm regards,
Velkan

scabug · 2016-12-21T14:58:02Z

John Goodyear (johngood) said:
Velkan, some time has past, since this last post; did you get past this recent issue with connecting to the master?

scabug closed this as completed May 5, 2016

scabug added reflection usability labels Apr 7, 2017

scabug added this to the 2.12.0-M5 milestone Apr 7, 2017

scabug assigned soc Apr 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala signature package has wrong version expected: 5.0 found: 45.0 in scala.package #9539

Scala signature package has wrong version expected: 5.0 found: 45.0 in scala.package #9539

scabug commented Oct 28, 2015

scabug commented Oct 28, 2015

scabug commented Oct 28, 2015

scabug commented Oct 28, 2015

scabug commented Oct 29, 2015

scabug commented Oct 29, 2015

scabug commented Oct 29, 2015

scabug commented Feb 17, 2016

scabug commented Mar 15, 2016

scabug commented Mar 15, 2016

scabug commented Mar 16, 2016

scabug commented Mar 16, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented Dec 21, 2016

Scala signature package has wrong version expected: 5.0 found: 45.0 in scala.package #9539

Scala signature package has wrong version expected: 5.0 found: 45.0 in scala.package #9539

Comments

scabug commented Oct 28, 2015

scabug commented Oct 28, 2015

scabug commented Oct 28, 2015

scabug commented Oct 28, 2015

scabug commented Oct 29, 2015

scabug commented Oct 29, 2015

scabug commented Oct 29, 2015

scabug commented Feb 17, 2016

scabug commented Mar 15, 2016

scabug commented Mar 15, 2016

scabug commented Mar 16, 2016

scabug commented Mar 16, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented May 12, 2016

scabug commented Dec 21, 2016