Imported From: https://issues.scala-lang.org/browse/SI-5046?orig=1
Reporter: Belinda Gu (belindagu)
Attachments:

• 0001-Add-SerialVersionUID-to-all-classes-of-WrappedArray.patch (created on Sep 30, 2011 5:08:24 AM UTC, 4440 bytes)

Belinda Gu (belindagu) said:
Attached patch should fix the problem.

@soc said:
I wonder what guarantees we have concerning serialization between different versions and how other collection classes handle this (and the consistency of such handling).
Either it should be made to work for everything (probably with tests) or it should be documented somewhere that it is not supported.

@soc said:
Not quite sure what to do here. As far as I see, we haven't marked any other class with a serialization version ID. I think the last breakage was caused by the change from Manifest to ClassTag, so I think it would have been broken either way.

As far as I understand, everything needs to be compatible within a major version, but outside of it, we don't give any guarantees, including (de)serialization compatibility.
I think if we wanted to start something like this, we would first have to start adding serialization tests to every single class, so that we can detect breakage in the first place. If we don't know whether things have changed, it doesn't make sense to just add SerialVersionUIDs, and if stuff hasn't changed, we wouldn't need SerialVersionUIDs to add in the first place, right?

Should I close this ticket as "Out of Scope" for now?

@adriaanm said:
Thanks for looking into this. We actually don't guarantee anything related to serialization...

https://github.com/scala/scala/blob/v2.13.0-M4/src/library/scala/collection/mutable/ArraySeq.scala#L26

mutable.ArraySeq ( renamed since Scala 2.13.0-M4 ) have explicit @SerialVersionUID annotation.

I see this ticket is closed. Is it now included in a release version of 2.12.x or 2.13.x or was it decided not to include it?

It's milestoned 2.13.0-M4, which means the fix should be in 2.13.0 and subsequent versions ,too. Let us know if you find evidence to the contrary.

Got it, thank you.

@SethTisue 2.12.x has the same issue, is it possible to backport solution to 2.12.x?

@dieu I can't think of any reason a PR with a backport wouldn't be merged

I'm seeing this in Scala 2.13.5. Is this supposed to be fixed in that version?

mutable.WrappedArray doesn't even exist in 2.13, so I find it difficult to believe you're seeing it in 2.13.5.

My mistake, I believe I had a mismatching spark-submit Scala version and build.sbt Scala version.

I get the same failure.

I am running on Spark 3.2.1 which in turn runs Scala 2.12.10. My code is also Scala 2.12.10, but I don' see where the issue is coming from or how to address it.

@hamidelmaazouz scala/scala#9166 was merged for 2.12.14, so it's expected that you might run into this on 2.12.10

@hamidelmaazouz I know this here is an old issue. But did you manage to fix your problem? I'm facing the same thing in a spark setup where the communication between the driver and the executors "dies" because:

java.io.InvalidClassException: scala.collection.mutable.WrappedArray$ofRef; local class incompatible: stream classdesc serialVersionUID = 3456489343829468865, local class serialVersionUID = 1028182004549731694

Can see the version of spark 3.2.1 I use is build with scala 2.12.11

<<K9s-Shell>> Pod: jhub-spark/test-spark-a3ad4882bac71f15-exec-4 | Container: spark-kubernetes-executor 
root@test-spark-a3ad4882bac71f15-exec-4:/opt/spark/work-dir# cd ../bin
root@test-spark-a3ad4882bac71f15-exec-4:/opt/spark/bin# ./spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
                        
Using Scala version 2.12.11, OpenJDK 64-Bit Server VM, 1.8.0_144
Branch HEAD
Compiled by user hgao on 2022-01-21T00:35:52Z
Revision 4f25b3f71238a00508a356591553f2dfa89f8290
Url https://github.com/apache/spark
Type --help for more information.

Maybe you have some pointers to how I can fix my issue.

I can confirm that bumping the scala version from 2.12.11 -> 2.12.14 on my spark setup made the error disappear.

> docker run -it -u root --entrypoint bash bazel/third_party/spark:spark_image
root@8635b6f9ca15:/opt/spark/work-dir# cd ../bin/
root@8635b6f9ca15:/opt/spark/bin# ./spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 3.2.1
      /_/
                        
Using Scala version 2.12.14, OpenJDK 64-Bit Server VM, 1.8.0_144
Branch HEAD
Compiled by user hgao on 2022-01-21T00:35:52Z
Revision 4f25b3f71238a00508a356591553f2dfa89f8290
Url https://github.com/apache/spark
Type --help for more information.
root@8635b6f9ca15:/opt/spark/bin# 

Hi @kaaquist,

Sorry for the late reply.

Indeed it's a Scala version problem. Scala AFAIK is not backwards compatible, so you need to match your scala version with the one used in Spark. So you have two solutions:

• Align your version to match that of Scala in Spark
• Build Spark yourself to match your Scala version

Best,