Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Get OutOfMemoryError in REPL when using parallel collections #9737

Closed
scabug opened this issue Apr 6, 2016 · 5 comments
Closed

Get OutOfMemoryError in REPL when using parallel collections #9737

scabug opened this issue Apr 6, 2016 · 5 comments
Assignees
@som-snytt
Labels
repl
Milestone
2.11.11

Comments

@scabug
Copy link

scabug commented Apr 6, 2016

I put a simple example for using scan method on collections in the REPL and I observe that I got OutOfMemoryError when using parallel collections. There is not problem, nor wait time when using standard collections.

Here is a listing of what I did:

$ scala -J-Xms1G -J-Xmx2G 
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_77).
Type in expressions for evaluation. Or try :help.

scala> Range(1, 10000).scanLeft(BigInt(1))(_ * _)
res0: scala.collection.immutable.IndexedSeq[scala.math.BigInt] = Vector(1, 1, 2, 6, 24, 120, 720, 5040, 40320, 362880, 3628800, 39916800, 479001600, 6227020800, 87178291200, 1307674368000, 20922789888000, 355687428096000, 6402373705728000, 121645100408832000, 2432902008176640000, 51090942171709440000, 1124000727777607680000, 25852016738884976640000, 620448401733239439360000, 15511210043330985984000000, 403291461126605635584000000, 10888869450418352160768000000, 304888344611713860501504000000, 8841761993739701954543616000000, 265252859812191058636308480000000, 8222838654177922817725562880000000, 263130836933693530167218012160000000, 8683317618811886495518194401280000000, 295232799039604140847618609643520000000, 10333147966386144929666651337523200000000, 3719933267899012174679994481508352...

scala> Range(1, 10000).par.scanLeft(BigInt(1))(_ * _)
java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:3332)
	at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
	at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
	at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
	at java.lang.StringBuilder.append(StringBuilder.java:202)
	at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:267)
	at scala.collection.mutable.StringBuilder.$plus$eq(StringBuilder.scala:175)
	at scala.collection.mutable.StringBuilder.$plus$eq(StringBuilder.scala:31)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
	at scala.collection.immutable.StringOps.foreach(StringOps.scala:29)
	at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
	at scala.collection.immutable.StringOps.map(StringOps.scala:29)
	at scala.tools.nsc.interpreter.Naming$class.unmangle(Naming.scala:35)
	at scala.tools.nsc.interpreter.IMain$naming$.unmangle(IMain.scala:188)
	at scala.tools.nsc.interpreter.IMain$ReplStrippingWriter.stripImpl(IMain.scala:1372)
	at scala.tools.nsc.interpreter.IMain$StrippingWriter$class.strip(IMain.scala:1346)
	at scala.tools.nsc.interpreter.IMain$StrippingTruncatingWriter.strip(IMain.scala:1357)
	at scala.tools.nsc.interpreter.IMain$StrippingTruncatingWriter.clean(IMain.scala:1363)
	at scala.tools.nsc.interpreter.IMain$StrippingTruncatingWriter.write(IMain.scala:1364)
	at java.io.PrintWriter.print(PrintWriter.java:603)
	at scala.tools.nsc.reporters.ConsoleReporter.printMessage(ConsoleReporter.scala:44)
	at scala.tools.nsc.interpreter.ReplReporter.printMessage(ReplReporter.scala:61)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:646)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest$$anonfun$loadAndRunReq$1.apply(IMain.scala:637)
	at scala.reflect.internal.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:31)
	at scala.reflect.internal.util.AbstractFileClassLoader.asContext(AbstractFileClassLoader.scala:19)
	at scala.tools.nsc.interpreter.IMain$WrappedRequest.loadAndRunReq(IMain.scala:637)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:569)
	at scala.tools.nsc.interpreter.IMain.interpret(IMain.scala:565)
	at scala.tools.nsc.interpreter.ILoop.interpretStartingWith(ILoop.scala:807)

That entry seems to have slain the compiler.  Shall I replay
your session? I can re-run each line except the last one.
[y/n]
Abandoning crashed session.

I've boost the memory of JVM the example run, as seen in the run arguments and during execution I don't see any spike in memory consumption.
@scabug
Copy link
Author

scabug commented Apr 6, 2016

Imported From: https://issues.scala-lang.org/browse/SI-9737?orig=1
Reporter: Antoniy Chonkov (antoniy)
Affected Versions: 2.11.8
Other Milestones: 2.12.0-M3

@scabug
Copy link
Author

scabug commented Apr 6, 2016

@SethTisue said:
This needs further investigation to narrow it down and isolate it.

Note that this doesn't OOM: {code}Range(1, 10000).par.scanLeft(BigInt(1))(_ * _).last{code}

So this seems to be a REPL-specific issue having to do with how results are printed.

@scabug
Copy link
Author

scabug commented Apr 7, 2016

@som-snytt said:
Works in 2.12.0-M3.

@scabug
Copy link
Author

scabug commented Apr 7, 2016

@SethTisue said:
yeah, works in M4, too. so, closing. (though it certainly remains possible there's something there that someone could isolate and submit as a fix to the 2.11.x series)

@scabug scabug closed this as completed Apr 7, 2016
@scabug
Copy link
Author

scabug commented Jun 3, 2016

@som-snytt said (edited on Jun 3, 2016 5:50:33 PM UTC):
In fact it was scala/scala#4086 which I had totally forgotten, and then tweaked by scala/scala@c5f3d3f

Also, backport scala/scala#5213

@scabug scabug added this to the 2.11.9 milestone Apr 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@som-snytt som-snytt

Labels
repl
Projects
None yet
Milestone
2.11.11
Development

No branches or pull requests
2 participants
@som-snytt @scabug