New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Large mapped streams cause OutOfMemoryError when using reduceLeft. #2239
Comments
Imported From: https://issues.scala-lang.org/browse/SI-2239?orig=1 |
@phaller said: object Test extends Application {
def nextNumber(number: Long): Long =
if (number % 2 == 0)
number / 2
else
(number * 3) + 1
def numberStream(number: Long): Stream[Long] =
if (number == 1)
Stream.cons(1, Stream.empty)
else
Stream.cons(number, numberStream(nextNumber(number)))
def numberRangeStream(high: Long, number: Long): Stream[Long] =
if (number == high)
Stream.cons(high, Stream.empty)
else
Stream.cons(number, numberRangeStream(high, number + 2))
val low = 500001
val high = 999999L
var mapped = numberRangeStream(high, low).map(numberStream(_))
val s = mapped.size
println("size of mapped stream: "+s)
var i = 0
var curr = mapped
//mapped = null
var el = curr.head
while (i < s) {
i += 1
el = curr.head
print(el.size+" ")
curr = curr.tail
}
} Therefore, no matter how Therefore, I am closing this ticket, since it has apparently nothing to do with |
@SethTisue said: Stream.const(1).reduceLeft(_ max _) it seems to me that it should run in constant space, but it runs out of memory on both 2.7 and trunk (r18571). if #692 is fixable, isn't this fixable too? |
Hans-Peter Strr (hans-peter.stoerr) said: (1 to 10000000).toStream.foldLeft(0)(_+_)
(1 to 10000000).toStream.reduceLeft(_+_) Both are semantically equivalent. The first one runs correctly, and the second one throws an OutOfMemoryError. The reason for this is that reduceLeft calls foldLeft for the actual work. foldLeft it itself allows the garbage collection of the already processed part of the stream because it discards the reference to the original head of the stream by using tailrecursion. On the other hand, the call to reduceLeft is forwarded to foldLeft, and thus the stackframe of reduceLeft holds a reference to the original head of the stream during the full time of the call, such that nothing can be garbage collected. I suppose the only way to fix this is some code duplication: if reduceLeft does it's own work instead of forwarding the call, this is solved. I would guess this applies to other methods of Stream as well, rendering it a little bit dangerous to use - compare [http://stackoverflow.com/questions/4132924/functional-processing-of-scala-streams-without-outofmemory-errors/4134541SI-4134541 this comment]. 8-} |
On my machine this runs for about 40 seconds and then the error occurs. I've tried this on both the nightly 2.8 build, 2.7.3final (Ubuntu) and whatever Netbeans is bringing along with it. There doesn't seem to be any reason for this that has become apparent to me from looking at the Scala code underlying the related methods.
The text was updated successfully, but these errors were encountered: