New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stream flatMap leaks memory if mapper returns chain of empty results #6409
Comments
Imported From: https://issues.scala-lang.org/browse/SI-6409?orig=1 |
Robin Palotai (ron) said: scala> @scala.annotation.tailrec def len[A](s: Stream[A], acc: Int = 0): Int = if (s.isEmpty) acc else len(s.tail, acc + 1)
len: [A](s: Stream[A], acc: Int)Int
scala> len(bigElemts)
res3: Int = 999999
scala> len(bigElemts.flatMap(_ => Stream.empty[Int]))
... OOM So it is flatMap holding to the head. |
@kmizu said (edited on Sep 25, 2012 2:21:45 AM UTC): However, my previous comment was also wrong. The reason of this problem is not that the run method holds to the stream head. In |
Robin Palotai (ron) said: However when the flatMap's mapper yields a long chain of empty Streams, this behavior does not hold, which is contrary to the (naive and wrong) expectations. An API doc entry would save many hours of debugging (especially in a setup where it is less trivial which Stream transform step is the cause of the leak). By the way, yielding a long run of empty streams is not artificial (I know you were not stating this, just expressing my opinion), it happened in a live dataset. |
@jsuereth said: Not a fix, but documentation for now. Workaround may be more extensive. Will look into it for 2.10.1. |
@adriaanm said: |
@Ichoran said: Stream can't know when you call flatMap that you're doing so in a context that allows it to release the head and thus that it shouldn't memoize the results. You've got the same issue with dropWhile or anything else. So, closing this as wontfix. If someone wants to open a feature request for an immutable collection that is lazy, re-traversable, and non-memoizing (a halfway point between Iterator and Stream), feel free. Otherwise, try to use Iterator in cases like these. |
@Ichoran said: |
@Ichoran said: |
The reference to the original stream head is not released until the mapper function produces a non-empty stream. This can be a problem if the elements consume considerable memory, and the mapper returns a long chain of empy streams.
With Stream's head not being lazy, I'm not sure this can be solved. But at least would be nice to include a hint in the flatMap apidocs.
The text was updated successfully, but these errors were encountered: