Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closures hang on to free variables of all their parents, causing subtle memory leaks #5367

Closed
scabug opened this issue Jan 11, 2012 · 4 comments

Comments

@scabug
Copy link

scabug commented Jan 11, 2012

The following code will throw an OutOfMemoryException (with heap size below 1GB):

class Engine {
  
  var handlers: List[(Int => Unit)] = Nil
  
  def registerHandler(f: Int => Unit) = handlers ::= f
  
  def setup() {
    
    val buffer = new Array[Byte](1000000)
    
    for (s <- List("foo")) {
      buffer(0) = 1

      registerHandler { n =>
        callback(n)
      }
    }
  }
  
  def callback(n: Int) = println(n)
}
  
object Test extends Engine {  
  def main(args: Array[String]) {
    
    println("start")
    
    for (i <- 0 until 1000) {
      println("setup handler " + i)
      setup()
    }
    
  } 
}

The OOM occurs because the buffer objects cannot be garbage collected. This is unexpected because only closures passed to registerHandler are retained, which have no free variables except (implicitly) the enclosing Engine instance. Certainly, they don't refer to any buffer.

However, closures are implemented as anonymous inner classes, and inner classes refer to their enclosing classes by chasing $outer pointers. Thus, the closures passed to registerHandler do not refer directly to the Engine instance but indirectly through the statically enclosing closure (the one passed to List("foo").map, which does refer to the buffer).

This behavior is particularly troublesome when using continuations, as the nesting of closures is no longer apparent from the program text:

val buffer = new Array[Byte](1000000)
sleep(100) // cps version
buffer(0) = 1
registerHandler { n => ... } // closure will hang on to buffer

A possible fix could be to treat this-references as proper free variables. More generally, anonymous classes could use separate outer_k pointers for the k-th enclosing object (called a 'display' in the literature). Of course only the used ones should be kept, so #1419 may be related.

@scabug
Copy link
Author

scabug commented Jan 11, 2012

Imported From: https://issues.scala-lang.org/browse/SI-5367?orig=1
Reporter: @TiarkRompf
See #2617, #1419

@scabug
Copy link
Author

scabug commented Jan 17, 2012

@sbocq said:
The leak may affect for-comprehensions as well and cause serialization overhead. The following snippet is extracted from a program where this is causing an issue.

  case class Id[A](a:A) {
    def map[B](f:A => B):Id[B] = Id(f(a))
    def flatMap[B](f: A => Id[B]):Id[B] = f(a) 
  }
    
  def foo(x:Id[Int]):Id[Unit] = for {
    u <- Id(4)
    // v <- Id(5)
    w <- x.map(_ + u)
  } yield w

  foo(Id(1))

The closure passed to map captures only 'u' if "v <- Id(5)" is commented out.

-> f	      Test$$anonfun$foo$1$$anonfun$apply$1  (id=21)	
 -> u$1	      4	

If the line is uncommented, although it is completely unrelated, the closure captures x.

-> f          Test$$anonfun$foo$1$$anonfun$apply$3$$anonfun$apply$1  (id=22)	
 -> $outer    Test$$anonfun$foo$1$$anonfun$apply$3  (id=29)	
  -> $outer   Test$$anonfun$foo$1  (id=34)	
   -> x$2     Test$Id<A>  (id=21)	
   -> u$1     4	

@scabug
Copy link
Author

scabug commented Jan 17, 2012

Turing Eret (turingeret) said:
The behavior of unnecessarily hanging on to variables by closures is related to inner classes unnecessarily hanging on to variables, due to how closures are implemented as inner classes.

@scabug
Copy link
Author

scabug commented Feb 26, 2015

@adriaanm said:
Sorry, the continuations plugin has been unsupported for a while now. Please open an issue over at https://github.com/scala/scala-continuations (we're still looking for a maintainer for it, though).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants