Imported From: https://issues.scala-lang.org/browse/SI-10045?orig=1
Reporter: Shixiong Zhu (zsxwing)
Affected Versions: 2.10.6, 2.11.8, 2.12.0

Shixiong Zhu (zsxwing) said (edited on Nov 14, 2016 6:10:26 PM UTC):
Sorry that I don't have a simple reproducer. Right now I can only reproduce this issue using very complicated Spark SQL codes in Spark shell. The REPL will throw some assertions when compiling codes. The issue will be gone if I use some hacky codes to make scala.tools.nsc.interpreter.IMain.TranslatingClassLoader thread-safe.

I think I found an easy way to reproduce this bug with high reliability.
On Windows 10 and Windows Server 2016 I did some experiments to run Apache Spark from a command line using the scala REPL that is included with that. If you simply do a standard install, open up the spark-shell and then :quit (without doing anything else) then the cleanup process of Spark will try to delete any jar files or classes that were loaded to serve the REPL.

Since Windows (unlike Linux) locks the files there is an exception thrown within Spark that shows the offending class as this one in scala REPL. I am interpreting this behaviour as meaning the scala REPL classloader was not closed (by Spark's shutdown manager) maybe because that action can only happen within the scala REPL itself (i.e. the global.close() action is maybe not catching everything).

scala.tools.nsc.interpreter.IMain.TranslatingClassLoader

See the issue over on the Spark thread where a number of Windows developers discuss temp files that are not unloading correctly. I think this may be because they were files loaded by the REPL, and by this particular class. That is guesswork but it may help in isolating what the bug is.

The Spark thread is here:
https://issues.apache.org/jira/browse/SPARK-12216
Windows is just more fussy than Linux on the final step in the spark command shell shutdown process. Since the scala.tools.nsc.interpreter.IMain.TranslatingClassLoader is flagged here as being non threadsafe and it is this classloader that seems to load any class created in the spark-shell REPL I am guessing that this may be the cause of the whole problem, and therefore a good test case.