Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

:cp does not work #6502

Closed
scabug opened this issue Oct 10, 2012 · 32 comments
Closed

:cp does not work #6502

scabug opened this issue Oct 10, 2012 · 32 comments
Assignees
Labels
Milestone

Comments

@scabug
Copy link

scabug commented Oct 10, 2012

import reports "not found: value x" for values on the additional classpath. For instance, the following no more works in 2.10.0-Mx:

scala> :cp C:\Users\u637116.ivy2\cache\junit\junit\jars\junit-4.10.jar
scala> import junit.runner.Version
:7: error: not found: value junit

@scabug
Copy link
Author

scabug commented Oct 10, 2012

Imported From: https://issues.scala-lang.org/browse/SI-6502?orig=1
Reporter: Sonnenschein (sonnenschein)
Affected Versions: 2.10.0-M7, 2.10.3

@scabug
Copy link
Author

scabug commented Sep 4, 2013

Markus Kahl (machisuji) said (edited on Sep 4, 2013 7:55:31 AM UTC):
I have checked the difference between 2.9.1 (where it works) and 2.10.+ (where it doesn't). The reason is that the compilerClasspath is not updated.

That happened in 2.9.1 because the interpreter is closed and recreated (passing the settings with the updated classpath from ILoop) during #replay() when :cp is executed.
In 2.10.+, however, in ILoop#replay() the interpreter is merely reset which means it reuses the same compilerClasspath.

So either IMain#reset must provide a way of passing new settings or ILoop should recreate the interpreter in #addClasspath.

@scabug
Copy link
Author

scabug commented Sep 6, 2013

Markus Kahl (machisuji) said:
I know someone else was already assigned but I still fixed it myself.
Wrote a test and all. Now I just have to see if it breaks any of the other tests which will probably take a day and a half ...

@scabug
Copy link
Author

scabug commented Sep 6, 2013

Markus Kahl (machisuji) said:
Here's the commit by the way: machisuji/scala@9a92b97

Unless it's been done before I will look into the contribution guidelines, check if it merges cleanly into master and create a pull request later on after/if the other tests are ok.

@scabug
Copy link
Author

scabug commented Sep 6, 2013

@som-snytt said (edited on Sep 6, 2013 4:53:07 PM UTC):
Please go ahead and help yourself to a bug fix. I'll help review.

(My old fix was nontrivial, so it was on the back burner.)

@scabug
Copy link
Author

scabug commented Sep 6, 2013

Markus Kahl (machisuji) said:
Ah alright. So how exactly does this work? I assign it to myself and then how does the review work? I created a topic on scala-internals.

@scabug
Copy link
Author

scabug commented Sep 6, 2013

@som-snytt said (edited on Sep 6, 2013 6:34:37 PM UTC):
I think this is the use case (shown in your version):

scala> :cp /tmp
Added '/tmp'.  Your new classpath is:
".:/tmp"
Nothing to replay.

scala> getClass.getResource("/sample/Test.class")
res0: java.net.URL = file:/tmp/sample/Test.class

scala> sample.Test main null
<console>:8: error: not found: value sample
              sample.Test main null
              ^

That is, we want to put something on the compiler's class path, not just the runtime class path of the interpreter.

The trick is to do that without blowing away the interpreter.

Replaying might be OK; sometimes I find I have res6 references that depend on how things are replayed.

My fix was to use a custom ClassPath for ReplGlobal that lets you modify it in-flight. The replay option might turn out to be more robust.

@scabug
Copy link
Author

scabug commented Oct 25, 2013

Prashant Sharma (scrapcodes) said:
Hey, Is this still being looked at ? Would be great if this bug can be fixed ! or at least A.P. Marki, can you point me to your solution to this problem ?

@scabug
Copy link
Author

scabug commented Oct 25, 2013

Markus Kahl (machisuji) said:
Hey, I didn't have a lot of time in the last two months but it's getting better and I'm still working on it.
Although I'm currently stuck with my plan for a solution. So perhaps APM's path is a better idea.
I'm still working on figuring out my problems in the meantime.

It doesn't matter who will do it how as long as it gets done I suppose.

@scabug
Copy link
Author

scabug commented Feb 23, 2014

Jeff Hammerbacher (hammer) said:
Any progress on this issue? Adding jars to the classpath in the REPL is particularly useful for Scala notebooks, so breaking this functionality with 2.10 is quite unfortunate.

@scabug
Copy link
Author

scabug commented Feb 24, 2014

Prashant Sharma (scrapcodes) said:
One not so ideal way of fixing this is in my branch.

https://github.com/ScrapCodes/scala/tree/si-6502-fix

@som-snytt thoughts ?

@scabug
Copy link
Author

scabug commented Aug 14, 2014

Chip Senkbeil (senkwich) said (edited on Aug 14, 2014 7:07:06 PM UTC):
Do any of you know if dynamically updating the compiler's classpath and invalidating the new entries to reload them has any unexpected consequences?

I did that through reflection with regard to Spark:
apache/spark#1929

I can't imagine it would aside from the fact that older compiled code wouldn't have the new dependencies brought in, which might cause some weirdness.

Isn't this similar to what you did with the modifiable classpath, A. P. Marki?

@scabug
Copy link
Author

scabug commented Aug 14, 2014

@paulp said:
Given that invalidateClassPathEntries is never called from sbt, never called within the compiler itself, and has not a single test, you should expect nothing but unexpected consequences. (Russell's paradox, scalac edition.)

@scabug
Copy link
Author

scabug commented Aug 14, 2014

@som-snytt said:
I commented on the spark PR, but to note here:

scala/scala#3884

The fact that there's not an associated ticket means it was never a feature.

@scabug
Copy link
Author

scabug commented Aug 14, 2014

Chip Senkbeil (senkwich) said:
So, is there a recommended way to append to the compiler's classpath and get the entries reloaded (in case the jar has been updated)? For commands like :cp, throwing out the interpreter and replaying commands is not an ideal solution, especially not in Spark's case.

@scabug
Copy link
Author

scabug commented Aug 14, 2014

@paulp said:
It's.probably uncouth for me to do much in the way of advertising my own stuff here in the scala bug tracker, but I'll be shipping alternarepl soon. "Follow me on twitter" @extempore2

@scabug
Copy link
Author

scabug commented Aug 14, 2014

@som-snytt said:
"repaul".

@scabug
Copy link
Author

scabug commented Aug 19, 2014

@gkossakowski said:
Chip, there's no recommended way to append to compiler's classpath at the moment. The functionality removed by scala/scala#3884 was added as an experimental feature for sbt-only consumption. I'll add a bit more info about the original rationale behind the code and why we decided to remove it.

However, for :cp command to work, we need less ambitious API compared to what sbt needed. If you are interested in just appending to classpath you need to deal with two concerns:

what should happen when appended jar contains package that overlap with what's already on classpath?

what should happen when appended jar contains a class that shadows existing class?

We have to handle properly the first concern for appending to classpath to be any useful. Fortunately, it's fairly easy to specify and implement what should happen: we should merge packages. There might be some subtle issues with implicit search changing it's outcomes for certain queries due to different package contents. However, it's extremely unlikely any user would run into those problems in practice.

When it comes to shadowing (or reloading) existing class, that's order of magnitude more difficult. I don't think it's realistic :cp command could support that. For that reason, we should just detect duplicated classes when entries are being appended and throw an error.

If we narrow down the scope of classpath manipulation logic to what I outlined above we have a good chance to implement it as part of official compiler's API, with tests, documentation and long-term support.

I don't have plans to work on this myself for now. I'm happy to help in case someone is interested in tackling this issue.

@scabug
Copy link
Author

scabug commented Sep 11, 2014

Francois Armand (fanf) said:
Seriously, almost two years for that, and nothing, not even a correct error message ? What is repl good at if I cannot even import jars? That's really embarrassing.

At least, please, please: do put a message when the user try to use cp, or display something like ":cp : no more supported because we think you don't need external dependencies, see SI-6502" in the help. That's not pro at all.

@scabug
Copy link
Author

scabug commented Sep 12, 2014

@som-snytt said:
Speaking as someone who is easily embarrassed, I'd say this is the least embarrassing moment among many potentially mortifying ones that I've experienced with the Scala community. However, I'm still blushing at extempore's brazen self-promotion earlier in this thread.

An alternative approach is to distinguish :cp that discards the compiler from :require that does not. Then :cp, which could be renamed so it doesn't say copy, but not to :path because :pa is the prefix of :paste, would be shorthand for, in sbt, quit, edit build.sbt, reload, console. Maybe :reset could take args, :reset -cp my.jar. If you want replay as well, then :replay -cp my.jar.

That solution also obviates the need to improve upon the error message proposed in the previous post.

@scabug
Copy link
Author

scabug commented Sep 12, 2014

@paulp said:
You know what they say, "The world belongs to the shameless."

@scabug
Copy link
Author

scabug commented Sep 12, 2014

@gkossakowski said:
Francois, yes, it would be great to fix it. However, we don't have infinite resources so we would love to take some contributions. Adding a message is a good start. PR is welcome.

@scabug
Copy link
Author

scabug commented Sep 12, 2014

Sonnenschein (sonnenschein) said:
I believe we'd be fine with Grzegorz's previous suggestion to append additional classes only if they are not yet present (no shadowing).
For the time being an error message could be output telling that -cp on command line is a valid work around.

@scabug
Copy link
Author

scabug commented Sep 16, 2014

@som-snytt said:
Here's a simple implementation of :reset -classpath my.jar. Supplying settings args to reset and replay does a destructive creation, as in the old days. There's a todo comment for require, which would use fancy tricks for that dynamic feeling.

scala/scala#3986

This is convenient because some settings get baked in, so :settings doesn't take. And :replay -Xlint is handy to show behavior under different settings.

Also, my impression is that the world belongs to the shapeless.

@scabug
Copy link
Author

scabug commented Oct 7, 2014

@som-snytt said:
I just encountered #2403 testing out interactions of log4j2 and serialization.

No one has asked to put a jar on the child class path (that loads REPL artifacts; currently one location), but that could be a thing.

@scabug
Copy link
Author

scabug commented Oct 13, 2014

@heathermiller said:
WIP, to PR within the next 24 hours:
https://github.com/heathermiller/scala/compare/repl-cp-fix2?expand=1

still to do: clean ups, doc comments, tests

@scabug
Copy link
Author

scabug commented Oct 14, 2014

@heathermiller said:
PR'ed: scala/scala#4051

@scabug
Copy link
Author

scabug commented Nov 18, 2014

@retronym said:
:cp has been replaced by :require, which is more limited, but less broken.

scala/scala#4051

@scabug scabug closed this as completed Nov 18, 2014
@scabug
Copy link
Author

scabug commented Jan 15, 2015

David Perez said:
I've downloaded Scala 2.11.5 and tried to load a new jar to the classpath:

Welcome to Scala version 2.11.5 (OpenJDK 64-Bit Server VM, Java 1.7.0_65).
Type in expressions to have them evaluated.
Type :help for more information.

scala> :require /home/d/.m2/repository/junit/junit/4.11/junit-4.11.jar
java.lang.NoClassDefFoundError: junit/framework/TestCase
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
        at scala.tools.nsc.interpreter.ILoop$InfoClassLoader$1.classOf(ILoop.scala:638)
        at scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$classNameOf$1(ILoop.scala:657)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$10.apply(ILoop.scala:659)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$10.apply(ILoop.scala:659)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
        at scala.collection.Iterator$class.exists(Iterator.scala:776)
        at scala.collection.AbstractIterator.exists(Iterator.scala:1202)
        at scala.tools.nsc.interpreter.ILoop.require(ILoop.scala:659)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$standardCommands$13.apply(ILoop.scala:221)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$standardCommands$13.apply(ILoop.scala:221)
        at scala.tools.nsc.interpreter.LoopCommands$LineCmd.apply(LoopCommands.scala:62)
        at scala.tools.nsc.interpreter.ILoop.colonCommand(ILoop.scala:712)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:703)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:430)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:918)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:904)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:904)
        at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
        at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:904)
        at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
        at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:87)
        at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98)
        at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
        at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
        at java.lang.ClassLoader.findClass(ClassLoader.java:531)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 28 more


Unrecoverable error.

Failed to initialize compiler: NoClassDefFoundError.
This is most often remedied by a full clean and recompile.
Otherwise, your classpath may continue bytecode compiled by
different and incompatible versions of scala.

java.lang.NoClassDefFoundError: junit/framework/TestCase
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:643)
        at scala.tools.nsc.interpreter.ILoop$InfoClassLoader$1.classOf(ILoop.scala:638)
        at scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$classNameOf$1(ILoop.scala:657)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$10.apply(ILoop.scala:659)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$10.apply(ILoop.scala:659)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:370)
        at scala.collection.Iterator$class.exists(Iterator.scala:776)
        at scala.collection.AbstractIterator.exists(Iterator.scala:1202)
        at scala.tools.nsc.interpreter.ILoop.require(ILoop.scala:659)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$standardCommands$13.apply(ILoop.scala:221)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$standardCommands$13.apply(ILoop.scala:221)
        at scala.tools.nsc.interpreter.LoopCommands$LineCmd.apply(LoopCommands.scala:62)
        at scala.tools.nsc.interpreter.ILoop.colonCommand(ILoop.scala:712)
        at scala.tools.nsc.interpreter.ILoop.command(ILoop.scala:703)
        at scala.tools.nsc.interpreter.ILoop.processLine(ILoop.scala:404)
        at scala.tools.nsc.interpreter.ILoop.loop(ILoop.scala:430)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply$mcZ$sp(ILoop.scala:918)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:904)
        at scala.tools.nsc.interpreter.ILoop$$anonfun$process$1.apply(ILoop.scala:904)
        at scala.reflect.internal.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:97)
        at scala.tools.nsc.interpreter.ILoop.process(ILoop.scala:904)
        at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:74)
        at scala.tools.nsc.MainGenericRunner.run$1(MainGenericRunner.scala:87)
        at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:98)
        at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:103)
        at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)
Caused by: java.lang.ClassNotFoundException: junit.framework.TestCase
        at java.lang.ClassLoader.findClass(ClassLoader.java:531)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        ... 28 more

Is this a bug?
Am I missing something?

@scabug
Copy link
Author

scabug commented Jan 15, 2015

@retronym said:
Looks like a bug, I'll dig into it. I notice it also doesn't validate inputs, and fails with a NullPointerException if the file is absent.

scala> :require /xxxx
java.lang.NullPointerException
	at scala.tools.nsc.interpreter.ILoop.scala$tools$nsc$interpreter$ILoop$$flatten$1(ILoop.scala:651)
	at scala.tools.nsc.interpreter.ILoop.require(ILoop.scala:654)

@scabug
Copy link
Author

scabug commented Jan 15, 2015

@retronym said:
I've got a fix in the works: https://github.com/retronym/scala/compare/ticket/6502?expand=1

/cc @heathermiller

@scabug
Copy link
Author

scabug commented Jan 16, 2015

@retronym said (edited on Jan 16, 2015 1:44:43 AM UTC):
scala/scala#4251

@scabug scabug added the repl label Apr 7, 2017
@scabug scabug added this to the 2.11.5 milestone Apr 7, 2017
@scala scala deleted a comment from scabug Dec 9, 2019
@scala scala deleted a comment from scabug Dec 9, 2019
@scala scala deleted a comment from scabug Dec 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants