Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AnyRef specialization crashes when compiling against source Function1 #5545

Closed
scabug opened this issue Mar 5, 2012 · 17 comments
Closed

AnyRef specialization crashes when compiling against source Function1 #5545

scabug opened this issue Mar 5, 2012 · 17 comments
Assignees
Milestone

Comments

@scabug
Copy link

scabug commented Mar 5, 2012

This looks like something tied up in AnyRef specialization and generic signatures. It's not clear to me what it is yet though. I'd appreciate any assistance.
[Removed red herring, compiling Function1.scala is enough.]

% scalac -d /tmp src/library/scala/Function1.scala 
error: T1$sp in trait Function1$mcDL$sp cannot be instantiated from Function1$mcDL$sp
error: 
     while compiling:  src/library/scala/Function1.scala
       current phase:  jvm
     library version:  version 2.10.0-M2-0091-g43532f1192-2012-03-04
    compiler version:  version 2.10.0-M2-0091-g43532f1192-2012-03-04
  reconstructed args:  -d /tmp

T1$sp in trait Function1$mcDL$sp cannot be instantiated from Function1$mcDL$sp
	at scala.reflect.internal.SymbolTable.abort(SymbolTable.scala:38)
	at scala.tools.nsc.Global.abort(Global.scala:189)
	at scala.reflect.internal.Types$AsSeenFromMap.throwError$1(Types.scala:4078)
	at scala.reflect.internal.Types$AsSeenFromMap.instParam$1(Types.scala:4098)
	at scala.reflect.internal.Types$AsSeenFromMap.toInstance$1(Types.scala:4118)
	at scala.reflect.internal.Types$AsSeenFromMap.apply(Types.scala:4136)
	at scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$2.apply(Types.scala:3771)
	at scala.reflect.internal.Types$TypeMap$$anonfun$noChangeToSymbols$2.apply(Types.scala:3771)
	at scala.collection.LinearSeqOptimized$class.forall(LinearSeqOptimized.scala:69)
	at scala.collection.immutable.List.forall(List.scala:77)
	at scala.reflect.internal.Types$TypeMap.noChangeToSymbols(Types.scala:3771)
	at scala.reflect.internal.Types$TypeMap.mapOver(Types.scala:3784){noformat}
These look right:

scala> classOf[scala.Function1[,]].getTypeParameters.toList
res3: List[java.lang.reflect.TypeVariable[Class[Function1[_, _]]]] = List(T1, R)

scala> classOf[scala.Function1$mcDL$sp[]].getTypeParameters.toList
res4: List[java.lang.reflect.TypeVariable[Class[Function1$mcDL$sp[
]]]] = List(T1$sp)
{noformat}

@scabug
Copy link
Author

scabug commented Mar 5, 2012

Imported From: https://issues.scala-lang.org/browse/SI-5545?orig=1
Reporter: @paulp

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
I haven't pinpointed it further yet, but I have confirmed that this regressed sometime since 4c48abbe5a (3 weeks ago.)

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
OK, it's bb23d766bc . I guess I am the winner.

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
Notes:

  • if nothing is specialized on AnyRef, no crash.
  • if the class is not scala.Function1, but bippy.Function1, no crash. So it seems the way Function1 is read from bytecode comes into conflict with the way it is read from source.
  • I remember specialization bugs going all the way back to the appearance of @Specialize which have this quality, where you can't compile a specialized version of an existing scala class without crashing, but the same source code compiles fine in package bippy. So that part is not new.

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
Minimized.

trait F[@specialized(AnyRef) R] {
  def f(): R
  def g() = () => f()
}
% rm -rf out && mkdir out && scalac -cp out -d out ./a.scala && scalac -cp out -d out ./a.scala
error: R$sp in trait F$mcL$sp cannot be instantiated from F$mcL$sp

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
Diffing debug output from first compile and second, the first line after the crash point in the non-crashing compile is

[log jvm] sig(apply, method apply, anonymous class F$mcL$sp$$anonfun$g$mcL$sp$1)      ()TR$sp;

So the surviving type parameter after anyref specialization (TR$sp) is referenced from a closure.

The other difference of note is that these two lines appear only in the crashing version:

< [log flatten] lifted anonymous class $anonfun$g$1 in trait F, unlinked anonymous class F$$anonfun$g$1
< [log flatten] lifted anonymous class $anonfun$g$mcL$sp$1 in trait F$mcL$sp, unlinked anonymous class F$mcL$sp$$anonfun$g$mcL$sp$1

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@VladUreche said:
I can't reproduce the small testcase on trunk, but I managed to reproduce the bug with Function1:

$ git log --oneline -n 5
3525296 Oops, all the preceding meant SI-5541.
777dbd7 Revert attempt to limit private types in lubs.
f708b87 Overcame trait/protected/java limitation.
fb87f2d Test case closes SI-4777.
61d34ed An old patch on pattern matcher exhaustivity.
$ ant quick.clean; ant
[...]
$ cat a.scala
trait F[@specialized(AnyRef) R] {
  def f(): R
  def g() = () => f()
}
$ rm -rf out && mkdir out && scalac -cp out -d out ./a.scala && scalac -cp out -d out ./a.scala
$ # it ran fine

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
Oh man. I was working with bb23d766bc , as the first commit where I saw the crashing behavior. You're right, in current master the reduced example no longer crashes. Here's one which still crashes.

trait F[@specialized(Int, AnyRef) T1, @specialized(AnyRef) R] {
  def apply(v1: T1): R
  def compose[A](g: A => T1): A => R = { x => apply(g(x)) }
}

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@VladUreche said:
A bigger testcase, but it reliably crashes scalac:

$ cat a.scala
trait F[@specialized(Int) T1, R] {
  def f(v1: T1): R
  def g = v1 => f(v1)
}
$ rm -rf out; mkdir out; scalac -d out a.scala; scalac -d out -cp out a.scala
error: R in trait F$mcI$sp cannot be instantiated from F$mcI$sp
error: 
     while compiling:  a.scala
       current phase:  jvm
     library version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
    compiler version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
  reconstructed args:  -d out -classpath out

uncaught exception during compilation: scala.reflect.internal.FatalError
error: scala.reflect.internal.FatalError: 
     while compiling:  a.scala
       current phase:  erasure
     library version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
    compiler version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
  reconstructed args:  -d out -classpath out

R in trait F$mcI$sp cannot be instantiated from F$mcI$sp
	at scala.reflect.internal.SymbolTable.abort(SymbolTable.scala:38)
	at scala.tools.nsc.Global.abort(Global.scala:189)
	at scala.reflect.internal.Types$AsSeenFromMap.throwError$1(Types.scala:4085)
        [...]
	at scala.tools.nsc.Driver.main(Driver.scala:65)
	at scala.tools.nsc.Main.main(Main.scala)

error: fatal error: 
     while compiling:  a.scala
       current phase:  erasure
     library version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
    compiler version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
  reconstructed args:  -d out -classpath out

R in trait F$mcI$sp cannot be instantiated from F$mcI$sp

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
Oh, not even AnyRef specialization. That's valuable.

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
Here again is the logging diff for vlad's test case.

< [log namer] [overwrite] class F
> [log namer] [+symbol] trait F abstract <defaultparam/trait>
< [log flatten(->mixin)] lifted anonymous class $anonfun$g$1 in trait F, unlinked anonymous class F$$anonfun$g$1
< [log flatten(->mixin)] lifted anonymous class $anonfun$g$mcI$sp$1 in trait F$mcI$sp, unlinked anonymous class F$mcI$sp$$anonfun$g$mcI$sp$1
< [log jvm(->lazyvals)] [class] >> F$mcI$sp
> [log jvm] sig(apply, method apply, anonymous class F$mcI$sp$$anonfun$g$mcI$sp$1)      (I)TR;

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@VladUreche said (edited on Mar 5, 2012 3:29:43 PM UTC):
Now it makes sense. The first compilation 'forgets' the R type parameter for F$mcI$sp:

$ cd out
$ scalap -cp . 'F$mcI$sp'
package F$mcI$sp;
abstract trait F$mcI$sp with F {
  def g$mcI$sp(): scala.Function1;
  def g(): scala.Function1;
  def f(scala.Int): scala.Any;
}

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@paulp said:
The irony is that it forgets to remind itself, but succeeds in reminding java: the generic signatures look right.

scala> classOf[F$mcI$sp].getDeclaredMethods map (_.toGenericString) foreach println
public abstract R F$mcI$sp.f(int)
public abstract scala.Function1<java.lang.Object, R> F$mcI$sp.g()
public abstract scala.Function1<java.lang.Object, R> F$mcI$sp.g$mcI$sp()

Except, why are there two signatures attached to f and g, is that normal?

const #3 = Asciz	f;
const #4 = Asciz	(I)Ljava/lang/Object;;
const #5 = Asciz	(I)TR;;
const #7 = Asciz	g;
const #8 = Asciz	()Lscala/Function1;;
const #9 = Asciz	()Lscala/Function1<Ljava/lang/Object;TR;>;;

@scabug
Copy link
Author

scabug commented Mar 5, 2012

@non said:
Sorry, accidentally clicked the wrong link in JIRA.

@scabug
Copy link
Author

scabug commented Mar 6, 2012

@VladUreche said:
Okay, narrowed it down a bit more, I'm posting in case anyone can make more sense of this.

Before you dive into the details, I need someone to explain how the signatures work. If I understood well, only the main class F should have a Scala signature pickled into it, and when loading the class and the signature, it will undergo the specialization type transform and infer there must be other specialized classes lying around (from the @specialized on the type params) - thus F$mcI$sp should not pickle a Scala signature, right?
-- if this is so, the problem is that the F$mcI$sp class has a Scala signature pickled into it and it shouldn't
-- if it is expected for F$mcI$sp to contain a pickled Scala signature, then the signature is incorrectly generated (missing the type param)

Now, comparing the two logs (compile with and without existing bytecode) one can spot a bytecode loading taking place just before generating the class in GenJVM -- and this seems to change (break) the existing signature generated by specialization:

[log jvm] sig(g$mcI$sp, method g$mcI$sp#10450, trait F$mcI$sp#10444)      ()Lscala/Function1<Ljava/lang/Object;TR;>;
<omitted output>
[log jvm] sig(F$mcI$sp, trait F$mcI$sp#10444, package class <empty>#4)      <R:Ljava/lang/Object;>Ljava/lang/Object;LF<Ljava/lang/Object;TR;>;
[log jvm(->lazyvals)] [class] >> F$mcI$sp

This is trait F$mcI$sp#10444: I got my info changed!
new info:  lang#2520.this.Object#2592
        with <empty>#4.this.F#8[T1#7693,R#7694] {
  
}
phase:     parser
stack:
    at java.lang.Thread.getStackTrace(Thread.java:1479)
    at scala.reflect.internal.Symbols$Symbol.info_$eq(Symbols.scala:1023)
    at scala.reflect.internal.Symbols$TypeSymbol.info_$eq(Symbols.scala:2434)
    at scala.reflect.internal.Symbols$Symbol.setInfo(Symbols.scala:1031)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:559)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:100)
    at scala.tools.nsc.symtab.SymbolLoaders$ClassfileLoader.doComplete(SymbolLoaders.scala:234)
    at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:136)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:996)
    at scala.reflect.internal.Symbols$Symbol.initialize(Symbols.scala:1136)
    at scala.tools.nsc.symtab.SymbolLoaders$moduleClassLoader$.doComplete(SymbolLoaders.scala:259)
    at scala.tools.nsc.symtab.SymbolLoaders$SymbolLoader.complete(SymbolLoaders.scala:136)
    at scala.reflect.internal.Symbols$Symbol.info(Symbols.scala:996)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator$$anonfun$addInnerClasses$1$$anonfun$apply$mcV$sp$1.apply(GenJVM.scala:803)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator$$anonfun$addInnerClasses$1$$anonfun$apply$mcV$sp$1.apply(GenJVM.scala:803)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
    at scala.collection.immutable.List.foreach(List.scala:77)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator$$anonfun$addInnerClasses$1.apply$mcV$sp(GenJVM.scala:803)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator$$anonfun$addInnerClasses$1.apply(GenJVM.scala:803)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator$$anonfun$addInnerClasses$1.apply(GenJVM.scala:803)
    at scala.reflect.internal.SymbolTable.atPhase(SymbolTable.scala:134)
    at scala.reflect.internal.SymbolTable.afterPhase(SymbolTable.scala:143)
    at scala.tools.nsc.Global.afterErasure(Global.scala:872)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.addInnerClasses(GenJVM.scala:802)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.emitClass(GenJVM.scala:295)
    at scala.tools.nsc.backend.jvm.GenJVM$BytecodeGenerator.genClass(GenJVM.scala:426)
    at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase$$anonfun$run$3.apply(GenJVM.scala:165)
    at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase$$anonfun$run$3.apply(GenJVM.scala:164)
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59)
    at scala.collection.immutable.List.foreach(List.scala:77)
    at scala.tools.nsc.backend.jvm.GenJVM$JvmPhase.run(GenJVM.scala:164)
    at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1288)
    at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1261)
    at scala.tools.nsc.Global$Run.compileSources(Global.scala:1255)
    at scala.tools.nsc.Global$Run.compile(Global.scala:1385)
    at scala.tools.nsc.Driver.doCompile(Driver.scala:31)
    at scala.tools.nsc.Main$.doCompile(Main.scala:81)
    at scala.tools.nsc.Driver.process(Driver.scala:52)
    at scala.tools.nsc.Driver.main(Driver.scala:65)
    at scala.tools.nsc.Main.main(Main.scala)

This is module class F$mcI$sp#7: I got my info changed!
new info:   {
  
}
phase:     parser
stack:
    at java.lang.Thread.getStackTrace(Thread.java:1479)
    at scala.reflect.internal.Symbols$Symbol.info_$eq(Symbols.scala:1023)
    at scala.reflect.internal.Symbols$TypeSymbol.info_$eq(Symbols.scala:2434)
    at scala.reflect.internal.Symbols$Symbol.setInfo(Symbols.scala:1031)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:561)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:100)
    [ ... same as before ...]

This is module F$mcI$sp#6: I got my info changed!
new info:  object <empty>#4.this.F$mcI$sp#7
phase:     parser
stack:
    at java.lang.Thread.getStackTrace(Thread.java:1479)
    at scala.reflect.internal.Symbols$Symbol.info_$eq(Symbols.scala:1023)
    at scala.reflect.internal.Symbols$Symbol.setInfo(Symbols.scala:1031)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parseClass(ClassfileParser.scala:562)
    at scala.tools.nsc.symtab.classfile.ClassfileParser.parse(ClassfileParser.scala:100)
    [ ... same as before ...]

[log jvm] F$mcI$sp contains 1 inner classes.
[log jvm] //class F$mcI$sp$$anonfun$g$mcI$sp$1
[log jvm] Adding field: F$mcI$sp$$anonfun$g$mcI$sp$1.$outer
[log jvm] Generating method F$mcI$sp$$anonfun$g$mcI$sp$1.apply
[log jvm] Index value for value v1#19172{880484842}: 1
[log jvm] Linearizer adding block 1
[log jvm] Making labels for: F$mcI$sp$$anonfun$g$mcI$sp$1.apply
[log jvm] Generating code for block: 1 at pc: 0
[log jvm] LOAD_FIELD with owner: F$mcI$sp$$anonfun$g$mcI$sp$1 flags: final <synthetic> <lifted>
[log jvm] invokeinterface  F$mcI$sp.f:(int)java/lang/Object
/// starting debugging output added by me
parent = anonymous class F$mcI$sp$$anonfun$g$mcI$sp$1#19167
  signature after erasure: <root>#2#<empty>#4#F$mcI$sp$$anonfun$g$mcI$sp$1#19167
  signature before erasure: F$mcI$sp$$anonfun$g$mcI$sp$1#19167.this.type
we want child info: method apply#19170
  signature after erasure: (<param> <triedcooking> v1#19237: <root>#2#scala#30#Int#453)<root>#2#java#26#lang#2520#Object#2592
  now computing it before erasure
/// In the first test, without bytecode, the next line is:
/// instantiating type R#10434 from trait F$mcI$sp#10433 with List(type R#10434) and List(R#10434), pre = F$mcI$sp#10433.this.type, symclazz = trait F$mcI$sp#10433
/// but now it's missing the type parameter R in the signature:
instantiating type R#10445 from trait F$mcI$sp#10444 with List() and List(), pre = F$mcI$sp#10444.this.type, symclazz = trait F$mcI$sp#10444
error: R#10445 in trait F$mcI$sp#10444 cannot be instantiated from <empty>#4.this.F$mcI$sp#10444
/// end of debugging output
error: 
     while compiling:  a.scala
       current phase:  jvm
     library version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
    compiler version:  version 2.10.0-M2-0097-g35252961db-2012-03-06
  reconstructed args:  -Ydebug -d out -classpath out -uniqid -Ylog:all

Current unit body:
<empty>
uncaught exception during compilation: scala.reflect.internal.FatalError
error: scala.reflect.internal.FatalError: 
     while compiling:  a.scala
       current phase:  erasure
     library version:  version 2.10.0-M2-0097-g35252961db-2012-03-05
    compiler version:  version 2.10.0-M2-0097-g35252961db-2012-03-06
  reconstructed args:  -Ydebug -d out -classpath out -uniqid -Ylog:all

Current unit body:
<empty>
R#10445 in trait F$mcI$sp#10444 cannot be instantiated from <empty>#4.this.F$mcI$sp#10444

@scabug
Copy link
Author

scabug commented Mar 6, 2012

@VladUreche said:
Okay, I think I got it. I have a working patch but I'm trying to understand why the problem is limited to Function1 and doesn't affect Function2. I have to thank Lukas for clearing things up.

Short explanation:

  • during definition initialization, Scala creates symbols for F$mcI$sp since it finds it in the classpath - it creates 3 symbols - the class (trait), module and module class with Lazy types
  • if something would force .info on the lazy type, it would read the signatures and eliminate the symbol from scope, as it's a synthetic class generated by scala
  • until GenJVM, nobody forces the .info, so the F$mcI$sp class, module and module class symbols are kept in package
  • when specialization creates the correct F$mcI$sp, it overwrites the class symbol, but the module symbol is kept
  • in GenJVM, the addInnerClass attempts to get the owner class and module, getting the specialization-generated symbol for F$mcI$sp and the lazy type symbol for the module -- and forces .info on it
  • once this happens, the class is read and the signature for F$mcI$sp is set to an incorrect value (see above)
phase     before parser (Definition.<init>          specialization                 genJVM

source    ---------------------------------------- trait F -----------------------------------------------------
code                                       \
              / class sym (lazy type)  ----  ----- synthetic F$mcI$sp ------- force .info - signature messed up --- crash
existing  ---<- module sym (lazy type) -------------------------------------- force .info - eliminated
bytecode      \ mclass sym (lazy type) ---------------------------------------------------- eliminated

A localized solution is to force .info on all members with the same name before creating a specialized version of a class - and this fixes the particular problem - but shouldn't this be done everytime we create a synthetic class?

@scabug
Copy link
Author

scabug commented Mar 7, 2012

@VladUreche said:
Fixed in c5f68c11.

@scabug scabug closed this as completed Mar 7, 2012
@scabug scabug added this to the 2.10.0-M2 milestone Apr 7, 2017
retronym added a commit to retronym/scala that referenced this issue Mar 24, 2021
Since 8ae0fda, the specializion phase eagerly info transforms
all of FunctionN and TupleN. This was done to let us turn off
needless specialization info transforms (which incurs classfile parsing
up the base classes looking for @specialized annotations) of types after
the specialization tree transform is done.

However, in combination with an old fix for scala/bug#5545, we end
up parsing all of the class files of all the variants, just to
unlink them in favour of the info-transformed types.

I note that the test for scala/bug#5545 no longer crashes if the fix
is removed. I have have not investigated the reason.

This commit reworks the scala/bug#5545 to just unlink the stale
symbols directly, rather than calling `.info` to parse them and
do the same after noticing the ScalaRaw attribute.
retronym added a commit to retronym/scala that referenced this issue Mar 24, 2021
Since 8ae0fda, the specializion phase eagerly info transforms
all of FunctionN and TupleN. This was done to let us turn off
needless specialization info transforms (which incurs classfile parsing
up the base classes looking for @specialized annotations) of types after
the specialization tree transform is done.

However, in combination with an old fix for scala/bug#5545, we end
up parsing all of the class files of all the variants, just to
unlink them in favour of the info-transformed types.

I note that the test for scala/bug#5545 no longer crashes if the fix
is removed. I have have not investigated the reason.

This commit reworks the scala/bug#5545 to just unlink the stale
symbols directly, rather than calling `.info` to parse them and
do the same after noticing the ScalaRaw attribute.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants