change from resetAllAttrs to resetLocalAttrs in reifiers broke Slick #8316

scabug · 2014-02-19T16:44:35Z

import scala.slick.direct._

case class Coffee(name: String)

object Test extends App {
  def test() {
    val query: Queryable[Coffee] = ???
    query.map(_.name).filter(_ == "")
  }
}

-------------------------------------------------
I believe the first bad commit is scala/scala@6015418 (removing resetAllAttrs from reify).
-------------------------------------------------
% scalac-hash 6015418~1 -Yreify-debug -uniqid @args.txt 2>&1 > good.txt
% scalac-hash 6015418 -Yreify-debug -uniqid @args.txt 2>&1 > bad.txt
diff -u {good,bad}.txt
--- good.txt    2014-02-19 17:30:36.000000000 +0100
+++ bad.txt 2014-02-19 17:30:54.000000000 +0100
@@ -365,9 +365,9 @@
       def apply[U >: Nothing#2986 <: Universe#8159 with Singleton#5702]($m$untyped: Mirror#8252[U]): U#Tree = {
         val $u: U = $m$untyped.universe;
         val $m: $u.Mirror = $m$untyped.asInstanceOf[$u.Mirror];
-        val free$query1 = $u.build.newFreeTerm("query", query#15744, $u.build.FlagsRepr.apply(17592190238720L), "defined by test in Test.scala:7:9");
+        val free$query1 = $u#19444.build#12680.newFreeTerm#12699("query", query#15744, $u#19444.build#12680.FlagsRepr#12773.apply#12775(17592190238720L), "defined by test in Test.scala:7:9");
         $u.build.setTypeSignature(free$query1, $u.TypeRef($u.ThisType($m.staticPackage("scala.slick.direct").asModule.moduleClass), $m.staticClass("scala.slick.direct.Queryable"), scala.collection.immutable.List.apply($m.staticClass("Coffee").asType.toTypeConstructor)));
-        $u.Apply($u.Select($u.Apply($u.Select($u.build.Ident($m.staticModule("scala.slick.direct.QueryOps")), $u.TermName("query")), scala.collection.immutable.List.apply($u.Apply.apply($u.Select.apply($u.Apply.apply($u.Select.apply($u.build.Ident($m.staticModule("scala.slick.direct.QueryOps")), $u.TermName.apply("query")), scala.collection.immutable.List.apply($u.build.Ident(free$query1))), $u.TermName.apply("map")), scala.collection.immutable.List.apply($u.Function.apply(scala.collection.immutable.List.apply($u.ValDef.apply($u.Modifiers.apply($u.build.FlagsRepr.apply(2105344L), $u.TypeName.apply(""), immutable#5682.this.Nil), $u.TermName.apply("x$1"), $u.TypeTree.apply(), $u.EmptyTree)), $u.Select.apply($u.Ident.apply($u.TermName.apply("x$1")), $u.TermName.apply("name"))))))), $u.TermName("filter")), scala.collection.immutable.List.apply($u.Function(scala.collection.immutable.List.apply($u.ValDef($u.Modifiers($u.build.FlagsRepr(2105344L), $u.TypeName(""), scala.collection.immutable.List.apply()), $u.TermName("x$2"), $u.TypeTree(), $u.EmptyTree)), $u.Apply($u.Select($u.Ident($u.TermName("x$2")), $u.TermName("$eq$eq")), scala.collection.immutable.List.apply($u.Literal($u.Constant("")))))))
+        $u.Apply($u.Select($u.Apply($u.Select($u.build.Ident($m.staticModule("scala.slick.direct.QueryOps")), $u.TermName("query")), scala.collection.immutable.List.apply($u#19444.Apply#10440.apply#10443($u#19444.Select#10477.apply#10480($u#19444.Apply#10440.apply#10443($u#19444.Select#10477.apply#10480($u#19444.build#12680.Ident#12721($m#19445.staticModule#19420("scala.slick.direct.QueryOps")), $u#19444.TermName#9933.apply#9936("query")), scala#22.collection#2627.immutable#5681.List#14910.apply#19532($u#19444.build#12680.Ident#12721(free$query1))), $u#19444.TermName#9933.apply#9936("map")), scala#22.collection#2627.immutable#5681.List#14910.apply#19532($u#19444.Function#10298.apply#10301(scala#22.collection#2627.immutable#5681.List#14910.apply#19532($u#19444.ValDef#10118.apply#10121($u#19444.Modifiers#10973.apply#10977($u#19444.build#12680.FlagsRepr#12773.apply#12775(2105344L), $u#19444.TypeName#9940.apply#9943(""), immutable#5682.this.Nil#14130), $u#19444.TermName#9933.apply#9936("x$1"), $u#19444.TypeTree#10605.apply#10608(), $u#19444.EmptyTree#9997)), $u#19444.Select#10477.apply#10480($u#19444.Ident#10485.apply#10488($u#19444.TermName#9933.apply#9936("x$1")), $u#19444.TermName#9933.apply#9936("name"))))))), $u.TermName("filter")), scala.collection.immutable.List.apply($u.Function(scala.collection.immutable.List.apply($u.ValDef($u.Modifiers($u.build.FlagsRepr(2105344L), $u.TypeName(""), scala.collection.immutable.List.apply()), $u.TermName("x$2"), $u.TypeTree(), $u.EmptyTree)), $u.Apply($u.Select($u.Ident($u.TermName("x$2")), $u.TermName("$eq$eq")), scala.collection.immutable.List.apply($u.Literal($u.Constant("")))))))
       }
     };
     new $treecreator3()

The text was updated successfully, but these errors were encountered:

scabug · 2014-02-19T16:44:35Z

Imported From: https://issues.scala-lang.org/browse/SI-8316?orig=1
Reporter: @retronym

scabug · 2014-02-19T16:46:28Z

@retronym said:
As discussed: https://github.com/slick/slick/pull/685/files#r9870223

Standalone reproduction pending.

scabug · 2014-02-19T17:43:39Z

@retronym said (edited on Feb 19, 2014 5:43:59 PM UTC):
I believe that without the "luxury" of stripping types and starting again, you'll need to change:

      case TreeSplice(ReifiedTree(universe, mirror, symtab, rtree, tpe, rtpe, concrete)) =>
        if (reifyDebug) println("entering inlineable splice: " + tree)
        val inlinees = symtab.syms filter (_.isLocalToReifee)
        inlinees foreach (inlinee => symtab.symAliases(inlinee) foreach (alias => inlineableBindings(alias) = symtab.symBinding(inlinee)))
        val symtab1 = symtab -- inlinees
        if (reifyDebug) println("trimmed %s inlineable free defs from its symbol table: %s".format(inlinees.length, inlinees map (inlinee => symtab.symName(inlinee)) mkString(", ")))
        withinSplice { super.transform(TreeSplice(ReifiedTree(universe, mirror, symtab1, rtree, tpe, rtpe, concrete))) }

.. to substitute the "$u" from the outer universe for the splicee's universe.

Debug brain dump:

Slick expands:

query#15744.map#15757(((x$1#15957: Coffee#7531) => x$1#15957.name#8072)#15956).filter(((x$2#28373: String#7136) => x$2#28373.==#5721(""))#28372)

This runs through:

 QueryableMacros.filter:90
   QueryableMacros._helper:78
     QueryableMacros._reifyTree():58

Which calls reifyTree in the reflection API with:

       mirror = {scala.reflect.internal.Trees$EmptyTree$@3498}"<empty>"
       tree = {scala.reflect.internal.Trees$Apply@3727}"QueryOps#7847.query#15960[String#7136]({\n  val $u#16000: scala#23.reflect#2606.runtime#2792.package#13974.universe#14081.type = package#13974.this.universe#14081;\n  val $m#16001: $u#16000.Mirror#16006 = package#13974.this.universe#14081.runtimeMirror#1...
       universe = {scala.reflect.internal.Trees$Select@3726}"package#13974.this.universe#14081"

The method involved in the slick Macro:

  def filter[T:c.WeakTypeTag]
    (c: scala.reflect.macros.Context)
    (projection: c.Expr[T => Boolean]): c.Expr[scala.slick.direct.Queryable[T]] = _helper[c.type,T]( c )( "filter", projection )
  def _apply( queryable:Tree, method: String, args:Tree* ) = removeDoubleReify(
    Apply( Select( queryTree(queryable), newTermName( method )), args.toList )
  )
  def _reifyTree[T]( tree:Tree ) = context.Expr[ru.Expr[T]](
      context.reifyTree( context.universe.treeBuild.mkRuntimeUniverseRef, EmptyTree, context.typeCheck(
        tree
      ).asInstanceOf[Tree]))

Note the similarity to test/files/run/macro-reify-nested-a/Impls_Macros_1.scala

The reifiee:

QueryOps#7847.query#15960[String#7136]({
  val $u#16000: scala#23.reflect#2606.runtime#2792.package#13974.universe#14081.type = package#13974.this.universe#14081;
  val $m#16001: $u#16000.Mirror#16006 = package#13974.this.universe#14081.runtimeMirror#16011(this.getClass#2985().getClassLoader#18969());
  $u#16000.Expr#11087.apply#11089[scala#23.slick#2638.direct#7645.BaseQueryable#7690[String#7136]]($m#16001, {
    final class $treecreator1#19428 extends TreeCreator#8237 {
      def <init>#19429(): $treecreator1#19428 = {
        $treecreator1#19428.super.<init>#19424();
        ()
      };
      def apply#19430[U#19431 <: scala#23.reflect#2606.api#2798.Universe#8159 with Singleton#5702]($m$untyped#19434: scala#23.reflect#2606.api#2798.Mirror#8252[U#19432]): U#19431#Tree#9949 = {
        val $u#19444: U#19432 = $m$untyped#19434.universe#19413;
        val $m#19445: $u#19444.Mirror#12975 = $m$untyped#19434.asInstanceOf#5714[$u#19444.Mirror#12975];
        val free$query1#19446: $u#19444.FreeTermSymbol#9516 = $u#19444.build#12680.newFreeTerm#12699("query", query#15744, $u#19444.build#12680.FlagsRepr#12773.apply#12775(17592190238720L), "defined by test in Test.scala:7:9");
        $u#19444.build#12680.setTypeSignature#12708[$u#19444.FreeTermSymbol#9516](free$query1#19446, $u#19444.TypeRef#9703.apply#9706($u#19444.ThisType#9653.apply#9656($m#19445.staticPackage#19422("scala.slick.direct").asModule#9491.moduleClass#9489), $m#19445.staticClass#19418("scala.slick.direct.Queryable"), scala#22.collection#2627.immutable#5681.List#14910.apply#19532[$u#19444.Type#9600]($m#19445.staticClass#19418("Coffee").asType#9434.toTypeConstructor#9429)));
        $u#19444.Apply#10440.apply#10443($u#19444.Select#10477.apply#10480($u#19444.Apply#10440.apply#10443($u#19444.Select#10477.apply#10480($u#19444.build#12680.Ident#12721($m#19445.staticModule#19420("scala.slick.direct.QueryOps")), $u#19444.TermName#9933.apply#9936("query")), scala#22.collection#2627.immutable#5681.List#14910.apply#19532[$u#19444.Ident#10164]($u#19444.build#12680.Ident#12721(free$query1#19446))), $u#19444.TermName#9933.apply#9936("map")), scala#22.collection#2627.immutable#5681.List#14910.apply#19532[$u#19444.Function#10293]($u#19444.Function#10298.apply#10301(scala#22.collection#2627.immutable#5681.List#14910.apply#19532[$u#19444.ValDef#10065]($u#19444.ValDef#10118.apply#10121($u#19444.Modifiers#10973.apply#10977($u#19444.build#12680.FlagsRepr#12773.apply#12775(2105344L), $u#19444.TypeName#9940.apply#9943(""), immutable#5682.this.Nil#14130), $u#19444.TermName#9933.apply#9936("x$1"), $u#19444.TypeTree#10605.apply#10608(), $u#19444.EmptyTree#9997)), $u#19444.Select#10477.apply#10480($u#19444.Ident#10485.apply#10488($u#19444.TermName#9933.apply#9936("x$1")), $u#19444.TermName#9933.apply#9936("name")))))
      }
    };
    new $treecreator1#19428()
  })($u#16000.TypeTag#11206.apply#11238[scala#23.slick#2638.direct#7645.BaseQueryable#7690[String#7136]]($m#16001, {
    final class $typecreator2#28318 extends TypeCreator#8531 {
      def <init>#28319(): $typecreator2#28318 = {
        $typecreator2#28318.super.<init>#28314();
        ()
      };
      def apply#28320[U#28321 <: scala#23.reflect#2606.api#2798.Universe#8159 with Singleton#5702]($m$untyped#28324: scala#23.reflect#2606.api#2798.Mirror#8252[U#28322]): U#28321#Type#9600 = {
        val $u#28333: U#28322 = $m$untyped#28324.universe#19413;
        val $m#28334: $u#28333.Mirror#12975 = $m$untyped#28324.asInstanceOf#5714[$u#28333.Mirror#12975];
        $u#28333.TypeRef#9703.apply#9706($u#28333.ThisType#9653.apply#9656($m#28334.staticPackage#19422("scala.slick.direct").asModule#9491.moduleClass#9489), $m#28334.staticClass#19418("scala.slick.direct.BaseQueryable"), scala#22.collection#2627.immutable#5681.List#14910.apply#19532[$u#28333.Type#9600]($u#28333.TypeRef#9703.apply#9706($u#28333.SingleType#9665.apply#9668($u#28333.ThisType#9653.apply#9656($m#28334.staticPackage#19422("scala").asModule#9491.moduleClass#9489), $m#28334.staticModule#19420("scala.Predef")), $u#28333.build#12680.selectType#12683($m#28334.staticModule#19420("scala.Predef").asModule#9491.moduleClass#9489, "String"), immutable#5682.this.Nil#14130)))
      }
    };
    new $typecreator2#28318()
  }))
}.splice#11078).filter#15969(((x$2#28373: String#7136) => x$2#28373.==#5721(""))#28372)

Which is reified into rtree

rtree = $u.Apply($u.Select($u.Apply($u.Select($u.build.Ident($m.staticModule("scala.slick.direct.QueryOps")), $u.TermName("query")), scala.collection.immutable.List.apply($u#19444.Apply#10440.apply#10443($u#19444.Select#10477.apply#10480($u#19444.Apply#10440.apply#10443($u#19444.Select#10477.apply#10480($u#19444.build#12680.Ident#12721($m#19445.staticModule#19420("scala.slick.direct.QueryOps")), $u#19444.TermName#9933.apply#9936("query")), scala#22.collection#2627.immutable#5681.List#14910.apply#19532($u#19444.build#12680.Ident#12721(free$query1#19446))), $u#19444.TermName#9933.apply#9936("map")), scala#22.collection#2627.immutable#5681.List#14910.apply#19532($u#19444.Function#10298.apply#10301(scala#22.collection#2627.immutable#5681.List#14910.apply#19532($u#19444.ValDef#10118.apply#10121($u#19444.Modifiers#10973.apply#10977($u#19444.build#12680.FlagsRepr#12773.apply#12775(2105344L), $u#19444.TypeName#9940.apply#9943(""), immutable#5682.this.Nil#14130), $u#19444.TermName#9933.apply#9936("x$1"), $u#19444.TypeTree#10605.apply#10608(), $u#19444.EmptyTree#9997)), $u#19444.Select#10477.apply#10480($u#19444.Ident#10485.apply#10488($u#19444.TermName#9933.apply#9936("x$1")), $u#19444.TermName#9933.apply#9936("name"))))))), $u.TermName("filter")), scala.collection.immutable.List.apply($u.Function(scala.collection.immutable.List.apply($u.ValDef($u.Modifiers($u.build.FlagsRepr(2105344L), $u.TypeName(""), scala.collection.immutable.List.apply()), $u.TermName("x$2"), $u.TypeTree(), $u.EmptyTree)), $u.Apply($u.Select($u.Ident($u.TermName("x$2")), $u.TermName("$eq$eq")), scala.collection.immutable.List.apply($u.Literal($u.Constant("")))))))

Before, the resetAllAttrs made the references to $u#19444 bind to the outer `$u.

scabug · 2014-02-19T17:59:53Z

@xeno-by said:
Iirc that substitution is impossible, because one of those symbols isn't created yet. But that's just my faint memory of these events from two years ago. And yes, Slick fails on almost the same code that is part of our test suite. That's why this is totally unexpected to me.

scabug · 2014-02-19T18:05:17Z

@retronym said:
One would need to create the symbol for '$u' eagerly and manually, and go the route of a typing transformer, etc. Sounds familiar :)

scabug · 2014-02-19T20:07:42Z

@xeno-by said (edited on Feb 19, 2014 8:19:12 PM UTC):
To add more drama to the situation, it's not just resetAllAttrs, but also the change to whitebox type inference. Apparently, if we change the boxity of macros defined in those partests from blackbox to whitebox, then all the hell breaks loose, so we were just one word short of discovering this issue early.

Along with being wow'd by another unexpected effect of boxity, I also threw together a simple hack that should take care of the lack of resetAllAttrs: xeno-by/scala@d047a72. Or I should've said "should've taken care of", because it didn't quite work.

Here's the summary of my experiments:
whitebox slick + vanilla master = a lot of tests fail to compile
blackbox slick + vanilla master = runtime exception in QueryableTest
whitebox slick + aforementioned hack = just a couple tests fail to compile
blackbox slick + aforementioned hack = runtime exception in QueryableTest

I guess this is it for me for today, because this is going to need scrupulous analysis as we have at least three different bugs here, and that needs rest for the analyzer :)

scabug · 2014-02-20T15:41:58Z

@xeno-by said:
scala/scala#3569

scabug closed this as completed Feb 20, 2014

scabug added blocker reflection has PR labels Apr 7, 2017

scabug added this to the 2.11.0-RC1 milestone Apr 7, 2017

scabug assigned xeno-by Apr 7, 2017

scabug mentioned this issue Apr 7, 2017

reifier generates incorrect code with existential #8448

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change from resetAllAttrs to resetLocalAttrs in reifiers broke Slick #8316

change from resetAllAttrs to resetLocalAttrs in reifiers broke Slick #8316

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 20, 2014

change from resetAllAttrs to resetLocalAttrs in reifiers broke Slick #8316

change from resetAllAttrs to resetLocalAttrs in reifiers broke Slick #8316

Comments

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 19, 2014

scabug commented Feb 20, 2014