Scala Programming Language
  1. Scala Programming Language
  2. SI-8104

whitebox materializers are only whitebox in type parameters, not type members

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: Scala 2.10.2, Scala 2.11.0
    • Fix Version/s: Backlog
    • Component/s: None
    • Labels:
      None

      Description

      import scala.reflect.macros.WhiteboxContext
      
      object Macros {
        def impl[T](c: WhiteboxContext)(implicit T: c.WeakTypeTag[T]) = {
          import c.universe._
          import definitions._
          val fields = T.tpe.declarations.toList.collect{ case x: TermSymbol if x.isVal && x.isCaseAccessor => x }
          val Repr = appliedType(TupleClass(fields.length).asType.toType, fields.map(_.typeSignature))
          q"new Generic[$T]{ type Repr = $Repr }"
        }
      }
      
      trait Generic[T] { type Repr }
      object Generic {
        type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 }
        import scala.language.experimental.macros
        implicit def materializeGeneric[T]: Generic[T] = macro Macros.impl[T]
      }
      
      object Test extends App {
        case class C(x: Int, y: Int)
      
        // does work: the implicit is materialized and Repr is successfully inferred as (Int, Int)
        def reprify[T, Repr](x: T)(implicit generic: Generic.Aux[T, Repr]) = ???
        reprify(C(40, 2))
      
        // doesn't work: can't find implicit value for parameter e: Generic.Aux[Test.C,(Int, Int)]
        // implicitly[Generic.Aux[C, (Int, Int)]]
      }
      

      upd. Note that the original bug report looked differently, involving an anonymous class that inherited from a class that takes an implicit of type Generic.Aux. However, it all boiled down to a very simple reproduction, so I decided to update the description. Therefore if initial comments look confusing to you, don't worry - it's okay. If you really wonder about what was going on, take a look at the issue's history.

        Activity

        Hide
        Paul Phillips added a comment -

        Boy, I scratched my head a while before realizing there must be a macro involved. That is pretty uncool that it can so easily masquerade as normal source (although I guess that's a "feature".) The bug is in the macro code. When the compiler is inferring type arguments it widens anonymous classes for exactly this reason.

        object Generic extends LowPriorityGeneric {
          type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 }
        
          // Refinement for products, here we can provide the calling context with
          // a proof that the resulting Repr <: HList
          implicit def product[T <: Product]: Generic[T] = macro GenericMacros.materializeForProduct[T]
        }
        
        Show
        Paul Phillips added a comment - Boy, I scratched my head a while before realizing there must be a macro involved. That is pretty uncool that it can so easily masquerade as normal source (although I guess that's a "feature".) The bug is in the macro code. When the compiler is inferring type arguments it widens anonymous classes for exactly this reason. object Generic extends LowPriorityGeneric { type Aux[T, Repr0] = Generic[T] { type Repr = Repr0 } // Refinement for products, here we can provide the calling context with // a proof that the resulting Repr <: HList implicit def product[T <: Product]: Generic[T] = macro GenericMacros.materializeForProduct[T] }
        Hide
        Eugene Burmako added a comment -

        Could you elaborate on why materializeForProduct fails to expand?

        Show
        Eugene Burmako added a comment - Could you elaborate on why materializeForProduct fails to expand?
        Hide
        Paul Phillips added a comment -

        It's not 100% clear to me and I don't wish to investigate it further, but I do wager the behavior cannot be reproduced without a macro. My supposition would be that something is throwing the inferred type of the expression blindly into a TypeTree in a manner which the compiler itself does not, which leads eventually to it being faced with "anon class foo" where it would normally see "A[T1?, T2?]" and that's always enough to break things.

        Show
        Paul Phillips added a comment - It's not 100% clear to me and I don't wish to investigate it further, but I do wager the behavior cannot be reproduced without a macro. My supposition would be that something is throwing the inferred type of the expression blindly into a TypeTree in a manner which the compiler itself does not, which leads eventually to it being faced with "anon class foo" where it would normally see "A [T1?, T2?] " and that's always enough to break things.
        Hide
        Eugene Burmako added a comment -

        By the way, anonymous classes don't have anything to do with this bug. Replacing `new A((1, 'b')) {}` with `class X extends A[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]]((1, 'b'))` leads to the same compilation error

        23:58 ~/Projects/shapeless-test$ sbt compile
        [info] Set current project to shapeless-test (in build file:/Users/xeno_by/Projects/shapeless-test/)
        [info] Compiling 1 Scala source to /Users/xeno_by/Projects/shapeless-test/target/scala-2.11.0-M7/classes...
        X.super.<init>(scala.Tuple2.apply[Int, Char](1, 'b'))
        [error] /Users/xeno_by/Projects/shapeless-test/Test.scala:13: could not find implicit value for parameter tupleGeneric: shapeless.Generic.Aux[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]]
        [error]   class X extends A[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]]((1, 'b'))
        [error]                   ^
        
        Show
        Eugene Burmako added a comment - By the way, anonymous classes don't have anything to do with this bug. Replacing `new A((1, 'b')) {}` with `class X extends A[(Int, Char),shapeless.::[Int,shapeless.:: [Char,shapeless.HNil] ]]((1, 'b'))` leads to the same compilation error 23:58 ~/Projects/shapeless-test$ sbt compile [info] Set current project to shapeless-test (in build file:/Users/xeno_by/Projects/shapeless-test/) [info] Compiling 1 Scala source to /Users/xeno_by/Projects/shapeless-test/target/scala-2.11.0-M7/classes... X.super.<init>(scala.Tuple2.apply[Int, Char](1, 'b')) [error] /Users/xeno_by/Projects/shapeless-test/Test.scala:13: could not find implicit value for parameter tupleGeneric: shapeless.Generic.Aux[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]] [error] class X extends A[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]]((1, 'b')) [error] ^
        Hide
        Paul Phillips added a comment -

        I didn't mean to place the emphasis on "anon", but on the fact that the relevant type parameters are no longer visible without taking the base type.

        Show
        Paul Phillips added a comment - I didn't mean to place the emphasis on "anon", but on the fact that the relevant type parameters are no longer visible without taking the base type.
        Hide
        Eugene Burmako added a comment - - edited

        This is how this problem manifests itself in shapeless:

        import shapeless._ 
        object Test extends App { 
          implicitly[Generic.Aux[(Int, Char), Int :: Char :: HNil]] 
        } 
        
        [error] ~/Projects/shapeless-test/Test.scala:3: could not find implicit value for parameter e: shapeless.Generic.Aux[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]] 
        [error] implicitly[Generic.Aux[(Int, Char), Int :: Char :: HNil]] 
        [error] ^ 
        

        This happens because of matchesPt discarding Generic.product as an implicit candidate for a lookup for Generic[T1] { type Repr = T2 } with both T1 and T2 being ground types:

        matchesPt(
          => Generic[?]{type Repr <: HList},                     // signature of Generic.product
          Generic.Aux[(Int, Char), ::[Int, ::[Char, HNil]]], Nil // expected type
        ) == false
        

        This is quite reasonable, because the Repr type member ruins the subtyping check underlying matchesPt. Repr's counterpart, the T type parameter of Generic, gets wildcarded for the matchesPt check (denoted by the question mark in the printout), whereas Repr does not.

        Show
        Eugene Burmako added a comment - - edited This is how this problem manifests itself in shapeless: import shapeless._ object Test extends App { implicitly[Generic.Aux[(Int, Char), Int :: Char :: HNil]] } [error] ~/Projects/shapeless-test/Test.scala:3: could not find implicit value for parameter e: shapeless.Generic.Aux[(Int, Char),shapeless.::[Int,shapeless.::[Char,shapeless.HNil]]] [error] implicitly[Generic.Aux[(Int, Char), Int :: Char :: HNil]] [error] ^ This happens because of matchesPt discarding Generic.product as an implicit candidate for a lookup for Generic[T1] { type Repr = T2 } with both T1 and T2 being ground types: matchesPt( => Generic[?]{type Repr <: HList}, // signature of Generic.product Generic.Aux[(Int, Char), ::[Int, ::[Char, HNil]]], Nil // expected type ) == false This is quite reasonable, because the Repr type member ruins the subtyping check underlying matchesPt. Repr's counterpart, the T type parameter of Generic, gets wildcarded for the matchesPt check (denoted by the question mark in the printout), whereas Repr does not.
        Hide
        Eugene Burmako added a comment - - edited

        I have no idea whether there's a principled solution here, since I have only superficial knowledge of how type inference is supposed to work, but luckily there's a very simple workaround that involves converting latent whiteboxity of Generic to explicit whiteboxity of Generic.Aux:

        - implicit def materializeGeneric[T]: Generic[T] = macro Macros.impl[T]
        + implicit def materializeGeneric[T, Repr]: Generic.Aux[T, Repr] = macro Macros.impl[T]
        

        This, however, requires 2.11 and is not going to work in 2.10, because fundep materialization was only merged into 2.11 (https://github.com/scala/scala/pull/2499).

        Show
        Eugene Burmako added a comment - - edited I have no idea whether there's a principled solution here, since I have only superficial knowledge of how type inference is supposed to work, but luckily there's a very simple workaround that involves converting latent whiteboxity of Generic to explicit whiteboxity of Generic.Aux: - implicit def materializeGeneric[T]: Generic[T] = macro Macros.impl[T] + implicit def materializeGeneric[T, Repr]: Generic.Aux[T, Repr] = macro Macros.impl[T] This, however, requires 2.11 and is not going to work in 2.10, because fundep materialization was only merged into 2.11 ( https://github.com/scala/scala/pull/2499 ).
        Hide
        Eugene Burmako added a comment -

        It'd be nice to fix this issue, but it requires a more or less fundamental change in the typechecker, so let's revisit it later, after 2.11.0-final is released.

        Show
        Eugene Burmako added a comment - It'd be nice to fix this issue, but it requires a more or less fundamental change in the typechecker, so let's revisit it later, after 2.11.0-final is released.
        Show
        Eugene Burmako added a comment - Also see https://groups.google.com/forum/#!topic/scala-language/7VMD1TRIywA

          People

          • Assignee:
            Eugene Burmako
            Reporter:
            Nikita Volkov
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development