Scala quasiquote invalid behavior of the type system #9711

scabug · 2016-03-20T21:49:58Z

There's something really off about the type inference when I use quasiquotes

Take this example (from Programming Scala book, 2nd Edition):

// src/main/scala/progscala2/metaprogramming/invariant2.scala
package metaprogramming

import scala.language.experimental.macros
import scala.reflect.api.Trees

import scala.reflect.macros.blackbox.Context

/**
  * A Macro written using the current macro syntax along with quasiquotes.
  * Requires a predicate for an invariant to be true before each expression
  * is evaluated.
  */
object invariant2 {

  def execute[T](predicate: => Boolean)(block: => T): T = macro executeMacro

  def executeMacro(context: Context)(predicate: context.Tree)(block: context.Tree) = {
    import context.universe._
    val predicateAsString = showCode(predicate)

    type SyntaxTree = context.Tree
    type TreeNode = Trees#Tree // a syntax tree node that is in and of itself a tree

    val q"..$stmts" = block
    val statements = stmts
    val statementsWithInvariants: Seq[SyntaxTree] = statements.flatMap { statement =>
      // showCode requires "context.universe.Tree"
      val exceptionMessage = s"FAILURE! $predicateAsString == false, for statement: " + showCode(statement)
      val throwStatement = q"throw new metaprogramming.invariant.InvariantFailure($exceptionMessage)"
      val predicateStatement = q"if (false == $predicate) $throwStatement"
      List(q"{ val tmp = $statement; $predicateStatement; tmp };")
    }
    val throwStatement = q"throw new metaprogramming.invariant.InvariantFailure($predicateAsString)"
    val predicateStatement = q"if (false == $predicate) $throwStatement"
    q"$predicateStatement; ..$statementsWithInvariants"
  }

  case class InvariantFailure(msg: String) extends RuntimeException(msg)
}

^ Everything works. Now look at what happens when I ask for the types:

type of "statements":
Pattern: statements: scala.Seq[Trees#Tree]

type of "statement":
statement: Trees#Tree

Okay. Now I put these type annotations on "statement" and "statements" and compile

// src/main/scala/progscala2/metaprogramming/invariant2.scala
package metaprogramming

import scala.language.experimental.macros
import scala.reflect.api.Trees

import scala.reflect.macros.blackbox.Context

/**
  * A Macro written using the current macro syntax along with quasiquotes.
  * Requires a predicate for an invariant to be true before each expression
  * is evaluated.
  */
object invariant2 {

  def execute[T](predicate: => Boolean)(block: => T): T = macro executeMacro

  def executeMacro(context: Context)(predicate: context.Tree)(block: context.Tree) = {
    import context.universe._
    val predicateAsString = showCode(predicate)

    type SyntaxTree = context.Tree
    type TreeNode = Trees#Tree // a syntax tree node that is in and of itself a tree

    val q"..$stmts" = block
    val statements: scala.Seq[Trees#Tree] = stmts
    val statementsWithInvariants: Seq[SyntaxTree] = statements.flatMap { (statement: Trees#Tree) =>
      // showCode requires "context.universe.Tree"
      val exceptionMessage = s"FAILURE! $predicateAsString == false, for statement: " + showCode(statement)
      val throwStatement = q"throw new metaprogramming.invariant.InvariantFailure($exceptionMessage)"
      val predicateStatement = q"if (false == $predicate) $throwStatement"
      List(q"{ val tmp = $statement; $predicateStatement; tmp };")
    }
    val throwStatement = q"throw new metaprogramming.invariant.InvariantFailure($predicateAsString)"
    val predicateStatement = q"if (false == $predicate) $throwStatement"
    q"$predicateStatement; ..$statementsWithInvariants"
  }

  case class InvariantFailure(msg: String) extends RuntimeException(msg)
}

Compile/Run:

{quote}> test:run
[info] Compiling 1 Scala source to /home/johnreed/sbtProjects/scala-trace-debug/target/scala-2.11/test-classes...
[error] /home/johnreed/sbtProjects/scala-trace-debug/src/test/scala/mataprogramming/invariant2.scala:29: type mismatch;
[error] found : scala.reflect.api.Trees#Tree
[error] required: context.universe.Tree
[error] val exceptionMessage = s"FAILURE! $predicateAsString == false, for statement: " + showCode(statement)
[error] ^
[error] one error found
[error] (test:compile) Compilation failed
[error] Total time: 0 s, completed Mar 20, 2016 5:40:33 PM
{quote}

This is very peculiar. The type inference told me that "statement" IS of type Trees#Tree

What if I annotate it with "context.universe.Tree" instead

// src/main/scala/progscala2/metaprogramming/invariant2.scala
package metaprogramming

import scala.language.experimental.macros
import scala.reflect.api.Trees

import scala.reflect.macros.whitebox.Context

/**
  * A Macro written using the current macro syntax along with quasiquotes.
  * Requires a predicate for an invariant to be true before each expression
  * is evaluated.
  */
object invariant2 {

  def execute[T](predicate: => Boolean)(block: => T): T = macro executeMacro

  def executeMacro(context: Context)(predicate: context.Tree)(block: context.Tree) = {
    import context.universe._
    val predicateAsString = showCode(predicate)

    type SyntaxTree = context.Tree
    type TreeNode = Trees#Tree // a syntax tree node that is in and of itself a tree

    val q"..$stmts" = block
    val statements: scala.Seq[context.universe.Tree] = stmts
    val statementsWithInvariants: Seq[SyntaxTree] = statements.flatMap { (statement: context.universe.Tree) =>
      // showCode requires "context.universe.Tree"
      val exceptionMessage = s"FAILURE! $predicateAsString == false, for statement: " + showCode(statement)
      val throwStatement = q"throw new metaprogramming.invariant.InvariantFailure($exceptionMessage)"
      val predicateStatement = q"if (false == $predicate) $throwStatement"
      List(q"{ val tmp = $statement; $predicateStatement; tmp };")
    }
    val throwStatement = q"throw new metaprogramming.invariant.InvariantFailure($predicateAsString)"
    val predicateStatement = q"if (false == $predicate) $throwStatement"
    q"$predicateStatement; ..$statementsWithInvariants"
  }

  case class InvariantFailure(msg: String) extends RuntimeException(msg)
}

The IDE says "expression of type Tree#Tree does not conform to type context.universe.Tree", but it compiles. What is going on with the wacky type inference here?

scabug · 2016-03-20T21:49:58Z

Imported From: https://issues.scala-lang.org/browse/SI-9711?orig=1
Reporter: John Reed (JohnReedlol)
Affected Versions: 2.11.7

scabug · 2016-03-20T23:02:42Z

@retronym said:
Your description sounds like you are reporting an IDE bug. ("When I explicitly add in the types that the IDE tells me are inferred, my program fails to compile.")

What IDE are you using? Scala IDE's bug tracker is https://www.assembla.com/wiki/show/scala-ide/bug_reporting, and and IntellIJ's bug tracker is https://youtrack.jetbrains.com/issues/SCL

scabug · 2016-03-21T04:58:58Z

John Reed (JohnReedlol) said (edited on Mar 21, 2016 5:00:27 AM UTC):
http://stackoverflow.com/questions/36118298/scala-quasiquote-macro-example-broken-type-signatures-off

^ I don't think that this is just an IDE bug. The IDE gets its type information by querying. That being said, I tried two different versions of IntelliJ

IntelliJ IDEA 2016.1
Build #IC-145.258, built on March 17, 2016
JRE: 1.8.0_76-release-b18 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o

IntelliJ IDEA 15.0.4
Build #IC-143.1821.5, built on February 23, 2016

Are you sure that there isn't something that is providing IntelliJ with the wrong types of the expressions? Like "toType" or :type or reflection or something?

scabug · 2016-03-21T05:13:36Z

@som-snytt said:
Back-reference is this question.

OK, so now that I know the format of external links in JIRA, using the "chain links" icon in the editor is kind of useless. Selecting the dummy text is impossible for anyone over 25. It makes me love github all the more. (I can make an ageist joke because I am of age?)

scabug · 2016-03-21T05:17:18Z

@som-snytt said:
I guess it doesn't need to be said that splitting convo's over jira, SO, email and twitter is so early 2010's. Our children will laugh at us. For those of us lucky enough to suffer ridicule.

scabug · 2016-03-21T07:32:40Z

@retronym said:
IntelliJ does not query the Scala presentation compiler for its type information, it has an independent implementation of the a presentation compiler. In general, it does not offer code assistance based on the types computed by "whitebox macros" (quasiquotes are an example of these.) They do have some hard coded support for the quasiquote macro, which looks to be a fairly rough approximation: JetBrains/intellij-scala@8d259f3#diff-0249f636971b724322d805f318ef7239R157

The inherent difficulty in getting good IDE support when macros are allowed to compute arbitrary result types is why we created the distinction between "whitebox" and "blackbox" macros in Scala 2.11, and advice macro authors to favour blackbox macros when this is sufficiently powerful. Unfortunately pattern matching with quasiquotes really requires whitebox macros.

scabug · 2016-03-23T04:24:50Z

John Reed (JohnReedlol) said:
okay. It's probably not your job to do this, but another thing that looks like a bug is this:

macro implementation has incompatible shape:
[error]  required: (c: scala.reflect.macros.blackbox.Context)(x: c.Expr[=> T]): c.Expr[T]

^ It says that I can have a macro with that type signature, but when I copy and paste the type signature, I get

identifier expected but '=>' found.  c.Expr[=> T]

^ They should really remove the arrow from the warning in the type signature because you can't actually have a type signature like that. I ended up having to do:

def impl[T](c: scala.reflect.macros.blackbox.Context)(x: c.Expr[T]): c.Expr[T]

instead.

Side side note: WRITING AND TESTING MACRO CODE IS A HUGE PAIN IN THE ASS. I mean If I have to write like 50 macros and each macro needs test cases and I need to keep adjusting the macros and re-writing them, I keep getting:

macro implementation not found: apply
[error] (the most common reason for that is that you cannot use macro implementations in the same compilation run that defines them)

^ I ended up having to put all my macro tests into one huge file and comment the whole thing out for every single compile.

scabug · 2016-03-23T05:44:42Z

@retronym said:
See http://docs.scala-lang.org/overviews/macros/overview.html#using-macros-with-maven-or-sbt for tips on how to structure your project to put test cases in a place where they can use the macros they are testing.

scabug · 2016-03-23T05:50:13Z

@retronym said:
Steps to reproduce the problem with matching up the Expr types:

scala> import reflect.macros.blackbox._, language.experimental.macros
import reflect.macros.blackbox._
import language.experimental.macros

scala> def impl(c: Context) = c.literalUnit
warning: there was one deprecation warning; re-run with -deprecation for details
impl: (c: scala.reflect.macros.blackbox.Context)c.Expr[Unit]

scala> def m(a: => Int) = macro impl
<console>:18: error: macro implementation has incompatible shape:
 required: (c: scala.reflect.macros.blackbox.Context)(a: c.Expr[=> Int]): c.Expr[Unit]
 or      : (c: scala.reflect.macros.blackbox.Context)(a: c.Tree): c.Tree
 found   : (c: scala.reflect.macros.blackbox.Context): c.Expr[Unit]
number of parameter sections differ
       def m(a: => Int) = macro impl
                                ^

scala> def impl(c: Context)(a: c.Expr[=> Int]): c.Expr[Unit] = c.literalUnit
<console>:1: error: identifier expected but '=>' found.
def impl(c: Context)(a: c.Expr[=> Int]): c.Expr[Unit] = c.literalUnit
                               ^

scala> def impl(c: Context)(a: c.Expr[Int]): c.Expr[Unit] = c.literalUnit
warning: there was one deprecation warning; re-run with -deprecation for details
impl: (c: scala.reflect.macros.blackbox.Context)(a: c.Expr[Int])c.Expr[Unit]

scala> def m(a: => Int) = macro impl
<console>:18: error: macro implementation has incompatible shape:
 required: (c: scala.reflect.macros.blackbox.Context)(a: c.Expr[=> Int]): c.Expr[Unit]
 or      : (c: scala.reflect.macros.blackbox.Context)(a: c.Tree): c.Tree
 found   : (c: scala.reflect.macros.blackbox.Context)(a: c.Expr[Int]): c.Expr[Unit]
type mismatch for parameter a: c.Expr[=> Int] does not conform to c.Expr[Int]
       def m(a: => Int) = macro impl
                                ^

The workaround is to use macro implementations from Tree => Tree, or to avoid using by-name parameters in the macro declaration itself. By-name parameters don't really make a difference: a macro argument is not evaluated eagerly nor lazily, instead the AST is passed to the macro implementation at compile time, and the macro implementation has control over whether and when it is evaluated.

scabug · 2016-03-23T07:10:30Z

@som-snytt said:
I guess it really is his job. Both questions come up on the ML and SO. I recently encountered them during my annual macro refresher exercise. I think after about 50 macros, one would in fact start to get it right.

scabug · 2016-03-23T19:15:56Z

John Reed (JohnReedlol) said (edited on Mar 23, 2016 7:16:37 PM UTC):
@A. P. Marki "I think after about 50 macros, one would in fact start to get it right."

Dude. I'm not a professional developer and I'm not even 23. I just need to replace my debug tools from using function calls to using macros and I'm on macro number 4. But I'm sure that when I'm done with this re-write I will get it right.

Anyway, @jason Zaugg - how do I make it so that in my SBT project I only compile the macros and nothing else and then I only compile the macro tests and nothing else? I don't want to run into "macro implementation not found" anymore.

My sbt project has two files with macros in it:

"src/main/scala/package/Macros1.scala",
"src/main/scala/package/Macros2.scala"

and it has two files with macro tests in it:

"src/test/scala/package/Tests1.scala",
"src/test/scala/package/Tests2.scala"

If I'm going to be modifying these files, would it be better to do like "sbt console :load src/test/scala/package/Tests1.scala" or "sbt test:compile"?

scabug · 2016-03-23T19:28:31Z

@SethTisue said:
John, please use another forum for this question, such as the Scala mailing lists, Stack Overflow, Gitter, etc. See http://www.scala-lang.org/community/ . This ticket has been closed and we're well off the original topic.

scabug · 2016-03-23T23:31:37Z

@som-snytt said:
Just to rephrase, I just experienced these hiccups again when writing a quickie macro. They may be FAQ-worthy, or in the case of the shape error message, fix-worthy. I think I entered through the sbt door: http://www.scala-sbt.org/0.13/docs/Macro-Projects.html Obviously, it's easier when you're at steady-state churning out macro sausage.

On my first job, at eighteen, when a client on the phone began, "I'm a professional programmer...," we knew it was trouble.

scabug closed this as completed Mar 21, 2016

scabug added quickfix won't fix backend macros reflection infer labels Apr 7, 2017

scabug assigned retronym Apr 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Scala quasiquote invalid behavior of the type system #9711

Scala quasiquote invalid behavior of the type system #9711

scabug commented Mar 20, 2016

scabug commented Mar 20, 2016

scabug commented Mar 20, 2016

scabug commented Mar 21, 2016

scabug commented Mar 21, 2016

scabug commented Mar 21, 2016

scabug commented Mar 21, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

Scala quasiquote invalid behavior of the type system #9711

Scala quasiquote invalid behavior of the type system #9711

Comments

scabug commented Mar 20, 2016

scabug commented Mar 20, 2016

scabug commented Mar 20, 2016

scabug commented Mar 21, 2016

scabug commented Mar 21, 2016

scabug commented Mar 21, 2016

scabug commented Mar 21, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016

scabug commented Mar 23, 2016