Scala Programming Language
  1. Scala Programming Language
  2. SI-4603

Unable to inherit from Hadoop Mapper (new or old API) in 2.9.0 with code that functions in 2.8.1

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: Scala 2.10.0
    • Component/s: Misc Compiler
    • Labels:
      None

      Description

      === What steps will reproduce the problem (please be specific and use wikiformatting)? ===

      The following code works in 2.8.1:

      import org.apache.hadoop.io.{NullWritable, LongWritable, Text}
      import org.apache.hadoop.mapreduce.{Mapper, Job}
      
      
      object MyJob {
        def main(args:Array[String]) {
          val job = new Job(new Configuration())
          job.setMapperClass(classOf[MyMapper])
          
        }
      }
      
      class MyMapper extends Mapper[LongWritable,Text,Text,Text] {
        override def map(key: LongWritable, value: Text, context: Mapper[LongWritable,Text,Text,Text]#Context) {
      
        }
      }
      
      

      === What do you see instead? ===
      I get this compiler error:

      error: type mismatch;
      found   : java.lang.Class[MyMapper](classOf[MyMapper])
      required: java.lang.Class[_ <: org.apache.hadoop.mapreduce.Mapper]
      job.setMapperClass(classOf[MyMapper])
      

      === Additional information ===
      Daniel Sobral vry helpfully directed me here from my Stack Overflow question at (http://stackoverflow.com/questions/6028221/how-does-one-implement-a-hadoop-mapper-in-scala-2-9-0).

      The method definition of the failing call (Job.setMapperClass) is this:

      public void setMapperClass(java.lang.Class<? extends org.apache.hadoop.mapreduce.Mapper> cls) throws java.lang.IllegalStateException { /* compiled code */ }
      

      The method definition of Mapper itself is this:

      public class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT>

      === What versions of the following are you using? ===

      • Scala: 2.8.1 and 2.9.0
      • Java: latest
      • Operating system: OSX / Linux

        Activity

        Hide
        Chris Bissell added a comment -

        Verified that the problem is fixed in 2.9.1.RC3 for our code. Many thanks! And the compiler is snappier too!

        Show
        Chris Bissell added a comment - Verified that the problem is fixed in 2.9.1.RC3 for our code. Many thanks! And the compiler is snappier too!
        Hide
        Jiamin Zhao added a comment - - edited

        Will this fix be backported to 2.8.1 or 2.9.0? Cuz I'm using Lift and GridGain. I have the same problem even with 2.8.1.

        Show
        Jiamin Zhao added a comment - - edited Will this fix be backported to 2.8.1 or 2.9.0? Cuz I'm using Lift and GridGain. I have the same problem even with 2.8.1.
        Hide
        Federico Ragona added a comment - - edited

        Hello,
        I'm facing a similar issue in Scala 2.10.3, only when extending a Java class from a Scala class. This issue is not reproduced if AMapper (see code below) is implemented as a Scala class.

        Code

        Here is my project layout:

        src
          main
            java
              myhadoop
                AMapper.java
            scala
              myhadoop
                MyMapper.scala
        
        

        And these are my classes' sources:

        AMapper.java source code
        package myhadoop;
        
        import org.apache.hadoop.io.BytesWritable;
        import org.apache.hadoop.io.Text;
        import org.apache.hadoop.mapreduce.Mapper;
        
        public class AMapper extends Mapper<Text, BytesWritable, Text, Text> {
          public void handleThis(Text key, Mapper<Text, BytesWritable, Text, Text>.Context context, Text text) {
            // not relevant to this test
          }
        }
        
        MyMapper.scala source code
        package myhadoop
        
        import org.apache.hadoop.io.{BytesWritable, Text}
        import org.apache.hadoop.mapreduce.Mapper
        
        class MyMapper extends AMapper {
         
          override def map(key: Text, value: BytesWritable, context: Mapper[Text, BytesWritable, Text, Text]#Context) {
            handleThis(key, context, new Text(value.getBytes))
          }
        }
        

        Error

        Compiling this code will raise the following error:

        error: type mismatch;
        found: org.apache.hadoop.mapreduce.Mapper[org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text]#Context
        required: org.apache.hadoop.mapreduce.Mapper[KEYIN,VALUEIN,KEYOUT,VALUEOUT]#Context
        

        pointing to the handleThis(key, context, new Text(bytes)) line.

        Environment

        Scala 2.10.3
        Java 1.7.0u21
        Ubuntu Linux 12.04

        Show
        Federico Ragona added a comment - - edited Hello, I'm facing a similar issue in Scala 2.10.3, only when extending a Java class from a Scala class. This issue is not reproduced if AMapper (see code below) is implemented as a Scala class. Code Here is my project layout: src main java myhadoop AMapper.java scala myhadoop MyMapper.scala And these are my classes' sources: AMapper.java source code package myhadoop; import org.apache.hadoop.io.BytesWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class AMapper extends Mapper<Text, BytesWritable, Text, Text> { public void handleThis(Text key, Mapper<Text, BytesWritable, Text, Text>.Context context, Text text) { // not relevant to this test } } MyMapper.scala source code package myhadoop import org.apache.hadoop.io.{BytesWritable, Text} import org.apache.hadoop.mapreduce.Mapper class MyMapper extends AMapper { override def map(key: Text, value: BytesWritable, context: Mapper[Text, BytesWritable, Text, Text]#Context) { handleThis(key, context, new Text(value.getBytes)) } } Error Compiling this code will raise the following error: error: type mismatch; found: org.apache.hadoop.mapreduce.Mapper[org.apache.hadoop.io.Text,org.apache.hadoop.io.BytesWritable,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text]#Context required: org.apache.hadoop.mapreduce.Mapper[KEYIN,VALUEIN,KEYOUT,VALUEOUT]#Context pointing to the handleThis(key, context, new Text(bytes)) line. Environment Scala 2.10.3 Java 1.7.0u21 Ubuntu Linux 12.04
        Hide
        Simon Ochsenreither added a comment -

        Hi Frederico, could you provide some self-contained piece of code? That would make it easier and faster to investigate this issue. Thanks!

        Show
        Simon Ochsenreither added a comment - Hi Frederico, could you provide some self-contained piece of code? That would make it easier and faster to investigate this issue. Thanks!
        Hide
        Federico Ragona added a comment -

        Simon, I have to apologize to you.
        I thought that the snippet I pasted in my original comment was enough to reproduce the issue but it's definitely not. I just edited my original comment to provide you with all the information needed to reproduce it. As I mentioned in the comment, the compiler will raise this issue only when extending a Java class.

        I hope this helps you.

        Show
        Federico Ragona added a comment - Simon, I have to apologize to you. I thought that the snippet I pasted in my original comment was enough to reproduce the issue but it's definitely not. I just edited my original comment to provide you with all the information needed to reproduce it. As I mentioned in the comment, the compiler will raise this issue only when extending a Java class. I hope this helps you.

          People

          • Assignee:
            Martin Odersky
            Reporter:
            Chris Bissell
            TracCC:
            Ismael Juma, Paul Phillips, Robbie Coleman
          • Votes:
            8 Vote for this issue
            Watchers:
            17 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development