Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deserializing large mutable.HashMaps results in ArrayIndexOutOfBoundsExceptions #5632

Closed
scabug opened this issue Mar 29, 2012 · 8 comments
Closed
Assignees
Milestone

Comments

@scabug
Copy link

scabug commented Mar 29, 2012

Basically, put 2147484 entries in a hashmap. write to disk. try to read from disk. kaboom! ArrayIndexOutOfBoundsExceptions. 2147483 is fine. Yeah, I did the binary search.

import scala.collection.mutable.HashMap
import java.io._
import java.util.zip._

case class Feature(x: Int) 

object Foo extends App {
  def test(size: Int) { 
    println(size)
    val map = new HashMap[Feature,Double]() ++= (0 until size map {Feature(_) -> 1.0})
    val oout = new ObjectOutputStream(new GZIPOutputStream(new BufferedOutputStream(new FileOutputStream("test.ser.gz"))))
    oout.writeObject(map)
    oout.close()
    val in = new ObjectInputStream(new GZIPInputStream(new BufferedInputStream(new FileInputStream("test.ser.gz"))))
    in.readObject()
    in.close()
  }
  test(2147483)
  test(2147484)
}

java.lang.ArrayIndexOutOfBoundsException: -3733
	at scala.collection.mutable.HashTable$class.addEntry(HashTable.scala:119)
	at scala.collection.mutable.HashMap.addEntry(HashMap.scala:43)
	at scala.collection.mutable.HashTable$class.init(HashTable.scala:81)
	at scala.collection.mutable.HashMap.init(HashMap.scala:43)
	at scala.collection.mutable.HashMap.readObject(HashMap.scala:130)
@scabug
Copy link
Author

scabug commented Mar 29, 2012

Imported From: https://issues.scala-lang.org/browse/SI-5632?orig=1
Reporter: @dlwh
Affected Versions: 2.9.0-1, 2.9.1, 2.10.0-M2
Other Milestones: 2.10.0-M2, 2.10.0

@scabug
Copy link
Author

scabug commented Mar 30, 2012

@dcsobral said:
Collections of size greater than Int.MaxValue are technically illegal, even if some of them work for some things.

@scabug
Copy link
Author

scabug commented Mar 30, 2012

@dlwh said (edited on Mar 30, 2012 8:25:10 PM UTC):
That's all well and good, but this is .1% of that value (almost exactly, so this is probably the problem!)

scala> 2147484 * 1.0 / Int.MaxValue
res5: Double = 0.0010000001643784345

scala> 

@scabug
Copy link
Author

scabug commented Mar 30, 2012

@dlwh said:
Bingo. sizeForThreshold multiplies by loadFactorDenum ( == 1000) without converting to a Long.

  // Hashtable.scala
  private[collection] final def defaultLoadFactor: Int = 750 // corresponds to 75%
  private[collection] final def loadFactorDenum = 1000;  

  // ...
  
  private[collection] final def newThreshold(_loadFactor: Int, size: Int) = ((size.toLong * _loadFactor) / loadFactorDenum).toInt
  
  private[collection] final def sizeForThreshold(_loadFactor: Int, thr: Int) = thr * loadFactorDenum / _loadFactor
  

@scabug
Copy link
Author

scabug commented Mar 30, 2012

@non said:
Good catch! Seems like an easy fix.

@scabug
Copy link
Author

scabug commented Mar 30, 2012

@dlwh said:
yup. Assuming I can make Scala build I'll make a patch and send a pull request real soon now.

@scabug
Copy link
Author

scabug commented Mar 30, 2012

@non said:
OK. Let me know if you need help with any of it.

I encourage you to write a test that catches this issue to help confirm that you fixed it and also prevent future regressions. Makes testing your patch easier on the commiter.

@scabug
Copy link
Author

scabug commented Mar 31, 2012

@dlwh said:
scala/scala#345

@scabug scabug closed this as completed May 3, 2012
@scabug scabug added this to the 2.9.3-RC1 milestone Apr 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants