Google App Engine Support for Future #8362

scabug · 2014-03-06T10:00:47Z

When I try to use Scala's scala.concurrent.Future on Google App Engine, it fails because access to sun.misc.Unsafe is blocked by App Engine's security manager. This is reproducible using the App Engine Java SDK without uploading anything to Google, or needing a Google account:

HTTP ERROR 500

Problem accessing /test. Reason:

    INTERNAL_SERVER_ERROR

Caused by:

java.lang.ExceptionInInitializerError
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:26)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.<init>(Future.scala:20)
	at scala.concurrent.impl.Future$.apply(Future.scala:30)
	at scala.concurrent.Future$.apply(Future.scala:492)
	at zappenginefutures.TestServlet.doGet(TestServlet.scala:20)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:487)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.lang.ExceptionInInitializerError
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:32)
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:24)
	... 41 more
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
	at java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.security.AccessController.checkPermission(Unknown Source)
	at java.lang.SecurityManager.checkPermission(Unknown Source)
	at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:431)
	at java.lang.SecurityManager.checkPackageAccess(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.ClassLoader.checkPackageAccess(Unknown Source)
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:22)
	... 42 more

Caused by:

java.lang.ExceptionInInitializerError
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:32)
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.<init>(Future.scala:20)
	at scala.concurrent.impl.Future$.apply(Future.scala:30)
	at scala.concurrent.Future$.apply(Future.scala:492)
	at zappenginefutures.TestServlet.doGet(TestServlet.scala:20)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:487)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
	at java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.security.AccessController.checkPermission(Unknown Source)
	at java.lang.SecurityManager.checkPermission(Unknown Source)
	at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:431)
	at java.lang.SecurityManager.checkPackageAccess(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.ClassLoader.checkPackageAccess(Unknown Source)
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:22)
	... 42 more

Caused by:

java.security.AccessControlException: access denied ("java.lang.RuntimePermission" "accessClassInPackage.sun.misc")
	at java.security.AccessControlContext.checkPermission(Unknown Source)
	at java.security.AccessController.checkPermission(Unknown Source)
	at java.lang.SecurityManager.checkPermission(Unknown Source)
	at com.google.appengine.tools.development.DevAppServerFactory$CustomSecurityManager.checkPermission(DevAppServerFactory.java:431)
	at java.lang.SecurityManager.checkPackageAccess(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.lang.ClassLoader$1.run(Unknown Source)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.lang.ClassLoader.checkPackageAccess(Unknown Source)
	at scala.concurrent.util.Unsafe.<clinit>(Unsafe.java:22)
	at scala.concurrent.impl.AbstractPromise.<clinit>(AbstractPromise.java:24)
	at scala.concurrent.impl.Future$PromiseCompletingRunnable.<init>(Future.scala:20)
	at scala.concurrent.impl.Future$.apply(Future.scala:30)
	at scala.concurrent.Future$.apply(Future.scala:492)
	at zappenginefutures.TestServlet.doGet(TestServlet.scala:20)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1166)
	at com.google.appengine.api.socket.dev.DevSocketFilter.doFilter(DevSocketFilter.java:74)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.ResponseRewriterFilter.doFilter(ResponseRewriterFilter.java:123)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.HeaderVerificationFilter.doFilter(HeaderVerificationFilter.java:34)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.api.blobstore.dev.ServeBlobFilter.doFilter(ServeBlobFilter.java:63)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.apphosting.utils.servlet.TransactionCleanupFilter.doFilter(TransactionCleanupFilter.java:43)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.StaticFileFilter.doFilter(StaticFileFilter.java:125)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectRequest(DevAppServerModulesFilter.java:366)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doDirectModuleRequest(DevAppServerModulesFilter.java:349)
	at com.google.appengine.tools.development.DevAppServerModulesFilter.doFilter(DevAppServerModulesFilter.java:116)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1157)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:388)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:418)
	at com.google.appengine.tools.development.DevAppEngineWebAppContext.handle(DevAppEngineWebAppContext.java:97)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at com.google.appengine.tools.development.JettyContainerService$ApiProxyHandler.handle(JettyContainerService.java:487)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:923)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:547)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

The basic code I'm using to reproduce this is below, my actual code was somewhat more complicated and involved blocking HTTP access in my threads:

package zappenginefutures

import java.util.concurrent.Executors
import java.util.concurrent.TimeUnit

import scala.concurrent.ExecutionContext
import scala.concurrent.Future

import com.google.appengine.api.ThreadManager

import javax.servlet.http.HttpServlet
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse

class TestServlet extends HttpServlet {
  override def doGet(req: HttpServletRequest, resp: HttpServletResponse) {
    implicit val executionContext = ExecutionContext.fromExecutorService(
      Executors.newFixedThreadPool(4, ThreadManager.currentRequestThreadFactory))

    val f = Future {
      Thread.sleep(1000L)
    }

    executionContext.shutdown()
    executionContext.awaitTermination(10, TimeUnit.SECONDS)
    
    resp.getWriter.println("OK")
  }
}

The text was updated successfully, but these errors were encountered:

scabug · 2014-03-06T10:00:47Z

Imported From: https://issues.scala-lang.org/browse/SI-8362?orig=1
Reporter: Carey Evans (carey)
Affected Versions: 2.11.0-RC1, 2.11.0, 2.11.1, 2.11.2, 2.11.3, 2.11.4, 2.11.5, 2.11.6

scabug · 2014-03-06T18:04:56Z

@adriaanm said (edited on Mar 6, 2014 7:17:02 PM UTC):
I'd recommend asking on Stack Overflow or scala-user for work arounds, as we don't really support GAE (although we'd like to).

[EDIT: please ignore the rest of this comment -- it was more of a stream of consciousness than anything else]

My un-educated guess is that this is because we ship with our own copy of the fork join implementation, since we target Java 6. It looks like GAE requires using the one that comes with the/their jdk. I'll ask around for a work around.

Could you try this suggestion from SO? http://stackoverflow.com/a/20627692/276895

Specifically, create an execution context that uses the jdk's fork join. It would be better if this just worked, though.

implicit val execCtx = scala.concurrent.ExecutionContext fromExecutorService new java.util.concurrent.ForkJoinPool

Sorry, I see you're already doing this. Missed that line…

--

Looks like my diagnosis was wrong again, it's another use of Unsafe in AbstractPromise, as you say in the title. I'll go have some more coffee now and leave it to the experts.

scabug · 2014-03-06T19:30:42Z

@adriaanm said (edited on Mar 6, 2014 9:47:28 PM UTC):
Looks like it regressed in scala/scala#454 -- see comments there for potential improvements in 2.11.1

scabug · 2014-03-06T19:32:51Z

Carey Evans (carey) said:
I tried replacing scala.concurrent.util.FuturePromise with one that accesses sun.misc.Unsafe.compareAndSwapObject() officially via AtomicReferenceFieldUpdater, rather than unofficially via scala.concurrent.util.Unsafe, and that works fine in the App Engine development server:

package scala.concurrent.impl;

import java.util.concurrent.atomic.AtomicReferenceFieldUpdater;

public abstract class AbstractPromise {
	protected final static AtomicReferenceFieldUpdater<AbstractPromise, Object> updater = AtomicReferenceFieldUpdater
			.newUpdater(AbstractPromise.class, Object.class, "_ref");

	private volatile Object _ref;

	protected final Object getState() {
		return _ref;
	}

	protected final boolean updateState(Object oldState, Object newState) {
		return updater.compareAndSet(this, oldState, newState);
	}
}

Presumably the current code is a micro-optimisation that makes a noticeable difference even after Hotspot has inlined all the method calls. It would be nice if Scala could fall back to the standard Java API instead of raising java.lang.ExceptionInInitializerError when the internal Java classes are not accessible.

scabug · 2014-03-08T09:29:37Z

Carey Evans (carey) said:
I've pushed a change to https://github.com/carey/scala/commit/72e1927793c6dc808d9896e5571a97f136fd1a4e that should fix this with no change to performance in the normal case. The backward-compatibility check in the Ant tests doesn't understand that C:\ is an absolute path on Windows, though, so I'll need to get it going in Linux to test it.

scabug · 2014-08-05T15:12:54Z

@gkossakowski said:
The 2.11.2 is out so I'm rescheduling the issue for 2.11.3.

scabug · 2014-11-04T03:59:13Z

@retronym said:
Updating fix-by version to 2.11.5.

scabug · 2015-02-24T00:03:41Z

@adriaanm said:
scala/scala#4313

scabug · 2015-06-19T19:59:30Z

@SethTisue said:
newer PR is scala/scala#4443

scabug · 2015-07-24T01:43:36Z

Bart Jenkins (bauhaus9) said:
Yes, please fix this issue. It is stopping me from using Autowire with Scala-JS.

scabug closed this as completed Aug 6, 2015

scabug added enhancement quickfix library help wanted labels Apr 7, 2017

scabug added this to the 2.11.8 milestone Apr 7, 2017

scabug assigned retronym Apr 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Google App Engine Support for Future #8362

Google App Engine Support for Future #8362

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 8, 2014

scabug commented Aug 5, 2014

scabug commented Nov 4, 2014

scabug commented Feb 24, 2015

scabug commented Jun 19, 2015

scabug commented Jul 24, 2015

Google App Engine Support for Future #8362

Google App Engine Support for Future #8362

Comments

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 6, 2014

scabug commented Mar 8, 2014

scabug commented Aug 5, 2014

scabug commented Nov 4, 2014

scabug commented Feb 24, 2015

scabug commented Jun 19, 2015

scabug commented Jul 24, 2015