Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Future not completed and onFailure not called when OOM or InterruptedException is thrown #9554

Closed
scabug opened this issue Nov 11, 2015 · 14 comments
Assignees
Milestone

Comments

@scabug
Copy link

scabug commented Nov 11, 2015

scala.concurrent.Future is not completed and onFailure is not called on every case when attached example code is executed. When normal exceptions and "new Error()" is thrown it works as documented, but runs to timeout when OOM / IntEx is thrown. Document clearly states that on IntEx ExectutionException should be the result (and OOM is subclass of Error)

scala.concurrent.Future onFailure document says:

The future may contain a throwable object and this means that the future failed. Futures obtained through combinators have the same exception as the future they were obtained from. The following throwable objects are not contained in the future:

Error - errors are not contained within futures
InterruptedException - not contained within futures
all scala.util.control.ControlThrowable except NonLocalReturnControl - not contained within futures

Instead, the future is completed with a ExecutionException with one of the exceptions above as the cause. If a future is failed with a scala.runtime.NonLocalReturnControl, it is completed with a value from that throwable instead.

@scabug
Copy link
Author

scabug commented Nov 11, 2015

Imported From: https://issues.scala-lang.org/browse/SI-9554?orig=1
Reporter: Jani Räty (jraty)
Affected Versions: 2.11.7
Attachments:

  • bug.scala (created on Nov 11, 2015 9:01:16 PM UTC, 703 bytes)

@NthPortal
Copy link

NthPortal commented Feb 18, 2018

This bug does not occur if a Promise is completed with a Failure containing a fatal Throwable; only if a fatal Throwable is thrown during execution of a Future's body or callback.

@NthPortal
Copy link

Looking into this, I've discovered that CallbackRunnable does not handle Throwables properly. It only handles NonFatals; as it is where all Future bodies and callbacks are run, it needs to handle all Throwables.

@viktorklang
Copy link

@NthPortal Behavior in the face of fatal exception is undefined. Which version are you referring to specifically? And what code are you saying should be handled differently?

@NthPortal
Copy link

NthPortal commented Feb 18, 2018

@viktorklang This issue says the problem says that Futures are not completed for OOM and InterruptedException; in fact, they are not completed for any fatal Throwables.

Currently, the documentation for onFailure (soon onComplete) states that fatal exceptions/errors get wrapped in ExecutionExceptions. If that is not the case, and their behaviour is meant to be undefined (in this case never completed), then the documentation should be updated to reflect that.

I'm referring to 2.13, though the problem exists for 2.12 as well.

I was referring to run and executeWithValue (here for 2.12), which only catch NonFatals. Looking more closely, it's actually transform and transformWith which prevent the Future from completing (although it would be weird to handle fatals to complete Futures with them, but not report them in the ExecutionContext).

@viktorklang
Copy link

@NthPortal Where does the docs for the onFailure method say that fatals will get wrapped? (I read it a couple of times now but must have missed it or interpreted it differently)

@NthPortal
Copy link

NthPortal commented Feb 18, 2018

@viktorklang It doesn't explicitly say all fatals get wrapped, but it mentions InterruptedException and ControlThrowable, which are both fatal

Edit: it also mentions Error, which is ambiguous. All the other fatals are Errors, so you'd probably expect them to be wrapped because of that, but it doesn't explicitly say all Errors

@viktorklang
Copy link

@NthPortal Yeah, the docs clearly needs to be more explicit about this. I'll have a look at that as a part of my Futures overhaul.

@NthPortal
Copy link

@viktorklang Another thing I noticed is that Promises cannot be completed with exceptions that happen in the process of execution, but before (or after, I guess) the body is executed. For example, if ExecutionContext.execute throws a RejectedExecutionException, the Promise/Future will never complete, though it should probably fail with said exception.

Personally, I think Futures should contain wrapped fatals (otherwise the Future never completes, and also the fatal gets ignored), but it's the community's decision how it should behave.

@viktorklang
Copy link

@NthPortal The first case (REE thrown) is fixed in the overhaul. The second case (auto-wrapping fatals) won't happen. If you want to have fatals propagate, you are free to add such wrapping explicitly (add the fatal as cause to a RuntimeException and throw that)

@NthPortal
Copy link

@viktorklang Okay, that seems pretty reasonable. I assume they will still get reported though

@viktorklang
Copy link

@NthPortal I just implemented and verified support for both interruptions and rejections here: viktorklang/scala-futures@2293c15#diff-823e12da79dd0256861a4903168fab59R344

I hope all mods will be ready in time for 2.13 but cannot guarantee it.

@NthPortal
Copy link

@viktorklang Cheers!

@viktorklang
Copy link

With the Future code in 2.13.0-M5 InterruptedExceptions should now be correctly handled.
Fatal errors should also be rethrown such that they either surface in the Executor(Service) and can be dealt with there. Keep in mind that all execution after a Fatal Error is undefined.

I'd more than welcome some feedback on the new Future handling here.

otterc pushed a commit to linkedin/spark that referenced this issue Mar 22, 2023
When OutOfMemoryError thrown from BroadcastExchangeExec, scala.concurrent.Future will hit scala bug – scala/bug#9554, and hang until future timeout:

We could wrap the OOM inside SparkException to resolve this issue.

Manually tested.

Author: jinxing <jinxing6042@126.com>

Closes apache#21342 from jinxing64/SPARK-24294.

(cherry-picked from commit b7a036b)

Ref: LIHADOOP-40177

RB=1414263
BUG=LIHADOOP-40177
R=fli,mshen,yezhou
A=yezhou
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants