Imported From: https://issues.scala-lang.org/browse/SI-4599?orig=1
Reporter: J (tumeow)

J (tumeow) said:
Note the code there is a full scala script that can be run on your end to test.

@dcsobral said:
The problem is possibly related to the fact that stream:stream, the first tag of all, is never closed.

J (tumeow) said:
Replying to [comment:3 dcsobral]:

The problem is possibly related to the fact that stream:stream, the first tag of all, is never closed.

I very much doubt that. This is the way XML streams are.

XML does not permit multiple root nodes.

Even if it was permitted.. XMLEventReader is capable of detecting other nodes it comes across that have opened and not closed, and without the ability to do it at the root level it makes this class useless for XML streams.

J (tumeow) said:
My previous comment was slightly ambiguous.. the XML nodes at the end it misses out are closed, but the parent node not being closed shouldn't affect detection of that.

huynhjl said:
Here is more condensed test case:

val src = new io.Source {
  val iter = "<stream><features><foo/></features>".iterator ++
    Iterator.continually{ println("wait 5s"); Thread.sleep(5000); '\n' }.take(1) ++
    "</stream>".iterator
}
val reader = new xml.pull.XMLEventReader(src)
while (reader.hasNext) System.out.println(reader.next)

One would hope that EvElemEnd(null,features) is printed before waiting 5 second. Instead we get:

EvElemStart(null,stream,,)
EvElemStart(null,features,,)
EvElemStart(null,foo,,)
EvElemEnd(null,foo)
wait 5s
EvElemEnd(null,features)
EvText(
)
EvElemEnd(null,stream)

Not sure if it is something inherent to the XML parser.

J (tumeow) said:
With my XMPP server I am unable to do this:

• Send request to server
• Wait for response (final stream in node)
• Send repsonse based on entire final XML tag in response.

It will block forever before I can determine the final tag in the stream is closed... at best I can't make my second request knowing I've got the full information to respond to.

@dcsobral said:
My current theory is that the following is happening. When reading the last closing tag sent, this method is called:

  def xEndTag(startName: String) {
    xToken('/')
    if (xName != startName)
      errorNoEnd(startName)

    xSpaceOpt
    xToken('>')
  }

Since that tag is properly sent, it runs up to xToken('>'), which is defined thus:

  def xToken(that: Char) {
    if (ch == that) nextch
    else xHandleError(that, "'%s' expected instead of '%s'".format(that, ch))
  }

Evidently, it calls nextch, which does the obvious thing:

  def nextch = {
    if (curInput.hasNext) {
      ch = curInput.next
      pos = curInput.pos
    } else {

The point here is that it needs to know if there is a next or not. Since this is a stream, it blocks until the stream is closed or something else is sent.

Unfortunately, the whole parser is based on having ch available (it is a var), so it doesn't look like there's an easy fix for it.

@dcsobral said:
Having said that (which was actually a while ago, but it wasn't accepted because there was an answer already), I think it can be fixed. I'm presently compiling a patch that ought to do the trick.

My idea is moving nextch's logic into ch and creating a flag that ch checks to see if it executes nextch's logic or return the last read character. It also adds a var to keep that last read character.

Then, nextch is modified to call ch (in case it is called twice in a row), and then setting the flag to true. Finally, it is modified to return Unit instead of Char, and all (of two) places where its return value is used are changed to call ch afterwards.

Unfortunately, nextch and "var ch" are both public, so this means a change in their API, however unlikely it is for anyone to be using them directly.

J (tumeow) said:
Replying to [comment:9 dcsobral]:

Unfortunately, nextch and "var ch" are both public, so this means a change in their API, however unlikely it is for anyone to be using them directly.

Does this mean it will take a while to hit the repository? I don't mind using the bleeding edge version of scala but it it won't even drop to the repository for a while perhaps I should abandon this class so I can continue my project.

@dcsobral said:
I have now submitted a possible fix on[https://github.com/scala/scala/pull/18 github].

It does fix the issue reported, and it doesn't seem to break anything else. However, Scala's test are a bit low on XML tests.

J (tumeow) said (edited on May 20, 2011 3:34:04 PM UTC):
Not sure if that patch works.. checking the 2.9.0 codebase I can see the freeze occurs here around line 543 in MarkupParser.scala:

    xToken('>')
    handle.elemStart(pos, pre, local, aMap, scope)
    val tmp = content(scope)
    xEndTag(qname) // freezes here and doesn't get to tmp
    tmp

Fixing xEndTag here alone would be enough to at least get the most usual cases fixed.

@dcsobral said:
But that's precisely what the patch does: stops xEndTag from freezing -- you are confusing the xToken('>') at MarkupParser with the one at the end of xEndTag.

J (tumeow) said:
The xToken('>') within xEndTag(qname) is where it freezes yes I know.. not confused.

@dcsobral said:
Well, the patch makes it not freeze there, so why do you think it doesn't work?

J (tumeow) said:
The patch was missing a file I see you've added it now.

J (tumeow) said:
Compiled scala with your patch applied (from the github scala clone master branch). Now attempting my test gives this:

:1:244: '/' expected instead of '' ^
:1:244: name expected, but char '' cannot start a name ^
Exception in thread "XMLEventReader" scala.xml.parsing.FatalError: expected closing tag of stream:stream
at scala.xml.parsing.MarkupParser$class.errorNoEnd(MarkupParser.scala:41)
at scala.xml.pull.XMLEventReader$Parser.errorNoEnd(XMLEventReader.scala:56)
at scala.xml.parsing.MarkupParserCommon$class.xEndTag(MarkupParserCommon.scala:93)
at scala.xml.pull.XMLEventReader$Parser.xEndTag(XMLEventReader.scala:56)
at scala.xml.parsing.MarkupParser$class.element1(MarkupParser.scala:555)
at scala.xml.pull.XMLEventReader$Parser.element1(XMLEventReader.scala:56)
at scala.xml.parsing.MarkupParser$class.content1(MarkupParser.scala:408)
at scala.xml.pull.XMLEventReader$Parser.content1(XMLEventReader.scala:56)
at scala.xml.parsing.MarkupParser$class.content(MarkupParser.scala:429)
at scala.xml.pull.XMLEventReader$Parser.content(XMLEventReader.scala:56)
at scala.xml.parsing.MarkupParser$class.document(MarkupParser.scala:234)
at scala.xml.pull.XMLEventReader$Parser.document(XMLEventReader.scala:56)
at scala.xml.pull.XMLEventReader$Parser$$anonfun$run$1.apply(XMLEventReader.scala:90)
at scala.xml.pull.XMLEventReader$Parser$$anonfun$run$1.apply(XMLEventReader.scala:90)
at scala.xml.pull.ProducerConsumerIterator$class.interruptibly(XMLEventReader.scala:113)
at scala.xml.pull.XMLEventReader.interruptibly(XMLEventReader.scala:26)
at scala.xml.pull.XMLEventReader$Parser.run(XMLEventReader.scala:90)
at java.lang.Thread.run(Thread.java:679)

It seems it's matching the end of the stream (but not eof) as '' or the empty string.

Alas of course it cannot find a closing tag to stream:stream as.. well it is a stream.

J (tumeow) said:
Sorry ignore last comment. Error was due to sequencing change now full nodes are retrieved. My code works great now using the github master branch of scala.

Commit Message Bot (anonymous) said:
(extempore in r24998) Fix #4599: XMLEventReader issue with input stream.

Makes MarkupParser.nextch lazy, only reaching out for the next char
when calling ch or eof. To make it possible, nextch now returns Unit
instead of Char. As it happens, that's how it is used almost everywhere.

Contributed by Daniel Sobral, no review.