XOM API for Processing XML with Java

[Diego Campo <diego.campo AT properazzi.com>]

Ah, yes, sorry about that!

Yes, I did a hack by passing the input through some regex which replaces
"&lang" followed by a punctuation character which is not ';' to
"&lang". This seems to fix it, though it's a bit risky. I am trying
to find a better way to cope with this.

Thanks a lot Michael.


El mié, 16-01-2008 a las 15:36 +0000, Michael Kay escribió:
> Well, what we've learnt from this is that you are using the TagSoup HTML 
> parser (why didn't you tell us that before?). This is where you should now 
> focus your investigations, not on XOM.
> 
> The input HTML is invalid ("&" in HTML should be escaped as "&", just 
> as in XML). TagSoup usually does a very good job of coping with invalid 
> HTML, but you can't expect it to succeed every time. After all, the only 
> way it can tell whether "&lang" was supposed to be "&lang" or "⟨" 
> is by guessing, and sometimes it will guess wrong.
> 
> Michael Kay
> http://www.saxonica.com/
>  
> 
> > -----Original Message-----
> > From: Diego Campo [mailto:diego.campo AT properazzi.com] 
> > Sent: 16 January 2008 15:20
> > To: Michael Kay
> > Cc: elharo AT metalab.unc.edu; xom-interest AT lists.ibiblio.org
> > Subject: RE: [XOM-interest] open HTML entities
> > 
> > Sure, sorry about that.
> > 
> > I attach the xml code.
> > The important areas are like this:
> > 
> > <td><div align="right"><a
> > href="http://www.sucasita.es/search.cgi?wherefrom=web&lang=D&j
> > ob=detail&ref=PAN-100-A03">Details</a></div></td>
> > 
> > This attribute href comes as
> > http://www.sucasita.es/search.cgi?wherefrom=web
> > 〈=D&job=detail&ref=PAN-100-A03
> > 
> > because the &lang is understood as its HTML Entity 
> > conversion, '〈'. The &lang is not followed by a ';', so I 
> > assumed it wouldn't be converted as its value. I'd like to 
> > know if this is a bug or an expected behaviour.
> > 
> > There are a few of them on the code as you can see. This is a 
> > site of real estate properties, and this is the link to the 
> > page with details of the listing.
> > 
> > The code provided code is the printout of the String I have. 
> > This String come from a process of conversion from the page's 
> > charset to UTF8. This is already done so don't worry about that.
> > 
> > Then I feed a StringReader with it.
> > 
> > Then I get the nu.xom.Builder with the method getBuilder() below.
> > 
> > Should you need any more information about the 
> > TaggingNodeFactory.java, or anything else, please let me know. 
> > 
> > Thanks a lot for your time.
> > 
> > 
> > ================================================================
> > 
> > public final static String[] EMPTY_HTML_ELEMENTS = { 
> > "marquee","option", "script",
> >             "select", "style", "title" };
> > 
> > protected Builder getBuilder(){
> >             try{
> >                     XMLReader xmlreader;
> >                     long start = System.currentTimeMillis();
> >                     xmlreader =
> > XMLReaderFactory.createXMLReader("org.ccil.cowan.tagsoup.Parser");
> > 
> > xmlreader.setFeature(org.ccil.cowan.tagsoup.Parser.bogonsEmptyFeature,
> > false);
> >                     HTMLSchema theSchema = new HTMLSchema(){
> >                             public String getURI() {
> >                                     return "";
> >                             }
> >                             public String getPrefx() {
> >                                     return "";
> >                             }
> >                     };
> >                     for(int i=0;i < 
> > TaggingNodeFactory.EMPTY_HTML_ELEMENTS.length;i++){
> >                             /*
> >                              * there's no way to tell 
> > tagsoup to ignore certain elements and their content.
> >                              * the best way to do this is 
> > to pass them as cdata, and ignore their content in the
> >                              * NodeFactory
> >                              */
> >                             ElementType et=
> > theSchema.getElementType(TaggingNodeFactory.EMPTY_HTML_ELEMENTS[i]);
> >                             // ignorable elements are 
> > always type cdata
> >                             if(et == null){
> >                                     //add schema definition 
> > if it doesn't exist
> >                                     
> > theSchema.elementType(TaggingNodeFactory.EMPTY_HTML_ELEMENTS[i],
> > Schema.M_EMPTY, HTMLSchema.M_PCDATA|HTMLSchema.M_INLINE,
> > Schema.F_CDATA);
> >                                     //elementType("b", 
> > M_PCDATA|M_INLINE, M_INLINE|M_NOLINK, F_RESTART);
> >                             }
> >                             else{
> >                                     et.setFlags(Schema.F_CDATA);
> > 
> >                                     /* some html elements 
> > like <br> <p> don't require closing tags and have no content
> >                                      * 
> >                                      * html elements that 
> > require closing tags should be allowed to contain anything.
> >                                      * 
> >                                      */
> >                                     if(et.model() != Schema.M_EMPTY)
> >                                             
> > et.setModel(Schema.M_ANY);
> >                             }
> >                     }
> >                     
> > xmlreader.setProperty(org.ccil.cowan.tagsoup.Parser.schemaProperty,
> > theSchema);
> > 
> >                     long before = System.currentTimeMillis();
> >                     Builder builder = new 
> > Builder(xmlreader, false, new TaggingNodeFactory(this,true));
> >                     return builder;
> >             }
> >             catch(Exception e){
> >                     
> > _logger.error(EncodingUtils.exceptionToString(e));
> >                     return null;
> >             }
> > 
> > 
> >     }
> > 
> > 
> > El mié, 16-01-2008 a las 12:03 +0000, Michael Kay escribió:
> > > > > The thing is, I don't expect it to be taken as an entity,
> > > > because it's
> > > > > not. Since xom has correctness as its first goal, this 
> > surprises me.
> > > > > 
> > > > > I hope this is clear enough.
> > > > > 
> > > > 
> > > > I'm sorry but it's not. You need to provide a complete test case 
> > > > that shows the input XML document (not just the URL), the 
> > code that 
> > > > reads it, and the output.
> > > > 
> > > 
> > > It looks to me as if you are trying to parse the input 
> > twice. No idea 
> > > how, but if you want help from the list in debugging your code, you 
> > > will have to show us your code, and that is what I would be 
> > looking for.
> > > 
> > > Michael Kay
> > > http://www.saxonica.com/
> > > 
> > 
>