XOM API for Processing XML with Java

["Michael Kay" <mike AT saxonica.com>]

Well, what we've learnt from this is that you are using the TagSoup HTML 
parser (why didn't you tell us that before?). This is where you should now 
focus your investigations, not on XOM.

The input HTML is invalid ("&" in HTML should be escaped as "&", just as 
in XML). TagSoup usually does a very good job of coping with invalid HTML, 
but you can't expect it to succeed every time. After all, the only way it can 
tell whether "&lang" was supposed to be "&lang" or "⟨" is by 
guessing, and sometimes it will guess wrong.

Michael Kay
http://www.saxonica.com/
 

> -----Original Message-----
> From: Diego Campo [mailto:diego.campo AT properazzi.com] 
> Sent: 16 January 2008 15:20
> To: Michael Kay
> Cc: elharo AT metalab.unc.edu; xom-interest AT lists.ibiblio.org
> Subject: RE: [XOM-interest] open HTML entities
> 
> Sure, sorry about that.
> 
> I attach the xml code.
> The important areas are like this:
> 
> <td><div align="right"><a
> href="http://www.sucasita.es/search.cgi?wherefrom=web&lang=D&j
> ob=detail&ref=PAN-100-A03">Details</a></div></td>
> 
> This attribute href comes as
> http://www.sucasita.es/search.cgi?wherefrom=web
> 〈=D&job=detail&ref=PAN-100-A03
> 
> because the &lang is understood as its HTML Entity 
> conversion, '〈'. The &lang is not followed by a ';', so I 
> assumed it wouldn't be converted as its value. I'd like to 
> know if this is a bug or an expected behaviour.
> 
> There are a few of them on the code as you can see. This is a 
> site of real estate properties, and this is the link to the 
> page with details of the listing.
> 
> The code provided code is the printout of the String I have. 
> This String come from a process of conversion from the page's 
> charset to UTF8. This is already done so don't worry about that.
> 
> Then I feed a StringReader with it.
> 
> Then I get the nu.xom.Builder with the method getBuilder() below.
> 
> Should you need any more information about the 
> TaggingNodeFactory.java, or anything else, please let me know. 
> 
> Thanks a lot for your time.
> 
> 
> ================================================================
> 
> public final static String[] EMPTY_HTML_ELEMENTS = { 
> "marquee","option", "script",
>               "select", "style", "title" };
> 
> protected Builder getBuilder(){
>               try{
>                       XMLReader xmlreader;
>                       long start = System.currentTimeMillis();
>                       xmlreader =
> XMLReaderFactory.createXMLReader("org.ccil.cowan.tagsoup.Parser");
> 
> xmlreader.setFeature(org.ccil.cowan.tagsoup.Parser.bogonsEmptyFeature,
> false);
>                       HTMLSchema theSchema = new HTMLSchema(){
>                               public String getURI() {
>                                       return "";
>                               }
>                               public String getPrefx() {
>                                       return "";
>                               }
>                       };
>                       for(int i=0;i < 
> TaggingNodeFactory.EMPTY_HTML_ELEMENTS.length;i++){
>                               /*
>                                * there's no way to tell 
> tagsoup to ignore certain elements and their content.
>                                * the best way to do this is 
> to pass them as cdata, and ignore their content in the
>                                * NodeFactory
>                                */
>                               ElementType et=
> theSchema.getElementType(TaggingNodeFactory.EMPTY_HTML_ELEMENTS[i]);
>                               // ignorable elements are 
> always type cdata
>                               if(et == null){
>                                       //add schema definition 
> if it doesn't exist
>                                       
> theSchema.elementType(TaggingNodeFactory.EMPTY_HTML_ELEMENTS[i],
> Schema.M_EMPTY, HTMLSchema.M_PCDATA|HTMLSchema.M_INLINE,
> Schema.F_CDATA);
>                                       //elementType("b", 
> M_PCDATA|M_INLINE, M_INLINE|M_NOLINK, F_RESTART);
>                               }
>                               else{
>                                       et.setFlags(Schema.F_CDATA);
> 
>                                       /* some html elements 
> like <br> <p> don't require closing tags and have no content
>                                        * 
>                                        * html elements that 
> require closing tags should be allowed to contain anything.
>                                        * 
>                                        */
>                                       if(et.model() != Schema.M_EMPTY)
>                                               
> et.setModel(Schema.M_ANY);
>                               }
>                       }
>                       
> xmlreader.setProperty(org.ccil.cowan.tagsoup.Parser.schemaProperty,
> theSchema);
> 
>                       long before = System.currentTimeMillis();
>                       Builder builder = new 
> Builder(xmlreader, false, new TaggingNodeFactory(this,true));
>                       return builder;
>               }
>               catch(Exception e){
>                       
> _logger.error(EncodingUtils.exceptionToString(e));
>                       return null;
>               }
> 
> 
>       }
> 
> 
> El mié, 16-01-2008 a las 12:03 +0000, Michael Kay escribió:
> > > > The thing is, I don't expect it to be taken as an entity,
> > > because it's
> > > > not. Since xom has correctness as its first goal, this 
> surprises me.
> > > > 
> > > > I hope this is clear enough.
> > > > 
> > > 
> > > I'm sorry but it's not. You need to provide a complete test case 
> > > that shows the input XML document (not just the URL), the 
> code that 
> > > reads it, and the output.
> > > 
> > 
> > It looks to me as if you are trying to parse the input 
> twice. No idea 
> > how, but if you want help from the list in debugging your code, you 
> > will have to show us your code, and that is what I would be 
> looking for.
> > 
> > Michael Kay
> > http://www.saxonica.com/
> > 
>