Skip to Content.
Sympa Menu

xom-interest - Re: [XOM-interest] open HTML entities

xom-interest AT lists.ibiblio.org

Subject: XOM API for Processing XML with Java

List archive

Chronological Thread  
  • From: Diego Campo <diego.campo AT properazzi.com>
  • To: Michael Kay <mike AT saxonica.com>
  • Cc: elharo AT metalab.unc.edu, xom-interest AT lists.ibiblio.org
  • Subject: Re: [XOM-interest] open HTML entities
  • Date: Wed, 16 Jan 2008 17:21:53 +0100

Ah, yes, sorry about that!

Yes, I did a hack by passing the input through some regex which replaces
"&lang" followed by a punctuation character which is not ';' to
"&amp;lang". This seems to fix it, though it's a bit risky. I am trying
to find a better way to cope with this.

Thanks a lot Michael.


El mié, 16-01-2008 a las 15:36 +0000, Michael Kay escribió:
> Well, what we've learnt from this is that you are using the TagSoup HTML
> parser (why didn't you tell us that before?). This is where you should now
> focus your investigations, not on XOM.
>
> The input HTML is invalid ("&" in HTML should be escaped as "&amp;", just
> as in XML). TagSoup usually does a very good job of coping with invalid
> HTML, but you can't expect it to succeed every time. After all, the only
> way it can tell whether "&lang" was supposed to be "&amp;lang" or "&lang;"
> is by guessing, and sometimes it will guess wrong.
>
> Michael Kay
> http://www.saxonica.com/
>
>
> > -----Original Message-----
> > From: Diego Campo [mailto:diego.campo AT properazzi.com]
> > Sent: 16 January 2008 15:20
> > To: Michael Kay
> > Cc: elharo AT metalab.unc.edu; xom-interest AT lists.ibiblio.org
> > Subject: RE: [XOM-interest] open HTML entities
> >
> > Sure, sorry about that.
> >
> > I attach the xml code.
> > The important areas are like this:
> >
> > <td><div align="right"><a
> > href="http://www.sucasita.es/search.cgi?wherefrom=web&lang=D&j
> > ob=detail&ref=PAN-100-A03">Details</a></div></td>
> >
> > This attribute href comes as
> > http://www.sucasita.es/search.cgi?wherefrom=web
> > 〈=D&job=detail&ref=PAN-100-A03
> >
> > because the &lang is understood as its HTML Entity
> > conversion, '〈'. The &lang is not followed by a ';', so I
> > assumed it wouldn't be converted as its value. I'd like to
> > know if this is a bug or an expected behaviour.
> >
> > There are a few of them on the code as you can see. This is a
> > site of real estate properties, and this is the link to the
> > page with details of the listing.
> >
> > The code provided code is the printout of the String I have.
> > This String come from a process of conversion from the page's
> > charset to UTF8. This is already done so don't worry about that.
> >
> > Then I feed a StringReader with it.
> >
> > Then I get the nu.xom.Builder with the method getBuilder() below.
> >
> > Should you need any more information about the
> > TaggingNodeFactory.java, or anything else, please let me know.
> >
> > Thanks a lot for your time.
> >
> >
> > ================================================================
> >
> > public final static String[] EMPTY_HTML_ELEMENTS = {
> > "marquee","option", "script",
> > "select", "style", "title" };
> >
> > protected Builder getBuilder(){
> > try{
> > XMLReader xmlreader;
> > long start = System.currentTimeMillis();
> > xmlreader =
> > XMLReaderFactory.createXMLReader("org.ccil.cowan.tagsoup.Parser");
> >
> > xmlreader.setFeature(org.ccil.cowan.tagsoup.Parser.bogonsEmptyFeature,
> > false);
> > HTMLSchema theSchema = new HTMLSchema(){
> > public String getURI() {
> > return "";
> > }
> > public String getPrefx() {
> > return "";
> > }
> > };
> > for(int i=0;i <
> > TaggingNodeFactory.EMPTY_HTML_ELEMENTS.length;i++){
> > /*
> > * there's no way to tell
> > tagsoup to ignore certain elements and their content.
> > * the best way to do this is
> > to pass them as cdata, and ignore their content in the
> > * NodeFactory
> > */
> > ElementType et=
> > theSchema.getElementType(TaggingNodeFactory.EMPTY_HTML_ELEMENTS[i]);
> > // ignorable elements are
> > always type cdata
> > if(et == null){
> > //add schema definition
> > if it doesn't exist
> >
> > theSchema.elementType(TaggingNodeFactory.EMPTY_HTML_ELEMENTS[i],
> > Schema.M_EMPTY, HTMLSchema.M_PCDATA|HTMLSchema.M_INLINE,
> > Schema.F_CDATA);
> > //elementType("b",
> > M_PCDATA|M_INLINE, M_INLINE|M_NOLINK, F_RESTART);
> > }
> > else{
> > et.setFlags(Schema.F_CDATA);
> >
> > /* some html elements
> > like <br> <p> don't require closing tags and have no content
> > *
> > * html elements that
> > require closing tags should be allowed to contain anything.
> > *
> > */
> > if(et.model() != Schema.M_EMPTY)
> >
> > et.setModel(Schema.M_ANY);
> > }
> > }
> >
> > xmlreader.setProperty(org.ccil.cowan.tagsoup.Parser.schemaProperty,
> > theSchema);
> >
> > long before = System.currentTimeMillis();
> > Builder builder = new
> > Builder(xmlreader, false, new TaggingNodeFactory(this,true));
> > return builder;
> > }
> > catch(Exception e){
> >
> > _logger.error(EncodingUtils.exceptionToString(e));
> > return null;
> > }
> >
> >
> > }
> >
> >
> > El mié, 16-01-2008 a las 12:03 +0000, Michael Kay escribió:
> > > > > The thing is, I don't expect it to be taken as an entity,
> > > > because it's
> > > > > not. Since xom has correctness as its first goal, this
> > surprises me.
> > > > >
> > > > > I hope this is clear enough.
> > > > >
> > > >
> > > > I'm sorry but it's not. You need to provide a complete test case
> > > > that shows the input XML document (not just the URL), the
> > code that
> > > > reads it, and the output.
> > > >
> > >
> > > It looks to me as if you are trying to parse the input
> > twice. No idea
> > > how, but if you want help from the list in debugging your code, you
> > > will have to show us your code, and that is what I would be
> > looking for.
> > >
> > > Michael Kay
> > > http://www.saxonica.com/
> > >
> >
>





Archive powered by MHonArc 2.6.24.

Top of Page