使用htmlcleaner 和 xpath 解析html/xml

如果使用 xml parsers 则， 需要xml 是非常严格格式要求。 我们想用xpath去解析 html时， 由于html往往不是严格的xml 格式， 根本无法正确解析。 所以我们需要将 html 转换成 严格的 xml 格式。

不过有了 htmlcleaner， 我们可以直接使用并利用xpath进行解析了。 同样针对于写的不言个的xml来说， 也可以用htmlcleaner。

这里推荐一个在线进行校验 xpath，xquery， css3 的工具。
http://videlibri.sourceforge.net/cgi-bin/xidelcgi

public class OptionScraper {
 
    // EXAMPLE XPATH QUERIES IN THE FORM OF STRINGS - WILL BE USED LATER
    private static final String NAME_XPATH = "//div[@class='yfi_quote']/div[@class='hd']/h2";
 
    private static final String TIME_XPATH = "//table[@id='time_table']/tbody/tr/td[@class='yfnc_tabledata1']";
 
    private static final String PRICE_XPATH = "//table[@id='price_table']//tr//span";
 
    // TAGNODE OBJECT, ITS USE WILL COME IN LATER
    private static TagNode node;
 
    // A METHOD THAT HELPS ME RETRIEVE THE STOCK OPTION'S DATA BASED OFF THE NAME (I.E. GOUAA IS ONE OF GOOGLE'S STOCK OPTIONS)
    public static Option getOptionFromName(String name) throws XPatherException, ParserConfigurationException,SAXException, IOException, XPatherException {
 
        // THE URL WHOSE HTML I WANT TO RETRIEVE AND PARSE
        String option_url = "http://finance.yahoo.com/q?s=" + name.toUpperCase();
 
        // THIS IS WHERE THE HTMLCLEANER COMES IN, I INITIALIZE IT HERE
        HtmlCleaner cleaner = new HtmlCleaner();
        CleanerProperties props = cleaner.getProperties();
        props.setAllowHtmlInsideAttributes(true);
        props.setAllowMultiWordAttributes(true);
        props.setRecognizeUnicodeChars(true);
        props.setOmitComments(true);
 
        // OPEN A CONNECTION TO THE DESIRED URL
        URL url = new URL(option_url);
        URLConnection conn = url.openConnection();
 
        //USE THE CLEANER TO "CLEAN" THE HTML AND RETURN IT AS A TAGNODE OBJECT
        node = cleaner.clean(new InputStreamReader(conn.getInputStream()));
 
        // ONCE THE HTML IS CLEANED, THEN YOU CAN RUN YOUR XPATH EXPRESSIONS ON THE NODE, WHICH WILL THEN RETURN AN ARRAY OF TAGNODE OBJECTS (THESE ARE RETURNED AS OBJECTS BUT GET CASTED BELOW)
        Object[] info_nodes = node.evaluateXPath(NAME_XPATH);
        Object[] time_nodes = node.evaluateXPath(TIME_XPATH);
        Object[] price_nodes = node.evaluateXPath(PRICE_XPATH);
 
        // HERE I JUST DO A SIMPLE CHECK TO MAKE SURE THAT MY XPATH WAS CORRECT AND THAT AN ACTUAL NODE(S) WAS RETURNED
        if (info_nodes.length > 0) {
            // CASTED TO A TAGNODE
            TagNode info_node = (TagNode) info_nodes[0];
            // HOW TO RETRIEVE THE CONTENTS AS A STRING
            String info = info_node.getChildren().iterator().next().toString().trim();
 
            // SOME METHOD THAT PROCESSES THE STRING OF INFORMATION (IN MY CASE, THIS WAS THE STOCK QUOTE, ETC)
            processInfoNode(o, info);
        }
 
        if (time_nodes.length > 0) {
            TagNode time_node = (TagNode) time_nodes[0];
            String date = time_node.getChildren().iterator().next().toString().trim();
 
            // DATE RETURNED IN 15-JAN-10 FORMAT, SO THIS IS SOME METHOD I WROTE TO JUST PARSE THAT STRING INTO THE FORMAT THAT I USE
            processDateNode(o, date);
        }
 
        if (price_nodes.length > 0) {
            TagNode price_node = (TagNode) price_nodes[0];
            double price = Double.parseDouble(price_node.getChildren().iterator().next().toString().trim());
            o.setPremium(price);
        }
 
        return o;
    }
}

参考资料：

此篇文章已被阅读3110 次