用HtmlAgilityPack加载一个日语网页

本文关键字：一个 日语 网页 HtmlAgilityPack 加载 | 更新日期: 2023-09-27 17:52:36

我有这段代码来加载和解析使用HtmlAgilityPack的网页。它适用于大多数网页，但当我试图加载一个日语网页时，似乎编码是错误的。我该怎么做呢?实际上我如何设置基于网页编码的编码?

class Program {
    private const string HttpMethod = "GET";
    private const string UserAgent =
        "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7";
    static void Main(string[] args) {
        var request = WebRequest.Create("http://infoseek.co.jp/") as HttpWebRequest;
        if (request == null)
            return;
        request.Method = HttpMethod;
        request.UserAgent = UserAgent;
        var response = request.GetResponse() as HttpWebResponse;
        if (response == null)
            return;
        var stream = response.GetResponseStream();
        var document = new HtmlDocument {
            OptionCheckSyntax = true,
            OptionFixNestedTags = true,
            OptionAutoCloseOnEnd = true,
            OptionDefaultStreamEncoding = Encoding.UTF8,
            OptionReadEncoding = true
        };
        document.Load(stream, Encoding.UTF8);
        var d = document.DocumentNode;
    }
}

用HtmlAgilityPack加载一个日语网页

infoseek.co.jp响应HTTP头信息

Content-Type    text/html; charset=EUC-JP

在HTML标签

中镜像
<meta http-equiv="Content-Type" content="text/html; charset=EUC-JP">

在。net中，使用Code Page 51932解码EUC-JP

我试图通过下面的代码从HttpWebResponse对象获得编码。你觉得有什么问题吗?或者你有其他的想法吗?

class Program {
    private const string HttpMethod = "GET";
    private const string UserAgent =
        "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.7 (KHTML, like Gecko) Chrome/7.0.517.41 Safari/534.7";
    static void Main(string[] args) {
        var request = WebRequest.Create("http://infoseek.co.jp/") as HttpWebRequest;
        if (request == null)
            return;
        request.Method = HttpMethod;
        request.UserAgent = UserAgent;
        var response = request.GetResponse() as HttpWebResponse;
        if (response == null)
            return;
        var encoding = TryGetEncoding(response);
        var stream = response.GetResponseStream();
        var document = new HtmlDocument {
            OptionCheckSyntax = true,
            OptionFixNestedTags = true,
            OptionAutoCloseOnEnd = true,
            OptionReadEncoding = true,
            OptionDefaultStreamEncoding = encoding
        };
        document.Load(stream, encoding);
        var d = document.DocumentNode;
    }
    private static Encoding TryGetEncoding(HttpWebResponse response) {
        var charset = response.CharacterSet;
        if (string.IsNullOrWhiteSpace(charset))
            charset = response.ContentEncoding;
        if (string.IsNullOrWhiteSpace(charset))
            return Encoding.UTF8;
        try {
            return Encoding.GetEncoding(charset);
        } catch {
            return Encoding.UTF8;
        }
    }
}
相关文章：