C# - XML - Compression
本文关键字:Compression XML | 更新日期: 2023-09-27 18:27:35
我有一种情况,我正在生成一个要提交给Web服务的XML文件,有时是因为它的数据量超过了30mb或50mb。
我需要压缩文件,使用c#,.net framework 4.0,而不是一个拥有大部分数据的节点。。我不知道该怎么做。。如果有人能给我一个如何完成这项工作的例子,有可能吗。
xml文件看起来像这个
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<HeaderTalk xmlns="http://www.w3schools.com/xml">
<EnvelopeVersion>2.0</EnvelopeVersion>
<Header>
<MessageDetails>
<Class>CHAR-CLM</Class>
</MessageDetails>
<SenderDetails>
<IDAuthentication>
<SenderID>aaaaaa</SenderID>
<Authentication>
<Method>MD5</Method>
<Role>principal</Role>
<Value>a3MweCsv60kkAgzEpXeCqQ==</Value>
</Authentication>
</IDAuthentication>
<EmailAddress>Someone@somewhere.com</EmailAddress>
</SenderDetails>
</Header>
<TalkDetails>
<ChannelRouting>
<Channel>
<URI>1953</URI>
<Product>My product</Product>
<Version>2.0</Version>
</Channel>
</ChannelRouting>
</TalkDetails>
<Body>
<envelope xmlns="http://www.w3schools.com/xml/">
<PeriodEnd>2013-08-13</PeriodEnd>
<IRmark Type="generic">zZrxvJ7JmMNaOyrMs9ZOaRuihkg=</IRmark>
<Sender>Individual</Sender>
<Report>
<AuthOfficial>
<OffName>
<Fore>B</Fore>
<Sur>M</Sur>
</OffName>
<Phone>0123412345</Phone>
</AuthOfficial>
<DefaultCurrency>GBP</DefaultCurrency>
<Claim>
<OrgName>B</OrgName>
<ref>AB12345</ref>
<Repayment>
<Account>
<Donor>
<Fore>Barry</Fore>
</Donor>
<Total>7.00</Total>
</Account>
<Account>
<Donor>
<Fore>Anthony</Fore>
</Donor>
<Total>20.00</Total>
</Account>
</Repayment>
</Claim>
</Report>
</envelope>
</Body>
</HeaderTalk>
CLAIM节点是我想要压缩的,因为它可能是包含在XML中的数百万条记录。
我是一个编码新手,我花了很长时间才生成这个XML,并一直在寻找压缩节点的方法,但我就是无法让它工作。。Result需要完全相同,直到DefaultCurrency节点为止。。然后
</AuthOfficial>
<DefaultCurrency>GBP</DefaultCurrency>
<CompressedPart Type="zip">UEsDBBQAAAAIAFt690K1</CompressedPart>
</Report>
</envelope>
</Body>
</HeaderTalk>
或
</AuthOfficial>
<DefaultCurrency>GBP</DefaultCurrency>
<CompressedPart Type="gzip">UEsDBBQAAAAIAFt690K1</CompressedPart>
</Report>
</envelope>
</Body>
</HeaderTalk>
请提前感谢大家。或者,如果有人能建议我去哪里看看,并对我想做什么有一些想法。
要创建该文件,我只需遍历数据集并使用XmlElements编写节点,然后将innertext设置为我的值。。
我以前写的代码是。。//索赔
XmlElement GovtSenderClaim = xmldoc.CreateElement("Claim");
XmlElement GovtSenderOrgname = xmldoc.CreateElement("OrgName");
GovtSenderOrgname.InnerText = Charity_name;
GovtSenderClaim.AppendChild(GovtSenderOrgname);
XmlElement GovtSenderHMRCref = xmldoc.CreateElement("ref");
GovtSenderHMRCref.InnerText = strref ;
GovtSenderClaim.AppendChild(GovtSenderref);
XmlElement GovtSenderRepayments = xmldoc.CreateElement("Repayment");
while (reader.Read())
{
XmlElement GovtSenderAccount = xmldoc.CreateElement("Account");
XmlElement GovtSenderDonor = xmldoc.CreateElement("Donor");
XmlElement GovtSenderfore = xmldoc.CreateElement("Fore");
GovtSenderfore.InnerText = reader["EmployeeName_first_name"].ToString();
GovtSenderDonor.AppendChild(GovtSenderfore);
GovtSenderAccount .AppendChild(GovtSenderDonor);
XmlElement GovtSenderTotal = xmldoc.CreateElement("Total");
GovtSenderTotal.InnerText = reader["Total"].ToString();
GovtSenderAccount .AppendChild(GovtSenderTotal);
GovtSenderRepayments.AppendChild(GovtSenderAccount );
}
GovtSenderClaim.AppendChild(GovtSenderRepayments);
GovtSenderReport.AppendChild(GovtSenderClaim);
以及要关闭的其余节点。。
您可以尝试这样做:它将只压缩您选择的节点。这与您的要求有点不同,因为它将替换元素的内容,使元素及其属性保持原样。
{
// You are using a namespace!
XNamespace ns = "http://www.w3schools.com/xml/";
var xml2 = XDocument.Parse(xml);
// Compress
{
// Will compress all the XElement that are called Claim
// You should probably select the XElement in a better way
var nodes = from p in xml2.Descendants(ns + "Claim") select p;
foreach (XElement el in nodes)
{
CompressElementContent(el);
}
}
// Decompress
{
// Will decompress all the XElement that are called Claim
// You should probably select the XElement in a better way
var nodes = from p in xml2.Descendants(ns + "Claim") select p;
foreach (XElement el in nodes)
{
DecompressElementContent(el);
}
}
}
public static void CompressElementContent(XElement el)
{
string content;
using (var reader = el.CreateReader())
{
reader.MoveToContent();
content = reader.ReadInnerXml();
}
using (var ms = new MemoryStream())
{
using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Compress))
{
// So that the BOM isn't written we use build manually the encoder.
// See for example http://stackoverflow.com/a/2437780/613130
// But note that false is implicit in the parameterless constructor
using (StreamWriter sw = new StreamWriter(defl, new UTF8Encoding()))
{
sw.Write(content);
}
}
string base64 = Convert.ToBase64String(ms.ToArray());
el.ReplaceAll(new XText(base64));
}
}
public static void DecompressElementContent(XElement el)
{
var reader = el.CreateReader();
reader.MoveToContent();
var content = reader.ReadInnerXml();
var bytes = Convert.FromBase64String(content);
using (var ms = new MemoryStream(bytes))
{
using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Decompress))
{
using (StreamReader sr = new StreamReader(defl, Encoding.UTF8))
{
el.ReplaceAll(ParseXmlFragment(sr));
}
}
}
}
public static IEnumerable<XNode> ParseXmlFragment(StreamReader sr)
{
var settings = new XmlReaderSettings
{
ConformanceLevel = ConformanceLevel.Fragment
};
using (var xmlReader = XmlReader.Create(sr, settings))
{
xmlReader.MoveToContent();
while (xmlReader.ReadState != ReadState.EndOfFile)
{
yield return XNode.ReadFrom(xmlReader);
}
}
}
解压缩相当复杂,因为很难替换Xml的内容。最后,我将内容XNode
划分为ParseXmlFragment
中的Xnode
和DecompressElementContent
中的ReplaceAll
。
附带说明一下,您的XML中有两个相似但不同的名称空间:http://www.w3schools.com/xml
和http://www.w3schools.com/xml/
另一个变体将完全按照您的要求执行(因此它将创建一个CompressedPart节点),减去具有压缩类型的属性。
{
XNamespace ns = "http://www.w3schools.com/xml/";
var xml2 = XDocument.Parse(xml);
// Compress
{
// Here the ToList() is necessary, because we will replace the selected elements
var nodes = (from p in xml2.Descendants(ns + "Claim") select p).ToList();
foreach (XElement el in nodes)
{
CompressElementContent(el);
}
}
// Decompress
{
// Here the ToList() is necessary, because we will replace the selected elements
var nodes = (from p in xml2.Descendants("CompressedPart") select p).ToList();
foreach (XElement el in nodes)
{
DecompressElementContent(el);
}
}
}
public static void CompressElementContent(XElement el)
{
string content = el.ToString();
using (var ms = new MemoryStream())
{
using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Compress))
{
// So that the BOM isn't written we use build manually the encoder.
using (StreamWriter sw = new StreamWriter(defl, new UTF8Encoding()))
{
sw.Write(content);
}
}
string base64 = Convert.ToBase64String(ms.ToArray());
var newEl = new XElement("CompressedPart", new XText(base64));
el.ReplaceWith(newEl);
}
}
public static void DecompressElementContent(XElement el)
{
var reader = el.CreateReader();
reader.MoveToContent();
var content = reader.ReadInnerXml();
var bytes = Convert.FromBase64String(content);
using (var ms = new MemoryStream(bytes))
{
using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Decompress))
{
using (StreamReader sr = new StreamReader(defl, Encoding.UTF8))
{
var newEl = XElement.Parse(sr.ReadToEnd());
el.ReplaceWith(newEl);
}
}
}
}
我需要压缩文件,使用c#,.net框架4.0,而不是的一个节点
您可以使用GZip压缩。类似的东西
public static void Compress(FileInfo fileToCompress)
{
using (FileStream originalFileStream = fileToCompress.OpenRead())
{
if ((File.GetAttributes(fileToCompress.FullName) & FileAttributes.Hidden) != FileAttributes.Hidden & fileToCompress.Extension != ".gz")
{
using (FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".gz"))
{
using (GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress))
{
originalFileStream.CopyTo(compressionStream);
Console.WriteLine("Compressed {0} from {1} to {2} bytes.",
fileToCompress.Name, fileToCompress.Length.ToString(), compressedFileStream.Length.ToString());
}
}
}
}
}
public static void Decompress(FileInfo fileToDecompress)
{
using (FileStream originalFileStream = fileToDecompress.OpenRead())
{
string currentFileName = fileToDecompress.FullName;
string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);
using (FileStream decompressedFileStream = File.Create(newFileName))
{
using (GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress))
{
decompressionStream.CopyTo(decompressedFileStream);
Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
}
}
}
}
另一种可能的方法是通货紧缩。请参见此处。GZipStream和Deflate流之间的主要区别在于GZipStream将添加CRC以确保数据没有错误。
您所问的是可能的,但有点复杂。你需要在内存中创建压缩节点,然后编写它。我不知道你是如何编写XML的,所以我假设你有这样的东西:
open xml writer
write <MessageDetails>
write <SenderDetails>
write other nodes
write Claim node
write other stuff
close file
要编写索赔节点,您需要写入内存中的流,然后对其进行base64编码。生成的字符串就是您作为<CompressedPart>
写入文件的字符串。
string compressedData;
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gz = new GZipStream(CompressionMode.Compress, true))
{
using (XmlWriter writer = XmlWriter.Create(gz))
{
writer.WriteStartElement("Claim");
// write claim stuff here
writer.WriteEndElement();
}
}
// now base64 encode the memory stream buffer
byte[] buff = ms.GetBuffer();
compressedData = Convert.ToBase64String(buff, 0, buff.Length);
}
然后,您的数据在compressedData
字符串中,您可以将其作为元素数据写入。
正如我在评论中所说,GZip通常会使原始XML大小减少80%,因此50MB变成10MB。但是base64编码将使压缩后的大小增加33%。我预计结果大约是13.5 MB。
更新
根据您的附加代码,您尝试做的事情看起来并不太困难。我想你想做的是:
// do a bunch of stuff
GovtSenderClaim.AppendChild(GovtSenderRepayments);
// start of added code
// compress the GovtSenderClaim element
// This code writes the GovtSenderClaim element to a compressed MemoryStream.
// We then read the MemoryStream and create a base64 encoded representation.
string compressedData;
using (MemoryStream ms = new MemoryStream())
{
using (GZipStream gz = new GZipStream(CompressionMode.Compress, true))
{
using (StreamWriter writer = StreamWriter(gz))
{
GovtSenderClaim.Save(writer);
}
}
// now base64 encode the memory stream buffer
byte[] buff = ms.ToArray();
compressedData = Convert.ToBase64String(buff, 0, buff.Length);
}
// compressedData now contains the compressed Claim node, encoded in base64.
// create the CompressedPart element
XElement CompressedPart = xmldoc.CreateElement("CompressedPart");
CompressedPart.SetAttributeValue("Type", "gzip");
CompressedPart.SetValue(compressedData);
GovtSenderReport.AppendChild(CompressedPart);
// GovtSenderReport.AppendChild(GovtSenderClaim);
这就是我所做的工作。。
public void compressTheData(string xml)
{
XNamespace ns = "http://www.w3schools.com/xml/";
var xml2 = XDocument.Load(xml);
// Compress
{
var nodes = (from p in xml2.Descendants(ns + "Claim") select p).ToList();
foreach (XElement el in nodes)
{
CompressElementContent(el);
}
}
xml2.Save(xml);
}
public static void CompressElementContent(XElement el)
{
string content = el.ToString();
using (var ms = new MemoryStream())
{
using (GZipStream defl = new GZipStream(ms, CompressionMode.Compress))
{
using (StreamWriter sw = new StreamWriter(defl))
{
sw.Write(content);
}
}
string base64 = Convert.ToBase64String(ms.ToArray());
XElement newEl = new XElement("CompressedPart", new XText(base64));
XAttribute attrib = new XAttribute("Type", "gzip");
newEl.Add(attrib);
el.ReplaceWith(newEl);
}
}
感谢大家的投入。