C# - XML - Compression

本文关键字:Compression XML | 更新日期: 2023-09-27 18:27:35

我有一种情况,我正在生成一个要提交给Web服务的XML文件,有时是因为它的数据量超过了30mb或50mb。

我需要压缩文件,使用c#,.net framework 4.0,而不是一个拥有大部分数据的节点。。我不知道该怎么做。。如果有人能给我一个如何完成这项工作的例子,有可能吗。

xml文件看起来像这个

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<HeaderTalk xmlns="http://www.w3schools.com/xml">
<EnvelopeVersion>2.0</EnvelopeVersion>
<Header>
<MessageDetails>
  <Class>CHAR-CLM</Class>      
</MessageDetails>
<SenderDetails>
  <IDAuthentication>
    <SenderID>aaaaaa</SenderID>
    <Authentication>
      <Method>MD5</Method>
      <Role>principal</Role>
      <Value>a3MweCsv60kkAgzEpXeCqQ==</Value>
    </Authentication>
  </IDAuthentication>
  <EmailAddress>Someone@somewhere.com</EmailAddress>
</SenderDetails>
</Header>
<TalkDetails>
  <ChannelRouting>
   <Channel>
     <URI>1953</URI>
     <Product>My product</Product>
     <Version>2.0</Version>
    </Channel>
</ChannelRouting>
</TalkDetails>
<Body>
   <envelope xmlns="http://www.w3schools.com/xml/">       
     <PeriodEnd>2013-08-13</PeriodEnd>
     <IRmark Type="generic">zZrxvJ7JmMNaOyrMs9ZOaRuihkg=</IRmark>
     <Sender>Individual</Sender>
     <Report>
       <AuthOfficial>
          <OffName>
            <Fore>B</Fore>
            <Sur>M</Sur>
          </OffName>
          <Phone>0123412345</Phone>
        </AuthOfficial>
    <DefaultCurrency>GBP</DefaultCurrency>
    <Claim>
      <OrgName>B</OrgName>
      <ref>AB12345</ref>
      <Repayment>
        <Account>
          <Donor>
            <Fore>Barry</Fore>
           </Donor>
            <Total>7.00</Total>              
        </Account>           
        <Account>
          <Donor>
            <Fore>Anthony</Fore>               
          </Donor>             
          <Total>20.00</Total>
        </Account>                  
      </Repayment>
      </Claim>
      </Report>
   </envelope>
 </Body>
</HeaderTalk>

CLAIM节点是我想要压缩的,因为它可能是包含在XML中的数百万条记录。

我是一个编码新手,我花了很长时间才生成这个XML,并一直在寻找压缩节点的方法,但我就是无法让它工作。。Result需要完全相同,直到DefaultCurrency节点为止。。然后

 </AuthOfficial>
 <DefaultCurrency>GBP</DefaultCurrency>
 <CompressedPart Type="zip">UEsDBBQAAAAIAFt690K1</CompressedPart>
 </Report>
 </envelope>
 </Body>
 </HeaderTalk>

 </AuthOfficial>
 <DefaultCurrency>GBP</DefaultCurrency>
 <CompressedPart Type="gzip">UEsDBBQAAAAIAFt690K1</CompressedPart>
 </Report>
 </envelope>
 </Body>
 </HeaderTalk>

请提前感谢大家。或者,如果有人能建议我去哪里看看,并对我想做什么有一些想法。

要创建该文件,我只需遍历数据集并使用XmlElements编写节点,然后将innertext设置为我的值。。

我以前写的代码是。。//索赔

XmlElement GovtSenderClaim = xmldoc.CreateElement("Claim");
XmlElement GovtSenderOrgname = xmldoc.CreateElement("OrgName");
GovtSenderOrgname.InnerText = Charity_name;
GovtSenderClaim.AppendChild(GovtSenderOrgname);
 XmlElement GovtSenderHMRCref = xmldoc.CreateElement("ref");
 GovtSenderHMRCref.InnerText = strref ;
 GovtSenderClaim.AppendChild(GovtSenderref);
 XmlElement GovtSenderRepayments = xmldoc.CreateElement("Repayment");
 while (reader.Read())
 {
  XmlElement GovtSenderAccount = xmldoc.CreateElement("Account");
  XmlElement GovtSenderDonor = xmldoc.CreateElement("Donor");
   XmlElement GovtSenderfore = xmldoc.CreateElement("Fore");
   GovtSenderfore.InnerText = reader["EmployeeName_first_name"].ToString();
   GovtSenderDonor.AppendChild(GovtSenderfore);
   GovtSenderAccount .AppendChild(GovtSenderDonor);
   XmlElement GovtSenderTotal = xmldoc.CreateElement("Total");
   GovtSenderTotal.InnerText = reader["Total"].ToString();
   GovtSenderAccount .AppendChild(GovtSenderTotal);
   GovtSenderRepayments.AppendChild(GovtSenderAccount );
 }
  GovtSenderClaim.AppendChild(GovtSenderRepayments);

   GovtSenderReport.AppendChild(GovtSenderClaim);

以及要关闭的其余节点。。

C# - XML - Compression

您可以尝试这样做:它将只压缩您选择的节点。这与您的要求有点不同,因为它将替换元素的内容,使元素及其属性保持原样。

{
    // You are using a namespace! 
    XNamespace ns = "http://www.w3schools.com/xml/";
    var xml2 = XDocument.Parse(xml);
    // Compress
    {
        // Will compress all the XElement that are called Claim
        // You should probably select the XElement in a better way
        var nodes = from p in xml2.Descendants(ns + "Claim") select p;
        foreach (XElement el in nodes)
        {
            CompressElementContent(el);
        }
    }
    // Decompress
    {
        // Will decompress all the XElement that are called Claim
        // You should probably select the XElement in a better way
        var nodes = from p in xml2.Descendants(ns + "Claim") select p;
        foreach (XElement el in nodes)
        {
            DecompressElementContent(el);
        }
    }
}
public static void CompressElementContent(XElement el)
{
    string content;
    using (var reader = el.CreateReader())
    {
        reader.MoveToContent();
        content = reader.ReadInnerXml();
    }
    using (var ms = new MemoryStream())
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Compress))
        {
            // So that the BOM isn't written we use build manually the encoder.
            // See for example http://stackoverflow.com/a/2437780/613130
            // But note that false is implicit in the parameterless constructor
            using (StreamWriter sw = new StreamWriter(defl, new UTF8Encoding()))
            {
                sw.Write(content);
            }
        }
        string base64 = Convert.ToBase64String(ms.ToArray());
        el.ReplaceAll(new XText(base64));
    }
}
public static void DecompressElementContent(XElement el)
{
    var reader = el.CreateReader();
    reader.MoveToContent();
    var content = reader.ReadInnerXml();
    var bytes = Convert.FromBase64String(content);
    using (var ms = new MemoryStream(bytes))
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Decompress))
        {
            using (StreamReader sr = new StreamReader(defl, Encoding.UTF8))
            {
                el.ReplaceAll(ParseXmlFragment(sr));
            }
        }
    }
}
public static IEnumerable<XNode> ParseXmlFragment(StreamReader sr)
{
    var settings = new XmlReaderSettings
    {
        ConformanceLevel = ConformanceLevel.Fragment
    };
    using (var xmlReader = XmlReader.Create(sr, settings))
    {
        xmlReader.MoveToContent();
        while (xmlReader.ReadState != ReadState.EndOfFile)
        {
            yield return XNode.ReadFrom(xmlReader);
        }
    }
}

解压缩相当复杂,因为很难替换Xml的内容。最后,我将内容XNode划分为ParseXmlFragment中的XnodeDecompressElementContent中的ReplaceAll

附带说明一下,您的XML中有两个相似但不同的名称空间:http://www.w3schools.com/xmlhttp://www.w3schools.com/xml/

另一个变体将完全按照您的要求执行(因此它将创建一个CompressedPart节点),减去具有压缩类型的属性。

{
    XNamespace ns = "http://www.w3schools.com/xml/";
    var xml2 = XDocument.Parse(xml);
    // Compress
    {
        // Here the ToList() is necessary, because we will replace the selected elements
        var nodes = (from p in xml2.Descendants(ns + "Claim") select p).ToList();
        foreach (XElement el in nodes)
        {
            CompressElementContent(el);
        }
    }
    // Decompress
    {
        // Here the ToList() is necessary, because we will replace the selected elements
        var nodes = (from p in xml2.Descendants("CompressedPart") select p).ToList();
        foreach (XElement el in nodes)
        {
            DecompressElementContent(el);
        }
    }
}
public static void CompressElementContent(XElement el)
{
    string content = el.ToString();
    using (var ms = new MemoryStream())
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Compress))
        {
            // So that the BOM isn't written we use build manually the encoder.
            using (StreamWriter sw = new StreamWriter(defl, new UTF8Encoding()))
            {
                sw.Write(content);
            }
        }
        string base64 = Convert.ToBase64String(ms.ToArray());
        var newEl = new XElement("CompressedPart", new XText(base64));
        el.ReplaceWith(newEl);
    }
}
public static void DecompressElementContent(XElement el)
{
    var reader = el.CreateReader();
    reader.MoveToContent();
    var content = reader.ReadInnerXml();
    var bytes = Convert.FromBase64String(content);
    using (var ms = new MemoryStream(bytes))
    {
        using (DeflateStream defl = new DeflateStream(ms, CompressionMode.Decompress))
        {
            using (StreamReader sr = new StreamReader(defl, Encoding.UTF8))
            {
                var newEl = XElement.Parse(sr.ReadToEnd());
                el.ReplaceWith(newEl);
            }
        }
    }
}

我需要压缩文件,使用c#,.net框架4.0,而不是的一个节点

您可以使用GZip压缩。类似的东西

public static void Compress(FileInfo fileToCompress)
        {
            using (FileStream originalFileStream = fileToCompress.OpenRead())
            {
                if ((File.GetAttributes(fileToCompress.FullName) & FileAttributes.Hidden) != FileAttributes.Hidden & fileToCompress.Extension != ".gz")
                {
                    using (FileStream compressedFileStream = File.Create(fileToCompress.FullName + ".gz"))
                    {
                        using (GZipStream compressionStream = new GZipStream(compressedFileStream, CompressionMode.Compress))
                        {
                            originalFileStream.CopyTo(compressionStream);
                            Console.WriteLine("Compressed {0} from {1} to {2} bytes.",
                                fileToCompress.Name, fileToCompress.Length.ToString(), compressedFileStream.Length.ToString());
                        }
                    }
                }
            }
        }
        public static void Decompress(FileInfo fileToDecompress)
        {
            using (FileStream originalFileStream = fileToDecompress.OpenRead())
            {
                string currentFileName = fileToDecompress.FullName;
                string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);
                using (FileStream decompressedFileStream = File.Create(newFileName))
                {
                    using (GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress))
                    {
                        decompressionStream.CopyTo(decompressedFileStream);
                        Console.WriteLine("Decompressed: {0}", fileToDecompress.Name);
                    }
                }
            }
        }

另一种可能的方法是通货紧缩。请参见此处。GZipStream和Deflate流之间的主要区别在于GZipStream将添加CRC以确保数据没有错误。

您所问的是可能的,但有点复杂。你需要在内存中创建压缩节点,然后编写它。我不知道你是如何编写XML的,所以我假设你有这样的东西:

open xml writer
write <MessageDetails>
write <SenderDetails>
write other nodes
write Claim node
write other stuff
close file

要编写索赔节点,您需要写入内存中的流,然后对其进行base64编码。生成的字符串就是您作为<CompressedPart>写入文件的字符串。

string compressedData;
using (MemoryStream ms = new MemoryStream())
{
    using (GZipStream gz = new GZipStream(CompressionMode.Compress, true))
    {
        using (XmlWriter writer = XmlWriter.Create(gz))
        {
            writer.WriteStartElement("Claim");
            // write claim stuff here
            writer.WriteEndElement();
        }
    }
    // now base64 encode the memory stream buffer
    byte[] buff = ms.GetBuffer();
    compressedData = Convert.ToBase64String(buff, 0, buff.Length);
}

然后,您的数据在compressedData字符串中,您可以将其作为元素数据写入。

正如我在评论中所说,GZip通常会使原始XML大小减少80%,因此50MB变成10MB。但是base64编码将使压缩后的大小增加33%。我预计结果大约是13.5 MB。

更新

根据您的附加代码,您尝试做的事情看起来并不太困难。我想你想做的是:

// do a bunch of stuff
GovtSenderClaim.AppendChild(GovtSenderRepayments);
// start of added code
// compress the GovtSenderClaim element
// This code writes the GovtSenderClaim element to a compressed MemoryStream.
// We then read the MemoryStream and create a base64 encoded representation.
string compressedData;
using (MemoryStream ms = new MemoryStream())
{
    using (GZipStream gz = new GZipStream(CompressionMode.Compress, true))
    {
        using (StreamWriter writer = StreamWriter(gz))
        {
            GovtSenderClaim.Save(writer);
        }
    }
    // now base64 encode the memory stream buffer
    byte[] buff = ms.ToArray();
    compressedData = Convert.ToBase64String(buff, 0, buff.Length);
}
// compressedData now contains the compressed Claim node, encoded in base64.
// create the CompressedPart element
XElement CompressedPart = xmldoc.CreateElement("CompressedPart");
CompressedPart.SetAttributeValue("Type", "gzip");
CompressedPart.SetValue(compressedData);
GovtSenderReport.AppendChild(CompressedPart);
// GovtSenderReport.AppendChild(GovtSenderClaim);

这就是我所做的工作。。

public void compressTheData(string xml)
{
  XNamespace ns =  "http://www.w3schools.com/xml/";
  var xml2 = XDocument.Load(xml);   
  // Compress
  {
   var nodes = (from p in xml2.Descendants(ns + "Claim") select p).ToList();
    foreach (XElement el in nodes)
    {      
        CompressElementContent(el);           
    }
}
xml2.Save(xml);   
}

public static void CompressElementContent(XElement el)
{
  string content = el.ToString();    
  using (var ms = new MemoryStream())
  {
    using (GZipStream defl = new GZipStream(ms, CompressionMode.Compress))
    {           
        using (StreamWriter sw = new StreamWriter(defl))
        {
            sw.Write(content); 
        }
    }
    string base64 = Convert.ToBase64String(ms.ToArray());  
    XElement newEl = new XElement("CompressedPart", new XText(base64));
    XAttribute attrib = new XAttribute("Type", "gzip");
    newEl.Add(attrib);
    el.ReplaceWith(newEl);
  }
 }

感谢大家的投入。