HTTPClient Buffer超过2G;无法向缓冲区写入更多字节

本文关键字:缓冲区 多字节 超过 Buffer 2G HTTPClient | 更新日期: 2023-09-27 18:06:36

即将成为沃尔玛API传奇的最新消息。我正在提交一个调用,以获得使用HttpClient清除项目的列表。其他请求也可以正常工作,但是这个请求太大了,它破坏了HTTPRequest缓冲区。同样奇怪的是,这是一个REQUEST缓冲区错误而不是RESPONSE错误,因为请求只是URL。

异常信息:

 Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647. (System.Net.Http)
在System.Net.Http.HttpContent.LimitMemoryStream

。CheckSize(Int32 countToAdd)'r'n在System.Net.Http.HttpContent.LimitMemoryStream。写入(Byte[] buffer, Int32偏移量,Int32计数)'r'n在System.Net.Http.StreamToStreamCopy。'r'n at System.Net.Http.StreamToStreamCopy.StartRead()'r'n——从抛出异常的前一个位置开始的堆栈跟踪的结束——'r'n at System.Runtime.CompilerServices.TaskAwaiter。'r'n在System.Runtime.CompilerServices.TaskAwaiter。HandleNonSuccessAndDebuggerNotification(Task Task)'r'n at System.Runtime.CompilerServices.TaskAwaiter 1.GetResult()'r'n at Wal_Mart_Crawler.NET_IO.<walMart_Special_Feed_Lookup>d__4.MoveNext() in c:''users''user''documents''visual studio 2015''Projects''Wal-Mart_Inventory_Tracker''Wal-Mart_Crawler''NET_IO.cs:line 65'r'n--- End of stack trace from previous location where exception was thrown ---'r'n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)'r'n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)'r'n at System.Runtime.CompilerServices.TaskAwaiter . getresult ()'r'n at Wal_Mart_Crawler.Special_Feeds.d__2.MoveNext() in c:'users'user'documents'visual studio 2015'Projects'Wal-Mart_Inventory_Tracker'Wal-Mart_Crawler'Special_Feeds.cs:line 22

(对于好奇的人来说,第22行是):

 API_Json_Special_Feeds.RootObject Items = await net.walMart_Special_Feed_Lookup(specialFeedsURLs[i].Replace("{apiKey}", Wal_Mart_Crawler.Properties.Resources.API_Key_Walmart));
 //which uses HttpClient to call the API

一开始,当我看到"已配置"时,我的眼睛闪烁了一下,因为这意味着我可以更改它,对吗?毕竟我用的是x64——可能会破2G 不知道怎么回事

阅读SO,发现我应该禁用流媒体。试:

 var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);

没有好。

为WebRequest找到了AllowReadStreamBuffering设置-似乎无法为HttpClient找到一个

我不能控制我得到多少数据——那是沃尔玛的。对于如何处理响应流,我也受到了限制因为它会直接反序列化:

   var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
   return JsonConvert.DeserializeObject<API_Json_Special_Feeds.RootObject>(result);

所以即使我可以分解响应,我也会被卡住,因为我不认为我可以提供部分数据来反序列化。

问题:如何将缓冲区增加到2G以上或避免缓冲区大小超出异常?

我希望这是另一个简单的解决办法,我疲惫无知的大脑就是想不出来。一如既往,衷心感谢您的宝贵时间和您提供的任何帮助。

响应标头:

 StatusCode: 200, ReasonPhrase: 'OK', Version: 1.1, Content: System.Net.Http.StreamContent, Headers:
{
 X-Mashery-Responder: prod-j-worker-us-west-1c-63.mashery.com
 transfer-encoding: chunked
 Connection: keep-alive
 Date: Sat, 10 Sep 2016 00:57:38 GMT
 Server: Mashery
 Server: Proxy
 Content-MD5: BvJMDJiZPUvmAxxmwKGSog==
 Content-Type: application/json; charset=utf-8
 Last-Modified: Fri Sep 09 15:31:08 PDT 2016
}

HTTPClient Buffer超过2G;无法向缓冲区写入更多字节

这样的东西会起作用吗?

using (var http = new HttpClient())
using (var response = await http.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
using (StreamReader sr = new StreamReader(await response.Content.ReadAsStreamAsync()))
{
    var serializer = new JsonSerializer();               
    ITraceWriter tw = new MemoryTraceWriter();
    serializer.TraceWriter = tw; 
    var obj = (API_Json_Special_Feeds.RootObject)serializer.Deserialize(sr, typeof(API_Json_Special_Feeds.RootObject));
    // Stop and inspect the tracewriter object here to diagnose 
    return obj;                
}

编辑:通过包含tracewriter,您可以尝试确定为什么反序列化的行为不像您期望的那样。JSON反序列化倾向于无声地丢弃它不知道如何处理的信息,并导致一个空的或稀疏填充的对象。tracewriter将帮助诊断这些问题。示例