TPL数据流IO读写操作实现中的内存问题
本文关键字:内存 问题 实现 操作 数据流 IO 读写 TPL | 更新日期: 2023-09-27 18:27:52
我尝试使用File IO操作来实现读写操作,并将这些操作封装到TransformBlock
中,以使这些操作线程安全,而不是使用锁定机制。
但问题是,当我尝试并行写入甚至5个文件时,会出现内存溢出异常,并且在使用此实现时会阻塞UI线程。实现是在Windows Phone项目中完成的。请指出这句话的错误之处。
文件IO操作
public static readonly IsolatedStorageFile _isolatedStore = IsolatedStorageFile.GetUserStoreForApplication();
public static readonly FileIO _file = new FileIO();
public static readonly ConcurrentExclusiveSchedulerPair taskSchedulerPair = new ConcurrentExclusiveSchedulerPair();
public static readonly ExecutionDataflowBlockOptions exclusiveExecutionDataFlow
= new ExecutionDataflowBlockOptions
{
TaskScheduler = taskSchedulerPair.ExclusiveScheduler,
BoundedCapacity = 1
};
public static readonly ExecutionDataflowBlockOptions concurrentExecutionDataFlow
= new ExecutionDataflowBlockOptions
{
TaskScheduler = taskSchedulerPair.ConcurrentScheduler,
BoundedCapacity = 1
};
public static async Task<T> LoadAsync<T>(string fileName)
{
T result = default(T);
var transBlock = new TransformBlock<string, T>
(async fName =>
{
return await LoadData<T>(fName);
}, concurrentExecutionDataFlow);
transBlock.Post(fileName);
result = await transBlock.ReceiveAsync();
return result;
}
public static async Task SaveAsync<T>(T obj, string fileName)
{
var transBlock = new TransformBlock<Tuple<T, string>, Task>
(async tupleData =>
{
await SaveData(tupleData.Item1, tupleData.Item2);
}, exclusiveExecutionDataFlow);
transBlock.Post(new Tuple<T, string>(obj, fileName));
await transBlock.ReceiveAsync();
}
MainPage.xaml.cs用法
private static string data = "vjdsskjfhkjsdhvnvndjfhjvkhdfjkgd"
private static string fileName = string.Empty;
private List<string> DataLstSample = new List<string>();
private ObservableCollection<string> TestResults = new ObservableCollection<string>();
private static string data1 = "hjhkjhkhkjhjkhkhkjhkjhkhjkhjkh";
List<Task> allTsk = new List<Task>();
private Random rand = new Random();
private string fileNameRand
{
get
{
return rand.Next(100).ToString();
}
}
public MainPage()
{
InitializeComponent();
for (int i = 0; i < 5; i ++)
{
DataLstSample.Add((i % 2) == 0 ? data : data1);
}
}
private void Button_Click(object sender, RoutedEventArgs e)
{
AppIsolatedStore_TestInMultiThread_LstResultShouldBeEqual();
}
public async void AppIsolatedStore_TestInMultiThread_LstResultShouldBeEqual()
{
TstRst.Text = "InProgress..";
allTsk.Clear();
foreach(var data in DataLstSample)
{
var fName = fileNameRand;
var t = Task.Run(async () =>
{
await AppIsolatedStore.SaveAsync<string>(data, fName);
});
TestResults.Add(string.Format("Writing file name: {0}, data: {1}", fName, data));
allTsk.Add(t);
}
await Task.WhenAll(allTsk);
TstRst.Text = "Completed..";
}
异步保存和加载数据
/// <summary>
/// Load object from file
/// </summary>
private static async Task<T> LoadData<T>(string fileName)
{
T result = default(T);
try
{
if (!string.IsNullOrWhiteSpace(fileName))
{
using (var file = new IsolatedStorageFileStream(fileName, FileMode.OpenOrCreate, _isolatedStore))
{
var data = await _file.ReadTextAsync(file);
if (!string.IsNullOrWhiteSpace(data))
{
result = JsonConvert.DeserializeObject<T>(data);
}
}
}
}
catch (Exception ex)
{
//todo: log the megatron exception in a file
Debug.WriteLine("AppIsolatedStore: LoadAsync : An error occured while loading data : {0}", ex.Message);
}
finally
{
}
return result;
}
/// <summary>
/// Save object from file
/// </summary>
private static async Task SaveData<T>(T obj, string fileName)
{
try
{
if (obj != null && !string.IsNullOrWhiteSpace(fileName))
{
//Serialize object with JSON or XML serializer
string storageString = JsonConvert.SerializeObject(obj);
if (!string.IsNullOrWhiteSpace(storageString))
{
//Write content to file
await _file.WriteTextAsync(new IsolatedStorageFileStream(fileName, FileMode.Create, _isolatedStore), storageString);
}
}
}
catch (Exception ex)
{
//todo: log the megatron exception in a file
Debug.WriteLine("AppIsolatedStore: SaveAsync : An error occured while saving the data : {0}", ex.Message);
}
finally
{
}
}
编辑:
它出现内存异常的原因之一是我获取的数据字符串太大。字符串是链接:http://1drv.ms/1QWSAsc
但第二个问题是,如果我也添加了小数据,那么它就会阻塞UI线程。代码是否在UI踏板上执行任何任务?
不,您使用的并发对为其任务使用默认线程池,并且您使用Run
方法实例化任务,所以问题不在这里。但是这里的代码有两个主要威胁:
var transBlock = new TransformBlock<string, T>
(async fName =>
{
// process file here
}, concurrentExecutionDataFlow);
你真的不应该每次都创建transBlock
。TPL Dataflow
的主要思想是创建一次块,然后使用它们。因此,您应该重构应用程序以减少正在实例化的块的数量,否则就不应该使用TPL Dataflow
。
代码中的另一个威胁是显式地阻止线程!
// Right here
await Task.WhenAll(allTsk);
TstRst.Text = "Completed..";
从同步事件处理程序的async void
方法中为任务调用await
会阻塞线程,因为默认情况下它会捕获同步上下文。首先,应避免async void
。第二,如果您是异步的,那么您应该一直是异步的。所以事件处理程序也应该是异步的(async)。第三,您可以使用任务的延续来更新UI或使用当前同步上下文。
所以,你的代码应该是这样的:
// store the sync context in the field of your form
SynchronizationContext syncContext = SynchronizationContext.Current;
// avoid the async void :)
public async Task AppIsolatedStore_TestInMultiThread_LstResultShouldBeEqual()
// make event handler async - this is the only exception for the async void use rule from above
private async void Button_Click(object sender, RoutedEventArgs e)
// asynchronically wait the result without capturing the context
await Task.WhenAll(allTsk).ContinueWith(
t => {
// you can move out this logic to main method
syncContext.Post(new SendOrPostCallback(o =>
{
TstRst.Text = "Completed..";
}));
}
);
您是否尝试过在ExecutionDataflowBlockOptions上使用BoundedCapacity参数?TPL简介中提到了区块容量:
[…]绑定在数据流网络中很有用,可以避免无边界内存发育出于可靠性原因,如果生产商最终可能更快地生成数据消费者无法处理…
我建议尝试使用此选项,以限制已处理项目的排队,并查看它是否有助于解决内存问题