RavenDB 在高负载下抛出 InvalidOperationException
本文关键字:InvalidOperationException 负载 高负载 RavenDB | 更新日期: 2023-09-27 18:31:32
我不知道为什么,但是当我对RavenDB进行并发写入负载测试时,在高负载下,我收到以下异常:"集合已修改;枚举操作可能无法执行。
高负载,我说的是每秒 1000 个请求,更新 100 个文档。为了对此提供一些上下文,我将使用 RavenDB 在 MVC 站点上记录操作,并且我正在运行一个测试工具来调用 log 方法,以查看它是否可以处理记录这么多事情。每个文档表示用户在站点会话中的操作。
如果有帮助,则具有相同数量请求的较少文档运行良好。每秒请求数比例较高的文档较少也运行良好。
我从研究中发现,在线程之间共享 IDocumentSession 有时是负责任的,但我不相信我的代码会这样做。
下面是相关的日志记录逻辑:
public class ActivityLogger : IActivityLogger
{
private static ActivityLogger _instance;
public static ActivityLogger Instance
{
get
{
if (_instance == null)
{
_instance = new ActivityLogger();
}
return _instance;
}
set { _instance = value; }
}
private static DocumentStore docStoreInstance;
// keeping this as a singleton - we only ever want one as it's expensive
private static DocumentStore DocumentStore
{
get
{
if (docStoreInstance == null)
{
docStoreInstance = new DocumentStore() {ConnectionStringName = "RavenLogging"};
docStoreInstance.Initialize();
}
return docStoreInstance;
}
}
...
/// <summary>
/// Logs the given LogItem to Raven, to the given UserSession
/// </summary>
public void Log(LogItem item, UserSession userSession)
{
if (userSession == null)
{
return;
}
if (!userSession.Activities.Contains(item))
{
userSession.Activities.Add(item);
if (Cache != null)
Cache.AddToSession(CachedObjects.USER_LOGGING_SESSION, userSession);
}
// done in a task since we don't want the app to wait for us to finish logging
Task.Factory.StartNew(() =>
{
using (var session = DocumentStore.OpenSession())
{
try
{
session.Store(userSession);
session.SaveChanges();
}
catch (Raven.Abstractions.Exceptions.ConcurrencyException ce)
{
if (ce.ExpectedETag != ce.ActualETag)
{
// the user session is stale, reload it and try again
userSession = session.Load<UserSession>(userSession.Id);
Log(item, userSession);
}
}
}
});
}
}
这是我的测试工具中的代码
public void Begin()
{
startTime = DateTime.Now;
DateTime endTime = startTime.AddSeconds(Duration);
int itemsLoggedThisSecond, secondsRemaining = 0;
while (DateTime.Now < endTime)
{
// if we're still on the same second somehow, wait
if (endTime.Subtract(DateTime.Now).Seconds == secondsRemaining)
{
Thread.Sleep(10);
continue;
}
itemsLoggedThisSecond = 0;
while (itemsLoggedThisSecond < RequestsPerSecond)
{
int innerItemsLoggedThisSecond = itemsLoggedThisSecond;
Parallel.ForEach(Sessions, (session, state) =>
{
if (innerItemsLoggedThisSecond == RequestsPerSecond)
{
state.Break();
}
var item = MakeRandomLogItem();
ActivityLogger.Instance.Log(item, session);
if (!session.Activities.Contains(item))
{
session.Activities.Add(item);
}
innerItemsLoggedThisSecond++;
});
itemsLoggedThisSecond = innerItemsLoggedThisSecond;
}
secondsRemaining = (int) endTime.Subtract(DateTime.Now).TotalSeconds;
Console.Write("'r" + secondsRemaining + " seconds remaining ");
}
}
如果它有帮助,这里是例外。
System.InvalidOperationException was unhandled by user code
HResult=-2146233079
Message=Collection was modified; enumeration operation may not execute.
Source=mscorlib
StackTrace:
at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)
at System.Collections.Generic.List`1.Enumerator.MoveNextRare()
at System.Collections.Generic.List`1.Enumerator.MoveNext()
at Raven.Imports.Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeList(JsonWriter writer, IWrappedCollection values, JsonArrayContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'Serialization'JsonSerializerInternalWriter.cs:line 524
at Raven.Imports.Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'Serialization'JsonSerializerInternalWriter.cs:line 129
at Raven.Imports.Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeObject(JsonWriter writer, Object value, JsonObjectContract contract, JsonProperty member, JsonContainerContract collectionContract, JsonProperty containerProperty) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'Serialization'JsonSerializerInternalWriter.cs:line 383
at Raven.Imports.Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.SerializeValue(JsonWriter writer, Object value, JsonContract valueContract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerProperty) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'Serialization'JsonSerializerInternalWriter.cs:line 124
at Raven.Imports.Newtonsoft.Json.Serialization.JsonSerializerInternalWriter.Serialize(JsonWriter jsonWriter, Object value) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'Serialization'JsonSerializerInternalWriter.cs:line 62
at Raven.Imports.Newtonsoft.Json.JsonSerializer.SerializeInternal(JsonWriter jsonWriter, Object value) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'JsonSerializer.cs:line 627
at Raven.Imports.Newtonsoft.Json.JsonSerializer.Serialize(JsonWriter jsonWriter, Object value) in c:'Builds'RavenDB-Stable'Imports'Newtonsoft.Json'Src'Newtonsoft.Json'JsonSerializer.cs:line 599
at Raven.Json.Linq.RavenJToken.FromObjectInternal(Object o, JsonSerializer jsonSerializer) in c:'Builds'RavenDB-Stable'Raven.Abstractions'Json'Linq'RavenJToken.cs:line 83
at Raven.Json.Linq.RavenJObject.FromObject(Object o, JsonSerializer jsonSerializer) in c:'Builds'RavenDB-Stable'Raven.Abstractions'Json'Linq'RavenJObject.cs:line 159
at Raven.Client.Document.EntityToJson.GetObjectAsJson(Object entity) in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'EntityToJson.cs:line 74
at Raven.Client.Document.EntityToJson.ConvertEntityToJson(String key, Object entity, RavenJObject metadata) in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'EntityToJson.cs:line 41
at Raven.Client.Document.InMemoryDocumentSessionOperations.EntityChanged(Object entity, DocumentMetadata documentMetadata) in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'InMemoryDocumentSessionOperations.cs:line 1033
at Raven.Client.Document.InMemoryDocumentSessionOperations.<PrepareForEntitiesPuts>b__14(KeyValuePair`2 pair) in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'InMemoryDocumentSessionOperations.cs:line 908
at System.Linq.Enumerable.WhereEnumerableIterator`1.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at Raven.Client.Document.InMemoryDocumentSessionOperations.PrepareForEntitiesPuts(SaveChangesData result) in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'InMemoryDocumentSessionOperations.cs:line 908
at Raven.Client.Document.InMemoryDocumentSessionOperations.PrepareForSaveChanges() in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'InMemoryDocumentSessionOperations.cs:line 901
at Raven.Client.Document.DocumentSession.SaveChanges() in c:'Builds'RavenDB-Stable'Raven.Client.Lightweight'Document'DocumentSession.cs:line 694
at Pendragon.Infrastructure.ActivityLogger.<>c__DisplayClass2.<Log>b__1() in c:'Users'me'repos'project'project'Infrastructure'Logger.cs:line 103
at System.Threading.Tasks.Task.InnerInvoke()
at System.Threading.Tasks.Task.Execute()
我在这里没有任何线索,所以任何帮助
在某个级别,一个线程枚举集合,然后另一个线程更改集合。简而言之,ActivityLogger.Instance
不是线程安全的。
我每个线程至少有一个乌鸦连接。您可以考虑根本不共享连接,并且每个请求都有一个连接,特别是因为托管时每个请求都会有一个 MVCController,因此这将提供更准确的压力测试。
简单的解决方案可能是松开单例ActivityLogger.Instance
并在Parallel.ForEach
内创建一个新,即
ActivityLogger.Instance.Log(item, session);
成为:
new ActivityLogger().Log(item, session);
(假设没有其他单身人士在起作用?