MongoDB Cursor Snapshot with C# Driver
up vote
2
down vote
favorite
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
|
show 1 more comment
up vote
2
down vote
favorite
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 at 3:13
SpecificallyMongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.
– Neil Lunn
Nov 20 at 3:18
|
show 1 more comment
up vote
2
down vote
favorite
up vote
2
down vote
favorite
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
I have a large collection that I need to iterate through. I am using a cursor for this.
However, I noticed I am getting duplicate records sometimes from this cursor. I did some reading and found that I can address this by using a snapshot.
Cursor Isolation
As a cursor returns documents, other operations may interleave with the query. For the MMAPv1 storage engine, intervening write operations on a document may result in a cursor that returns a document more than once if that document has changed. To handle this situation, see the information on Cursor Snapshot.
However, I am unable to find this method in the C# Driver (Version 2.7). I don't see it on FindFluent or in FindOptions.
Can someone point me to an example of how I can create a cursor snapshot via the C# MongoDB Driver?
Here is what my code looks like.
private async Task<int> IterateViaCursor<P>(FilterDefinition<User> filter, ProjectionDefinition<User, P> pdef, Func<IEnumerable<P>, Task> callback, int batchSize)
where P : class
{
var o = new FindOptions { BatchSize = batchSize };
var count = 0;
var findFluent = Users.Find(filter, o).Project(pdef);
using (var cursor = await findFluent.ToCursorAsync())
{
while (await cursor.MoveNextAsync())
{
var batchCount = cursor.Current.Count();
count += batchCount;
if (batchCount > 0)
await callback(cursor.Current.ToList());
}
}
return count;
}
c# mongodb cursor
c# mongodb cursor
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
edited Nov 20 at 2:58
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
asked Nov 20 at 1:59
Vadim Rybak
1,1921525
1,1921525
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 at 3:13
SpecificallyMongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.
– Neil Lunn
Nov 20 at 3:18
|
show 1 more comment
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 at 3:13
SpecificallyMongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.
– Neil Lunn
Nov 20 at 3:18
1
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 at 2:05
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 at 2:59
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 at 3:08
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 at 3:13
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 at 3:13
Specifically
MongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call this countDocuments() or similar, but it's not changed here. This is the recommended reliable way to count query results.– Neil Lunn
Nov 20 at 3:18
Specifically
MongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call this countDocuments() or similar, but it's not changed here. This is the recommended reliable way to count query results.– Neil Lunn
Nov 20 at 3:18
|
show 1 more comment
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53385168%2fmongodb-cursor-snapshot-with-c-sharp-driver%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
active
oldest
votes
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53385168%2fmongodb-cursor-snapshot-with-c-sharp-driver%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Snapshot was deprecated and then removed for MongoDB 4.0. jira.mongodb.org/browse/SERVER-32173. So you 'read some old posts" which talked about snapshot, but this was really just a "band aid" and not a real solution, hence the removal. Instead, explain to us "why" you think you need to iterate a long cursor and the actual code you are using showing you these "duplicates". Even if you are using an older MongoDB server, the solution to your code should not be one using the snapshot option.
– Neil Lunn
Nov 20 at 2:05
The reason I want to use a snapshot is because I am getting duplicate records when iterating my cursor. I updated my question with some more information
– Vadim Rybak
Nov 20 at 2:59
Can you provide an MCVE that actually reproduces the behavior then? The code provided seems fine, so any issue is likely more related to something else that is happening or other configuration. Note that if you follow though the links from the issue raised to deprecate and remove, it should become apparent that "snapshots" never actually did what most people were expecting them to do, and are also tied to the now removed MMAPv1 storage engine. Newer storage engines have different read isolation by default.
– Neil Lunn
Nov 20 at 3:08
Was about to talk about something else, but is this the actual code in use? If so then all you are really doing here is "counting" results. Unless there is some other purpose for the code not included here, then there is a far more effective way to to that, instead of iterating a cursor and incrementing a counter.
– Neil Lunn
Nov 20 at 3:13
Specifically
MongoCollection.Count(IMongoQuery), which I just had to check because some language API's now call thiscountDocuments()or similar, but it's not changed here. This is the recommended reliable way to count query results.– Neil Lunn
Nov 20 at 3:18