CHanges in large Folder are missed?

[zaadstra]

Hi,

I've added two NASses into mij database, by adding all shares as separate Folders. They don't update (scheduled) as the shares aren't default connected / logged in with password.

Now I notice that 'Monitor Changes' does not notify changes, like rename, copy and delete on one of my shares. I tested on the other NAS and then on the same in a different folder, and there the changes were picked up and shown instantly. Is there an issue with very large folders? I did a rescan on the Folder (took a long time) but changes still are not shown, even after some while.

The Folder has about 904000 objects and more than 7 TB of size.

Running installed "Everything" 1.4.1.992 x64, non-admin with service on Win10.

[void]

Check your debug logs to see if the changes are being picked up at all.

Everything uses ReadDirectoryChanges to detect changes to folders.
This might be a limitation with ReadDirectoryChanges.

To enable debug mode:

• In Everything, type in the following search and press ENTER:
  /debug

This will show a debug console.

ReadDirectoryChanges notifications start with RDC in Cyan color.

Is Everything detecting the changes? -If in doubt, could you please send your debug logs?


To disable debug mode:

• In Everything, type in the following search and press ENTER:
  /debug

privacy

[zaadstra]

Done that, and for a moment I thought it works and I felt very silly. I had removed some index options (create date and attributes) as the mouse pointer now is very nervous and very many times briefly shows the hourglass (=win 10 circle thing). It seems to be a bit more quiet.

However, I discovered an interesting thing. In all shares are changes registered, but only in the root folder.
I could see the RDC's but these are missing when copying my testfile to a \sub\subfolder\. I hope that gives a clue!

I'm not sure what specific or personal things are logged in debug log, so I start showing some of the RDC parts. But is seems they are not there when using subfolder. Idea: I was missing a timestamp in front of the lines for this. Also, full path after the RDC would be helpful (options?).

Code: Select all

RDC ADDED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
new results 3
update m 0 12245808
update index C:

...

DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.001064 seconds
RDC ADDED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
new results 3
update m 0 12245808
update index C:

....

DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.001064 seconds
RDC ADDED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
RDC MODIFIED _MYTESTFILE_.txt
new results 3
update m 0 12245808
update index C:

...

resume ntfs monitor 0
processed usn records in 0.002637 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000409 seconds
RDC REMOVED _MYTESTFILE_.txt
new results 2
update m 0 12245808
update index C:

...

resume ntfs monitor 0
processed usn records in 0.000233 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000134 seconds
RDC ADDED Ricki_Lee-Fear_and_Freedom-2012-pLAN9
RDC MODIFIED Ricki_Lee-Fear_and_Freedom-2012-pLAN9
update m 0 12245808
update index C:

....

DB_WAIT: _db_ready_proc waited 0.000045 seconds
RDC RENAMED_OLD_NAME Ricki_Lee-Fear_and_Freedom-2012-pLAN9
RDC RENAMED_NEW_NAME Ricki_Lee-Fear_and_Freedom-2012-pLAN9s
parent backup count 0
update m 0 12245808
find excluded
readd folders

...

resume ntfs monitor 0
processed usn records in 0.000441 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000048 seconds
RDC REMOVED Ricki_Lee-Fear_and_Freedom-2012-pLAN9s
remove subfolders and files from Ricki_Lee-Fear_and_Freedom-2012-pLAN9s
parent backup count 0
removed 0 subfolders and

....

DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000123 seconds
update m 0 12245808
WM_ACTIVATE 0000000000000002 0000000000000000, lastfocus 0000000000030610, current focus 0000000000000000
RDC ADDED _MYTESTFILE_.txt
unable to set focus 0
RDC MODIFIED _MYTESTFILE_.txt
SYSCOMMAND 000000000000f012 0000000002300498
update index C:

...

updated C: in 0.005990 seconds
resume ntfs monitor 0
processed usn records in 0.000327 seconds
DB_WAIT: _db_ready_proc waiting...
DB_WAIT: _db_ready_proc waited 0.000149 seconds
RDC REMOVED _MYTESTFILE_.txt
new results 2
update m 0 12245808
update index C:

If you need more fragments from the log, I have saved it to a file.

[void]

Thanks for the debug logs.

I could see the RDC's but these are missing when copying my testfile to a \sub\subfolder\. I hope that gives a clue!

Was there no RDC message when moving files into \sub\subfolder? -this means ReadDirectoryChange is not working at all for this subfolder.
This large folder must be a limitation with ReadDirectoryChange.

I don't have a solution for the current build, sorry.

I'll look into ReadDirectoryChange alternatives, like SHChangeNotifyRegister.

[zaadstra]

Thanks for looking into this. Indeed when an action is missed I don't see a RDC.

The folder wasn't that deep, someting like (just tested) \\nas\video\New folder\_MYTESTFILE_.txt. I can make statistics about number of folders, but as it wasn't picked up on shares with a small number of files and folders, this makes no sense.
The first time I copied to the share root (\\nas\video) it was missed too, the second was registered.

Haha it is even more strange, when moving the file between \\nas\video\New folder and \\nas\video\ and back, it appears and disappears in the index!

As this is an unc path, I also tested when mapping the video share to Z:\ but the results are the same. When doing this ans seeing both unc path and Z:\, the moving file and dis/appear in index did not work anymore. Deleting and copying into share root is detected.

[zaadstra]

I was thinking .... If the partial update as future update works well (especially for folders), I'd probably stop "Attempt to monitor changes". Then I would only index small updates parts once in a while.
I would like it if the shares aren't connected all the time (and logged in with my user account). This prevents connecting to the nas with other accounts.