Folder Scan incomplete

[Fred0815]

Hi
still the scanning behavior looks pretty implausible to me. On large volumes I scanned subfolders separately and quite often the size of data found in some subfolders is larger than the top folder. Some subfolders may have access restrictions and I have the feeling this affects the overall result. I did a complete rescan with the most current version because the scanning behavior has been changed. Still it is the same.
Unfortunately, I cannot undisclose folder scans structure. But top level is like see attachment. I will detail the subfolders a bit more for a next test to see if a certain subfolder is causing it...
Best,
Fred

[horst.epp]

If you have an installation using the service there should be no access restriction.
If you can't do this, user folder indexing with folders you definetely have read access.

[void]

Please check the debug log.

Scanning errors are reported in red in the debug console.

The debug console can be shown from Tools -> Debug -> Console.

Please try rescanning when debug logging started:

• In Everything, from the Tools menu, under the Debug submenu, click Start Debug Logging.
• From the Tools menu, click Options.
• Click the Folders tab on the left.
• Select your \\server\share
• Please make sure Fast Rescan is unchecked.
• Click Rescan Now.
• After rescanning completes (current folder being rescanned is shown at the top right):
• In Everything, from the Tools menu, under the Debug submenu, click Stop Debug Logging.
  ---This will open your Everything Debug Log.txt---
  Please check this log for any FindFirstFileEx errors
• What error codes are reported? (if any)
• If unsure, please send this file to support@voidtools.com

[Fred0815]

Dear void team,
I did the following test and found some FindFirstFileEx error codes 1908.
Reason seems to be: the top folder takes 1-2 days to scan. In the afternoon I checked the progress and the top folder was about 50% done. Then I switched the notebook in power saving/hyberenate.
In the morning the rest of the top folder was imediatelly skipped. From the logs I can see that all the rest of the subfolders got FindFirstFileEx error codes 1908. => scan completed

I suspect that always after waking up the notebook in the morning the rest of the current rescan activity is ended the same way and contained files are gone from the database.

Best regards,
Martin

[void]

Thank you for your feedback Martin,

Error 1908 is ERROR_DOMAIN_CONTROLLER_NOT_FOUND

The next alpha update will wait for 1 minute for the network to reconnect when rescanning your network share.

I'll make another post here once this is ready for testing.

the top folder takes 1-2 days to scan

Everything is taking too long to index your network drive.

Everything should take roughly 1 minute to index 1 million files on a network drive.

Ideally, Everything should be indexing your network drive within 60 minutes.
Everything is not really designed to index for longer than 60 minutes.

How many files are you indexing?

[Fred0815]

Hi,
when I do DIR /S on a folder structure it takes roughly 1.5sec for a folder. There are >100.000 folders and 650k files. (at least to the point Everything did the indexing). So also Dir /s would take similarly long...
I assume there might be rate limiter on server side or active directory or whatever else...
Have a good day,
Martin

[void]

Everything 1.5.0.1334a will now wait for one minute for the network to reconnect after waking from sleep.

when I do DIR /S on a folder structure it takes roughly 1.5sec for a folder

Something is not right here, indexing ~1 million files should only take 1 minute, not 1-2days.

Please check your network, hardware and domain settings.

[Fred0815]

Hello, thank you for the improvement.
It took quite a while, but the folder sizes look plausible now!

The slower the browsing is the more useful is a tool like Everything
On my end I cannot do too much to improve the infrastructure. Maybe the limitation is intended.
Asking friends working for other companies the scan rate is 5-20 folders/second.
My 650k folders would still take 9h.. multiple cross domain scans seem to be the worst case...

Anyway, good that it continues scanning after waking from PC sleep mode now!

[Fred0815]

I forgot to highlight my conclusion: It is really important to keep scanned network data as long as possible.
If a scan still fails for reasons not known so far, the files/folders should be kept for a certain time.
Maybe a property "last seen" help or marking them "obsolete".
If large amount of data will be deleted (because Everything thinks the files are gone) users should acknowledge.
I can also image a "last seen" property may bring new fields of application to Everything because it included the history of a folder structure.
So, an option could be "show history" or not and never (or manually) delete valuable information!

[Fred0815]

Hi again,
unfortunately, there's still some issues with losing data from scanned folders and debug was not running.
A larger folder was scanning for almost two days and during that scan I moved from homeoffice to office. The notebook was in hybernate mode.
After logging in the scan was ended and a new one was running. All information from the previous scan is lost.

It may be of interest that we use Zscaler to connect to our drives.
During another event I noticed that Everything took its minute to let the network recover. Now it looks like Everything did not wait.

I am also wondering why ALL data of the scan is lost. Wouldn't it be possible to add new files/folder more frequently? e.g. every 1000 items?
In that case losses would be limited.

Re-scans should scan "unknown" folders first to continue after a partial scan is re-scanned.

Will post again as soon I can reproduce it with Debug ON.
Cheers,
Martin

[void]

Thank you for the issue report.

The wake from hibernate is most likely killing the rescan.

It will be interesting to see the error code in the debug logs.

During another event I noticed that Everything took its minute to let the network recover. Now it looks like Everything did not wait.

It will depend on the error code.
Some hard errors will force Everything to abandon the rescan immediately.
Debug logs will show the scanning error code.

I am also wondering why ALL data of the scan is lost. Wouldn't it be possible to add new files/folder more frequently? e.g. every 1000 items?

Adding files/folders maybe...
I'll consider an option to add on the fly.
Removing deleted files/folders no.



Thank you for enabling logging.

[Fred0815]

Good morning, I am happy that I captured the issue (hopefully).
Normally I have the Notebook in Hybernate, but now I found the problem occurs after a normal restart.

In our environment it may take several minutes ... 30min until all network drives are accessible. Some shares require signing in with different methods, others just take that long.

If the drives are accessed before they are properly working we get Error Code 0x80070035 (Network path not found).
I added the Windows error message as well the part of the log file.

In my opinion scanned files and folders must not easily be deleted! Imagine being on a business trip without logging in the proper way would cause all items deleted (in case a re-scan is running on that folder).

The folder it scanned this time took almost a week and at 90% done this one happened. I have six larger folders that never finished within several month now.

[Fred0815]

Hi, this one really drives me crazy! I had one folder scan running for 6days and to prevent the error above I put indexing to PAUSE > shutdown notebook completely.
Next day (in office environment) I continued indexing. It worked well for 2h but then Everything switched to the next folder and did not store any file or folder from the previous scan.
Debug ON has not been remembered and was off, so I cannot (again) identify the reason.
=> would it be possible to keep Debug On by default for the alpha versions? It's a pain in the a.. to dig into this error, but I believe the scanning mode needs to modified, chunks should be stored every 1k...100k items to not loose it + rescan shall scan "unknown" folders first, otherwise there will never be progress...
Disappointed Greetings,
Martin

[void]

=> would it be possible to keep Debug On by default for the alpha versions?

To keep debug logs on:

• Copy and paste the following into your Everything search box:
  /debug_log=1
• Press ENTER in your Everything search box.
• If successful, debug_log=1 is shown in the status bar for a few seconds.
  ---
• Copy and paste the following into your Everything search box:
  /restart
• Press ENTER in your Everything search box.

Please look in the logs for the following errors:
FindNextFile <error-code>
FindFirstFileEx <error-code> <path>

Please make sure you are using version 1339 or later for this detailed debug information.

Please let me know if you find any errors reported.

Please note, debug logs can grow very large.



To turn off debug logs:

• Copy and paste the following into your Everything search box:
  /debug_log=0
• Press ENTER in your Everything search box.
• If successful, debug_log=0 is shown in the status bar for a few seconds.
  ---
• Copy and paste the following into your Everything search box:
  /restart
• Press ENTER in your Everything search box.

Error 0x80070035 is ERROR_BAD_NETPATH
Everything expects the path to always be online during the scan.

You can do partial updates with a /rescan search command.
For example:

/rescan \\server\share\a folder\that needs\updating



If you can, instead of indexing \\server\share, please try indexing \\server\share\folder1, \\server\share\folder2, \\server\share\folder3 etc...
This will help split the updating done over multiple indexes.

[Fred0815]

Hi, I will mitigate and split the task as much as possible. Although there are some "flat" large folders with some hundred subfolders. These cannot be split...

Folder/Sub1
Folder/...
Folder/SubNx100

[Fred0815]

Hi
this time with debug on I caught the issue.
I did a normal restart and the folder that has been scanned before ended and approx. every 1minute another folder was scanned but also ended every 1min. After some minutes all re-scanning/scanning was finished without results.
Looking at the log I cannot see why it ended, so I add it here. Folders/server needed to be renamed, but structure in general is visible.
It started with P-Order and then changed as described above.
Best,
Martin

[void]

Thanks for the logs Martin,

I'm looking into the issue..

[void]

Everything is failing to access the root network share.

I'm unsure why at this stage.

I will add more debug information in the next alpha update.
I'll report back here again soon.

[void]

Everything 1.5.0.1340a adds more debug information when trying to access root folders.

Could you please repeat the same debug log capture with this version:

Please try rescanning when debug logging started:

• In Everything, from the Tools menu, under the Debug submenu, click Start Debug Logging.
• From the Tools menu, click Options.
• Click the Folders tab on the left.
• Select your \\server\share
• Click Rescan Now.
• After rescanning completes (current folder being rescanned is shown at the top right):
• In Everything, from the Tools menu, under the Debug submenu, click Stop Debug Logging.
  ---This will open your Everything Debug Log.txt---
• Please check this log for the following error:
  failed to get find data for <filename>: <error-code>
• What error codes are reported? (if any)
• If unsure, please send this file to support@voidtools.com

[Fred0815]

Good morning,

I got error code 1908. Note that the PC was restored from power saving/sleep at 9:06:00.
After only 19s Everything stopped scanning without waiting/retry. Shouldn't it wait/retry at least for 1min?
As I said, sometimes it takes up to 15min until all network connections work well. I guess active directory and other services take that time.
You might think "what a crappy network", but the company I am working with has >200.000 employees, so it is not like a garage-startup not able to handle networks... I asked many other friends to benchmark their network in other companies and the speed to scan a network is similar.
So, I guess in order to use Everything, there needs to be some modification to account for very long scan times.

The easiest workaround could be:
- write scan results after an error and do not ignore them
- do not delete scan results if the folder is not available
- continue scanning: skip known folders and concentrate on unknown folders. Hopefully, that avoids re-scanning from start again and again and never reach the end

Best regards,

2023-03-27 09:06:19.781: failed to get find data for \\Server1.net\a\a\a\a\20_CSY: 1908
2023-03-27 09:06:19.781: scan completed in 315845.332347 seconds
2023-03-27 09:06:19.781: rescan update db
2023-03-27 09:06:19.781: rescan update db in 0.000316 seconds
2023-03-27 09:06:19.781: next update at 8/05 0:06
2023-03-27 09:06:19.862: leave folder update
2023-03-27 09:06:19.864: check for folder updates 0000000000000000
2023-03-27 09:06:19.864: update folder start \\Server1.net\a\a\a\a\30_ES

PS: error 53 often occurs on folders with missing access permission. In that case Everything retries 10x? every 1min. I guess that's not necessary. Such folders may be skipped instead.

[void]

Thank you for your reply Fred0815,

Error 1908 is ERROR_DOMAIN_CONTROLLER_NOT_FOUND.

Everything 1.5.0.1341a will now retry scanning the folder after receiving this error.
Everything will retry every minute for up-to 5 minutes.



Error 53 is ERROR_BAD_NETPATH.
This error is common when the network is down or a server is offline.
It's important for Everything to wait and retry on this error.

Please try excluding folder paths where you don't have access under Tools -> Options -> Exclude.

[Fred0815]

Thank you, will try.
The folder structures are too deeply nested to individually exclude folders with no authorization.
I found this option that was ON so far. I am wondering whether that helps to keep "offline" data: Network drives/Automatically remove offline mapped network drives => now OFF

BTW: I modified Indexes options to add "date created" (without "force rebuild"). That caused ALL 7mio files to be deleted from the database... OK, starting from scratch again.
=> such huge delete operations must be avoided (some ideas in my previous mails). Minimum would be to have a confirmation dialog. + maybe a backup of the db to allow to restore it.

[horst.epp]

Thank you, will try.
The folder structures are too deeply nested to individually exclude folders with no authorization.
I found this option that was ON so far. I am wondering whether that helps to keep "offline" data: Network drives/Automatically remove offline mapped network drives => now OFF

BTW: I modified Indexes options to add "date created" (without "force rebuild"). That caused ALL 7mio files to be deleted from the database... OK, starting from scratch again.
=> such huge delete operations must be avoided (some ideas in my previous mails). Minimum would be to have a confirmation dialog. + maybe a backup of the db to allow to restore it.

Version 1341a now makes a backup of the database file.
viewtopic.php?p=55420#p55420

[Fred0815]

great, thank you

[Fred0815]

Hello again,
Everything scanned
folder A: \\srv1.net\a\b\c\d\gr\Projekte
after a restart of the PC it looks like without any error it ended folder A, did not write anything to db and continued with
folder B \\srv1.net\a\b\c\d\gr\Standardprojekt\Standards

Actually, Everything scans currently folder A and shows progress in the tooltip text. But the current log does not show scanning activities in folder A as far as I can see it.

Strange things happen.... any idea?

Thanks,
Martin

2023-04-13 13:03:21.805: scan completed in 1433.184708 seconds
2023-04-13 13:03:21.806: rescan update db
2023-04-13 13:03:21.912: rescan update db in 0.107055 seconds
2023-04-13 13:03:21.912: next update at 25/05 4:03
2023-04-13 13:03:21.913: leave folder update
2023-04-13 13:03:21.913: check for folder updates 0000000000000000
2023-04-13 13:03:21.913: update folder start \\srv1.net\a\b\c\d\gr\Projekte
2023-04-13 13:03:21.913: scanning \\srv1.net\a\b\c\d\gr\Projekte...
2023-04-13 13:03:21.914: RESCAN CREATE \\srv1.net\a\b\c\d\gr\Standardprojekt\Standards

[void]

Thank you for your reply Martin,

But the current log does not show scanning activities in folder A as far as I can see it.

Everything will only show the current folder being scanned in the console/log if verbose logging is enabled.

To enable verbose logging:

• In Everything, from the Tools menu, under the Debug submenu, check Verbose.

Everything will abandon a rescan if you restart your PC.

[Fred0815]

Hi again, ok I switched verbose ON.

looks like a few network error 64 showed up and after 1 minute showing the error scanning ended + all results are lost.
Again: I don't know how many other errors will show up causing to loose all the information from the current scan.
I my opinion only a different approach to have higher priority to keep data will help...

BR,
Martin

2023-04-18 12:28:52.376: FindFirstFileEx 64 srv1.net\a\b\c\d\sysadmi\Software\Remote\CiscoVPN\*
2023-04-18 12:28:52.538: MSG: 00000000000305a2 0113 0000000000000002 0000000000000000
2023-04-18 12:28:52.540: TRAY 00000113 0000000000000002 0000000000000000
2023-04-18 12:28:52.540: _db_query_results_changed_event_proc 0000000000000000
2023-04-18 12:28:52.540: _db_query_results_changed_delay_timer_create
2023-04-18 12:28:52.540: new results 2788969
2023-04-18 12:28:52.540: listview message 0000004e 0000000000002719 0000000000fbdbf0
2023-04-18 12:28:52.540: listview message 0000004e 0000000000002719 0000000000fbdbf0
2023-04-18 12:28:52.540: get_total_result_size 2346847 3627586310821
2023-04-18 12:28:52.541: MSG OK
2023-04-18 12:28:52.550: FindFirstFileEx 64 srv1.net\a\b\c\d\sysadmi\Software\Remote\CiscoVPN\*
2023-04-18 12:28:52.568: MSG: 00000000000305a2 0113 0000000000000003 0000000000000000
2023-04-18 12:28:52.570: TRAY 00000113 0000000000000003 0000000000000000
2023-04-18 12:28:52.570: MSG OK
2023-04-18 12:28:52.608: failed to get find data for srv1.net\a\b\c\d\sysadmi: 64
2023-04-18 12:28:52.708: MSG: 00000000000305a2 0113 0000000000000000 0000000000000000
2023-04-18 12:28:52.708: TRAY 00000113 0000000000000000 0000000000000000
2023-04-18 12:28:52.708: MSG OK
2023-04-18 12:28:52.709: MSG: 00000000000305a2 0113 0000000000000001 0000000000000000
2023-04-18 12:28:52.709: TRAY 00000113 0000000000000001 0000000000000000
2023-04-18 12:28:52.709: check for folder updates 000000001e0dab10
2023-04-18 12:28:52.709: MSG OK

[void]

Thank you for the logs Martin,

Everything 1.5.0.1344a simplifies rescanning folders.

If FindFirstFile/FindNextFile fails for any reason, Everything will check if the root volume is still online.
If the root volume is still online, the folder is treated as empty and Everything will continue.
If the root volume is offline, Everything will wait for up-to 5 minutes until the volume comes back online and try again.
If the root volume does not come back online after 5 minutes, Everything will abort the rescan process and reschedule a rescan for another time.

Please let me know if this helps with rescanning the folder.

[Fred0815]

Thank you, will try...
Just noticed that .1344 deletes all previous scan results...
will try to use db backup.... of course it does not help.. still empty...
ok, I am used to start from scratch... so let's do it again...
;-|

[horst.epp]

Thank you, will try...
Just noticed that .1344 deletes all previous scan results...
will try to use db backup.... of course it does not help.. still empty...
ok, I am used to start from scratch... so let's do it again...
;-|

As noted in the 1344a announcement the database format has changed.
You can't use an old database, it will be automatically rebuild.

[Fred0815]

after reboot network takes up to 30min to establish. Everything 1350 tries several times (total a few minutes) and then skips this folder.

All scan results done so far are still lost. I thought the database changes were meant to address it?

2023-06-20 09:21:12.036: failed to get find data for \\aserver.net\101\File\E\ORDERs\Transfer: 50

=> there is so much going on in Everything development, but the real basics are not addressed sufficiently. Am I really the only one suffering from slow network? We are >200.000 employees over here...

[horst.epp]

after reboot network takes up to 30min to establish. Everything 1350 tries several times (total a few minutes) and then skips this folder.

All scan results done so far are still lost. I thought the database changes were meant to address it?

2023-06-20 09:21:12.036: failed to get find data for \\aserver.net\101\File\E\ORDERs\Transfer: 50

=> there is so much going on in Everything development, but the real basics are not addressed sufficiently. Am I really the only one suffering from slow network? We are >200.000 employees over here...

For such environment, there is the Everything server.

[void]

after reboot network takes up to 30min to establish

This sounds like an unusually long time..



Please try increasing your folder_rescan_timeout:

• In Everything 1.5, from the Tools menu, click Options.
• Click the Advanced tab on the left.
• To the right of Show settings containing, search for:
  rescan
• Select folder_rescan_timeout.
• Set the value to: 3600000
  (this is one hour, the default is 300000, which is 5 minutes)
• Click OK.

folder_rescan_timeout



Everything Server

[Fred0815]

I implemented 30min delay to skip scanning a folder.
I observe some folder shares are even longer as 30min offline.

Issue: last scanned folder over night “…Pro_m_L” just was skipped to another one without obvious reason.
No result was committed to the database.

Looking at the long time we try to get this working I doubt that we will be successful by doing it half-hearted.

To make it right it takes the measures I suggested in my previous comments:
- Never drop any data (possible option/manual step)
- Commit to database every minute...10min
- Add e.g. “last seen/deleted” to the database to identify files that (temporarily) might not be available
- Continue scanning a folder where it ended before and not start from start again and again

I know that is much of a work, but it also opens a path to new features based on the history of files.

=> I sent the logfile separately via mail

[void]

Thank you for your feedback and logs Fred0815,

Adding a drive to your index is interrupting the folder scan.
I'll look into keeping the folder scan going when rebuilding the index.



For now, please disable automatically include NTFS volumes and please disable automatically remove NTFS volumes:

• In Everything, from the Tools menu, click Options.
• Click the NTFS tab on the left.
• Uncheck Automatically include new fixed volumes.
• Uncheck Automatically include new removable volumes.
• Uncheck Automatically remove offline volumes.
• Click OK.

Please let me know if the issue persists.

Never drop any data (possible option/manual step)

Everything should keep trying to read a directory listing on any error.
Everything will timeout after the setting folder_rescan_timeout (5 minutes by default -30 minutes in your case).

A rebuild will currently interrupt a folder rescan.
I'll look into keeping the folder scan going when rebuilding the index.

Commit to database every minute...10min

Tough to do without a full rescan.
I'll look into doing this.

Add e.g. “last seen/deleted” to the database to identify files that (temporarily) might not be available

I will consider adding properties to show this information.
For now, Last rescan information is available under Tools -> Debug -> Statistics -> Folder Index.

Continue scanning a folder where it ended before and not start from start again and again

I will consider support for saving the current scan to disk.

For now, please consider breaking your folder index into smaller folders.
For example, instead of indexing \\server\share, index:
\\server\share\folder1
\\server\share\folder2
\\server\share\folder3



Thank you for the suggestions.