ES CLI can hang (or return immediately) right after index rebuild initiation

[mq3]

Hi Everything Development Team and community,

Thanks very much for the wonderful application. It is definitely one of my most favorite programs -- it brings absolute ease to file management, in my opinion.


Anyway, my goal is to use the command-line to first rebuild the index, and then to get the results of a query. I would prefer for the query to be issued right when the index has finished being rebuilt.

Towards trying to accomplish this, I compromised and I have been using the below command (the ES call is an example):

Code: Select all

Everything.exe -startup -reindex && timeout /t 1 && ES -sort size folder: size:^>1gb

The reason I have the timeout in the above command is that without it, I have noticed the behavior to be unpredictable:

• Sometimes the ES call will wait until the index is rebuilt, and then return the expected results. (This is the desired behavior.)
• Other times the ES call will return with results right away.
• And other times it seems that the ES call will hang, requiring an interrupt to end the command.

So I was wondering: Is there a way I can achieve my goal without needing the timeout? Is there another cmd line command that can help me achieve my goal without the timeout? Perhaps there's a file I can query to know when the rebuild has finished?

Thanks very much.



More Information / Thoughts
I have noticed that the -reindex Everything.exe call only initiates the index rebuild, and then it returns. However, I have also noticed that the ES CLI seems to be designed to wait for the index to be completed being built when an index rebuild is occurring.

I don't know for sure, but what seems to be going on when there is no timeout is that the ES call can significantly progress before the Everything.exe rebuild call has already indicated that a rebuild is intended, or perhaps progress. Thus, this should explain the unpredictable three different behaviors.

(Just my 2 cents, but it seems that the proper way to fix this would be for there to exist a cmd line command that initiates a "blocking index rebuild", meaning a command that initiates an index rebuild AND it doesn't return until the rebuild is complete. This would allow for a consistently-behaved, quick succession of an errorlevel conditional sequence of an index rebuild followed by an ES CLI query against that new index.)

[therube]

See if this helps, Add an option to tell everything.exe to re-index nightly.

[mq3]

The above hyperlink seems to not be working, but I'm guessing the following is the correct resource? Add an option to tell everything.exe to re-index nightly

[therube]

(Fixed, above.
And yes.
Wonder how that happened.)

[mq3]

Hi therube, thanks for the information,

I read through once the nightly re-index thread. It gives me a little more general information, but so far I haven't figured out if it can help me achieve my goal. Would you mind giving me a hand here? Perhaps I just missed something important in the thread.

Thanks.

[void]

The Everything.exe -startup -reindex part will succeed immediately, and es will be called before the database rebuild command is issued.

ES will hang if the database is rebuilt, while the es query was active.
The rebuild will terminate the query and ES will never receive the results.
I've added this to my "Things to fix" list.

Please try:
START /WAIT Everything.exe -startup -reindex && ES -sort size folder: size:^>1gb
This will cause the ES call to wait until after the reindex command has been issued.

This assumes "Everything" is already running, otherwise ES will never get called.

Thanks for the bug report.

[void]

If Everything is not running you might like to try:
everything -startup && timeout 1 && START /WAIT Everything.exe -startup -reindex && ES -sort size folder: size:^>1gb

[mq3]

Hi void, thanks for the help,

I'm planning to have Everything always running, but thanks for that tip, could come in handy later on.

And I've tried the START /WAIT command line that you suggested, and so far it has worked consistently with the desired behavior. Amazing, thank you.

Now I'm wondering why this works.

I know that the Everything.exe -reindex call only non-immediately issues the rebuild command, and thus the START /WAIT command line still calls the ES CLI before the database is completely rebuilt. (I also double checked this by replacing the ES CLI call with `echo ES-is-awesome`, and "ES-is-awesome" was echoed right away when the command line was issued, but the database was still being rebuilt.)

However, unless I'm missing or don't know something, I don't see how this cmd line command makes sure that the database rebuild command (that is non-immediately issued by Everything.exe call) always happens before the search query (that is non-immediately issued by the ES CLI call).

The reason I want to know is that I am planning to use such a command line in a script that might be automated, and anything that can increase my confidence that a hang won't occur would greatly benefit me.

Thanks again.

[void]

Everything likes to have only one instance of Everything.exe running.

When Everything is already running in the background and you run:
Everything.exe -startup -reindex

This second instance of Everything will pass the parameters "-startup -reindex" to the first instance of Everything.
The second instance will return once it knows the first instance of Everything has received and processed these parameters. This allows you to use START /WAIT so you know the command has been processed.

-reindex is a non-blocking command, Everything will not wait for the full reindex to complete, it will return once the database has been emptied and the new reindex has started.

To simplify, when you run Everything.exe -startup -reindex, it is the first instance of Everything that actually processes these commands.

Useful links:
https://stackoverflow.com/questions/451 ... ll-command
https://superuser.com/questions/908664/ ... ntil-it-is

[mq3]

Thanks void, really helps.

[NotNull]

There might be another workaround: Start an Everything command that is dependent on the outcome of the reindexing.

I reasoned that an everything.exe -update (writing the database to disk) will not do so before the reindexing is complete.
Can't be sure, but the results indicate that this is the case:

Code: Select all

echo %time% & Everything.exe -reindex & start /w "" Everything.exe -update & call echo ^%time^%

18:39:16,33
18:39:24,80

For comaprison, just the writing to disk command:

Code: Select all

echo %time% & start /w "" Everything.exe -update & call echo ^%time^%

18:39:34,27
18:39:34,63

This works in both situations: Everything already running /not running.
That's because you wait for the -update to fInish, instead of waiting for the -reindex to finish (which will only happen if Everything was already running before)

[mq3]

Wonderful, this makes a lot of sense and seems good, thanks NotNull

[silverqx]

The Everything.exe -startup -reindex part will succeed immediately, and es will be called before the database rebuild command is issued.
I've added this to my "Things to fix" list.
everything -startup && timeout 1 && START /WAIT Everything.exe -startup -reindex && ES -sort size folder: size:^>1gb

I want to ask if this is fixed or I still need to use start /wait, I tested it now without start /wait, something like:

Code: Select all

Everything.exe -admin -startup && Everything.exe -startup -reindex && es.exe *.mkv

And it looks like it returns correct results.

Solution by NotNull with -update parameter and start /wait is guaranteed and works every time.

Thank you for info.

[void]

I want to ask if this is fixed or I still need to use start /wait, I tested it now without start /wait, something like:

There might still be a race condition here.

start /wait should be used.
Without the start /wait the following command:
Everything.exe -startup -reindex

may execute, but not complete before the following command is executed and completed:
es.exe *.mkv

es could return zero or out of date results depending on the current state of the Everything index.

start /wait Everything.exe -startup -reindex
will ensure Everything has started a rebuild before returning.