Custom property indexing default included filetypes

[JTCGiants56]

Hello - It seems when adding a custom property index, the default "include only files" does not include all the possible extensions.

For example, if I choose "width", which is used by pictures and videos, it only includes a portion of the extensions for these categories.

My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?

Edit: Third kind of related question: Do these custom indexes add a relatively large DB file size and\or RAM Usage load in comparison to the default indexes? Or it will generally use the same resources?

[void]

The default "include only files" is set to the extensions for files that Everything natively supports for the selected property.

My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?

Yes, if you set clear "include only files", Everything will include ALL files.

Edit: Third kind of related question: Is there a rule of thumb for setting "maximum size"? I'm not exactly sure how big these DBs can get and what will happen if the size it needs for all my files falls below the number.

Generally No, if it is a content-based property you may wish to set a limit.
I can't think of a good example, but maybe SHA256 hash for files < 1mb?

Do these custom indexes add a relatively large DB file size and\or RAM Usage load in comparison to the default indexes? Or it will generally use the same resources?

It depends on the property.

For example, if you index the length property for *.mp3, only mp3 files in your index will store this property.
The length property needs 8 bytes per file included, so if you do a search for *.mp3, multiply the number of results by 8 will give you the memory usage.

In other words, very little resources are needed.

[JTCGiants56]

The default "include only files" is set to the extensions for files that Everything natively supports for the selected property.

My other question is, if I totally wipe this line out, does that mean it will index ALL relevant filetypes?

Yes, if you set clear "include only files", Everything will include ALL files.

Hmm, so Everything will not support indexing "width" or "height" (aka resolution) for example for .WMV files which is not on this list? If this is a Windows limitation could the MediaInfo app which adds more properties than default be possibly integrated down the line?

[void]

Everything will still support width and height for wmv files, it's just not natively supported (yet).

If Everything doesn't understand width/height for a certain file, it will fall back to the Windows Property System.

You can simply add ;*.wmv to your "include only files" to include wmv files.

The default "include only files" is just a suggestion.

"include only files" is not empty by default, as indexing width for everything single file type may take a long time.
I would like to avoid using the Windows Property System by default for performance reasons.

[JTCGiants56]

Everything will still support width and height for wmv files, it's just not natively supported (yet).

If Everything doesn't understand width/height for a certain file, it will fall back to the Windows Property System.

You can simply add ;*.wmv to your "include only files".

The default "include only files" is just a suggestion.

"include only files" is not empty by default, as indexing width for everything single file type may take a long time.
I would like to avoid using the Windows Property System by default for performance reasons.

Ah okay gotcha. I'm sure you've heard of Icaros\Mediainfo which adds properties to filetypes (video bitrate etc) to filetypes that Windows does not show natively. I have no idea if it has an API or whatever but I believe this extra info would be helpful.

[void]

Thanks for the suggestions.

A plugin for Mediainfo might be added in a future version of Everything.

Everything will offer native support for common file types.

[void]

File types known to Everything

[JTCGiants56]

Sounds good. Thanks for the Info.

It's currently taking several hours to index the properties of my videos\pictures for width\height. Enabled fast sort too. I have a fairly fast pc as well. Is this normal or it might be because I included several non-officially supported filetypes?

[void]

You can check the indexing progress from Tools -> Options -> Properties.

Is there a filename shown in the top right corner?

Is this filename changing or stuck?

[JTCGiants56]

You can check the indexing progress from Tools -> Options -> Properties.

Is there a filename shown in the top right corner?

Is this filename changing or stuck?

Yep, it's moving. I guess I just have a lot of files.

[JTCGiants56]

I have remove offline drives set. I accidentally turned off 2 data drives and everything had to re-index all my properties for the two which took a bit of time. I'm guessing there's no way around this except for not removing offline drives?

I also have 3 properties being indexed. Is this normal for the Mem usage to be this much higher than 1.4?

[void]

Everything should not be rescanning properties when adding/removing a volume to your index.

Could you please let me know what is shown in your Tools -> Debug -> Statistics -> Last Rebuild Reason?

High memory usage

Could you please send me your statistics from Tools -> Debug -> Statistics?

You should expect to see about 8 bytes per file per property.
For example: Length for 1 million files will need about 8MB.

Are you using content indexing?

[JTCGiants56]

Everything should not be rescanning properties when adding/removing a volume to your index.

Could you please let me know what is shown in your Tools -> Debug -> Statistics -> Last Rebuild Reason?

High memory usage

Could you please send me your statistics from Tools -> Debug -> Statistics?

You should expect to see about 8 bytes per file per property.
For example: Length for 1 million files will need about 8MB.

Are you using content indexing?

Last rebuild reason: Add NTFS volume.

I will send over an email. Content indexing is disabled.

Total Objects: 11,449,366 - Property indexing only set to several video file types (some not officially supported by 1.5 like .wmv)

[void]

Last rebuild reason: Add NTFS volume.

Everything should be reusing the existing properties.
I'll look into this some more..

What properties are you indexing under Tools -> Options -> Properties?

[JTCGiants56]

Last rebuild reason: Add NTFS volume.

Everything should be reusing the existing properties.
I'll look into this some more..

What properties are you indexing under Tools -> Options -> Properties?

Just want to reiterate I do have "remove offline volumes" checked. Does it keep the existing properties for these files in the DB even if a drive is removed temporarily?

[void]

Thanks for the info JTCGiants56,

I'm still looking into the high memory usage issue.
There might be a memory leak that I haven't been able to track down yet..

For 11million files, with those 3 properties, I would expect to see about 1364 MB of memory usage.

~100 MB per million files.
+ ~24 MB per million files for the 3 properties (for the indexed-property-included-files only).

Without indexing properties you should see about 1100 MB of memory usage (the same as Everything 1.4)

Everything 1.5.0.1247 improves memory usage accuracy in Tools -> Debug -> Statistics.
Is Total size close to Everything's memory usage? -if not it may indicate a memory leak.
Does Everything's memory usage grow over time?

Just want to reiterate I do have "remove offline volumes" checked.
Does it keep the existing properties for these files in the DB even if a drive is removed temporarily?

These properties are removed from your index when the volume is removed.

Please try removing the Media created property from your index (from Tools -> Options -> Properties).
This is a Windows Property System property.
Everything will look for Windows Property System properties in every single file by default.

For an alternative, please try the Date encoded Everything property. (It's the same property)

[JTCGiants56]

Thanks for the info JTCGiants56,

I'm still looking into the high memory usage issue.
There might be a memory leak that I haven't been able to track down yet..

For 11million files, with those 3 properties, I would expect to see about 1364 MB of memory usage.

~100 MB per million files.
+ ~24 MB per million files for the 3 properties.

Without indexing properties you should see about 1100 MB of memory usage (the same as Everything 1.4)

Everything 1.5.0.1247 improves memory usage accuracy in Tools -> Debug -> Statistics.
Is Total size close to Everything's memory usage? -if not it may indicate a memory leak.
Does Everything's memory usage grow over time?

Just want to reiterate I do have "remove offline volumes" checked.
Does it keep the existing properties for these files in the DB even if a drive is removed temporarily?

These properties are removed from your index when the volume is removed.

Please try removing the Media created property from your index (from Tools -> Options -> Properties).
This is a Windows Property System property.
Everything will look for Windows Property System properties in every single file by default.

For an alternative, please try the Date encoded Everything property. (It's the same property)

Found the discrepancy. I've excluded image files in 1.5, just like I have in 1.4 and my object count dropped to 1.7M. I'm also on Everything 1.5.0.1247 now.

Now mem usage is almost exact.

[JTCGiants56]

Weird, it started crashing at startup for me after doing the above. I was able to quickly press force rebuild and now its reindexing properties. I'll try to enable debugging if it happens again.

[void]

Thanks for the crash report.

If you see a crash dialog, please send me a Mini crash dump.

[JTCGiants56]

Thanks for the crash report.

If you see a crash dialog, please send me a Mini crash dump.

It's crashing again, but I'm unable to create a dump since it crashes immediately at startup.

I tried going back several beta versions (which worked before), and they are crashing now as well.

So I think It may be related to the excludes I added after installing 1247a.

Code: Select all

*.jpg;*.lnk;*.jpeg;*.torrent;*.symlink;*.png

Wish I could keep it open long enough to try removing the excludes and see if the crashing stops.

[void]

Thank you for the crash report JTCGiants56,

Was there a crash dialog?

Please try the latest Everything 1.5 Alpha.

Version 1.5.0.1248 fixes a crash when a renamed file would be excluded.

Does the crash persist?

[JTCGiants56]

Thank you for the crash report JTCGiants56,

Was there a crash dialog?

Please try the latest Everything 1.5 Alpha.

Version 1.5.0.1248 fixes a crash when a renamed file would be excluded.

Does the crash persist?

No crash dialog.

No crash yet since installing this new version.

Thanks.