SDK Question

[Janus]

A question for void since I was unable to PM.

I am using the SDK with a currently private fork of explorer++.
So far I am using it for replacing the foldersize calculations, and will be adding more later.
However, folders with large file counts are still taking quite a while.
I have 8M+ files in my projects drive, it takes a moment.

Is it Okay if I recompile a modified everything32/64.dll from the SDK with a couple added functions, then add it to the downloads of my explorer++ fork?
Sources included of course.


Janus.

[void]

Yes, that is fine.

The SDK uses the MIT license.

Code: Select all

Copyright (C) 2016 David Carpenter

Permission is hereby granted, free of charge, 
to any person obtaining a copy of this software 
and associated documentation files (the "Software"), 
to deal in the Software without restriction, 
including without limitation the rights to use, 
copy, modify, merge, publish, distribute, sublicense, 
and/or sell copies of the Software, and to permit 
persons to whom the Software is furnished to do so, 
subject to the following conditions:

The above copyright notice and this permission notice shall be 
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, 
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES 
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, 
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, 
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Is there something I can change in the SDK to make it faster for you?

[Janus]

I actually do not at this moment have an answer.

Right now I am making a query, and getting back a number that more or less represents the total number of results.
Results being files and directories.
I then have to loop through according the result count.
At each step, I check to see if it is a directory, or a file.
If it is a Directory, I increment that count.
If it is a file, I increment that count, and add to the filesize total.
Similar to what I did alphabetizing the loading of addons for celestia in my fork.

A note on my goal here.

I am working on adding a couple of columns to explorer++ which I can use, but others likely won't, though I have been wrong before.

Instead of just foldersize, I also want subfoldercount & subfolderfilecount as well.
These are basically the same as the context menu 'properties' gives you, but in columnar format.
They are not something I will want all the time, but having them available will save me steps when I am shuffling directory trees.

Right now I have to call the loop from inside Explorer++, which adds some overhead.
I am hoping that moving that loop to the DLL, will reduce the overhead.

I return a record with elements I need pretallied.
I then copy the totals to the columns.
More or less like this.
Apologies for the pseudocode, but as I have said, I am not a real C/C++ programmer, I prefer pascal.


folderprops:record
foldercount : uint32;
filecount : uint32;
filesize :uint64;
end;

folderprops GetFolderProps( string FolderPath)
{
folderprops ThisFolder;

Count the files/folders and tabulate them;

ThisFolder.foldercount = foldercount;
ThisFolder,filecount = filecount;
ThisFolder.filesize = tiotalfilesize;

return ThisFolder;
}

Added to the DLL so a single call is made to it, and it calls the everything service/program, saving some overhead I hope.

This is in part to by me time until 1.5 comes out because you had indicated you were considering making these totals available via call, rather than having to brute force and loop through everything from the outside.
If you do include it, then a simple update fixes it, if you do not, then I am prepared.


Janus.

[void]

What about using the folder size in Everything? rather than calculating this yourself.

You can call Everything_GetResultSize on a folder result to get the folder size.

Note: you will need to enable folder size indexing in Everything:

• In Everything, from the Tools menu, click Options.
• Click the Indexes tab on the left.
• Check Index folder size.
• Click OK.

[Janus]

The result I got back from that did not correspond to the total size of the folder.

This is what I ended up doing to get the result I wanted.
This is ugly, but it just works, yet it is still faster than HDD/Network access, especially when large numbers of files are involved.
Fair warning though, I am not a real C/C++ programmer, so this may offend purists.

This is in \explorer++\helper\foldersize.cpp in case you want to look at the project.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
	TCHAR			InitialPath[MAX_PATH + 2];

	DWORD			LastEvError;
	LARGE_INTEGER	MatchSize;
	ULONGLONG		MatchSizeU;
	DWORD			TotResults;
	DWORD			TotIndex;


	StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));
	StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
	StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

	Everything_SetSearch(InitialPath);

	//EVERYTHING_REQUEST_SIZE                                 (0x00000010)
	Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE);

	Everything_QueryW(TRUE);

	LastEvError = Everything_GetLastError();
	TotResults = Everything_GetTotResults();

	MatchSizeU	= 0;
	if(TotResults == 0) 
		{
		lTotalFolderSize->QuadPart = MatchSizeU;
		return S_OK;
		}


	for (TotIndex = 0; TotIndex < TotResults; TotIndex++)
		{
		Everything_GetResultSize(TotIndex, &MatchSize);

		if (Everything_IsFileResult(TotIndex))
			{
			MatchSizeU = MatchSizeU + MatchSize.QuadPart;
			*nFiles++;
			}
		else
			{
			*nFolders++;
			}
		}

	lTotalFolderSize->QuadPart = MatchSizeU;
		
	return S_OK;
}

I hope this shows the hoops I am jumping through, and I am hoping there are fewer hoops I can use that will still get me the same result.

A couple of additions would be nice.
Everything_GetResultFileCount, Everything_GetResultFolderCount, and Everything_GetResultTotalFileSize would be greatly appreciated.
Or simply make the result database slice returned via in a predefined format so it can be parsed directly without the need for repeated calls.

Perhaps

FileRecord:
Uint32 FoldernameIndex;
strZ Filename;
Uint64 FileSize;
end;

Maybe include a count in a header.
The format doesn't matter all that much, as long as it is documented.
Though being able to directly grab the needed bits with a call would be great.

Janus.

[void]

Please try the following:

Code: Select all

HRESULT GetFolderSize(TCHAR *szPath,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR         InitialPath[MAX_PATH + 11];
   DWORD         LastEvError;
   LARGE_INTEGER MatchSize;

   // request filelist:"<filename>"
   // this will return the file matching the exact filename.
   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("filelist:\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

   Everything_SetSearch(InitialPath);
   
   //EVERYTHING_REQUEST_SIZE                                 (0x00000010)
   Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE);

   Everything_QueryW(TRUE);

   LastEvError = Everything_GetLastError();

   if (Everything_GetResultSize(0, &MatchSize))
   {
      lTotalFolderSize->QuadPart = MatchSize.QuadPart;
   }
   else
   {
      lTotalFolderSize->QuadPart = 0;
   }
      
   return S_OK;
}

Note the use of the new query:
filelist:"<filename>"

This will find the exact folder by filename.

This does not calculate nFolders or nFiles. Are these values still needed?

Everything_GetResultFileCount
Everything_GetResultFolderCount

There is Everything_GetNumFolderResults and Everything_GetNumFileResults.
However, these functions are no longer work with EVERYTHING_REQUEST_SIZE.
I'll consider adding support for these functions with EVERYTHING_REQUEST_SIZE.

Everything_GetResultTotalFileSize

Added to my TODO list.

[Janus]

Please try the following:

Note the use of the new query:
filelist:"<filename>"

This will find the exact folder by filename.

This does not calculate nFolders or nFiles. Are these values still needed?

Everything_GetResultFileCount
Everything_GetResultFolderCount

There is Everything_GetNumFolderResults and Everything_GetNumFileResults.
However, these functions are no longer work with EVERYTHING_REQUEST_SIZE.
I'll consider adding support for these functions with EVERYTHING_REQUEST_SIZE.

Everything_GetResultTotalFileSize

Added to my TODO list.

I tried the sample code.
Had to match up var names and parameters, but it worked.
Same issues as with mine.
99% cpu usage, and one or two seconds per folder to get size.

I also tried with file: as well.
Same basic result, though it was a little faster, even having to loop through it, and cpu usage stayed at 90% instead of 99%.

Not sure why the difference in return times between the gui with it, and an external call.
It would be nice however, to see that speed outside, or understand what it is from.
If I type in a drive letter 'C:\', it only takes a second to fill in.
If I highlight the 'C' then change it to a 'D', again, it only takes a second to fill it all in.
I was hoping for that sort of speed, or at least with in a magnitude of it.


Janus.

[NotNull]

There might be a completely different approach to get your results faster:

Just (?) do 2 extra Everything queries: c:\windows folder: and c:\windows file:
Those return the total number of results (= #folders/#files) instantaneous (the GUI displays them in the status bar).

EDIT: That should be: c:\windows\ folder: (added a \), to exclude the C:\windows folder itself and to get just the subfolders. And to prevent matching a possible "c:\windows settings" folder.

[Janus]

@NotNull

Everything gui response speed is awesome.
My problem is when doing a query from an outside source.

Right now to get the total file size of a query return, it is taking a long time.
I am calling the Everyting DLL from within explorer++ using a quoted path.
I then have to either iterate through the list, or use filelist which self iterates, either of which is magnitudes slower than the native gui is.

The filecount & foldercount are for the future, my only real annoyance is how long it takes to do the redundant calls to get totalfilesize of query result.
Making those two available in explorer++ is part of me making file/folder counts within archives available as columns.
After I have those added, then I am going to add the ability to mount archives as temporary drive letters.
That way they can be worked with easier, but do not pollute the dir tree in the left pane.

I am also hoping to use everything with Reactos when it goes beta, and I can drop M$ windows completely, using the opensource version instead.


Janus.

[NotNull]

OK, I thought you wanted the file/foldercount as well, as you said:

I then have to loop through according the result count.
At each step, I check to see if it is a directory, or a file.
If it is a Directory, I increment that count.
If it is a file, I increment that count, and add to the filesize total.

Total size should be trivial: let Everything index that (Menu:File > Options > Indexes: enable Index Folder size
(like @void already suggested)


What I would do: (and I'm not a programmer, so be aware of stupid ideas)

- If these results don't match up with the reults you get by stepping through the folders, analyze the differences. (probably hardlinks/ symlinks/etc). Maybe the Everything results are even better for your case? Or find a fast way to get the "different files" and measure those seperately. Add the result to the total.

- WizTree also has an MFT based file/folder count and -size column. Do these results match your's or Everything's? If they match your's: it should be doable in some way (WizTree is also very fast)

- I could live with a filemanager that doesn't show a 100% accurate foldersize (somewhere else on these forums was a discussion about actual filesize. TL;DR: you just don't know). Especially if the alternative would mean a slow filemanager.

[void]

99% cpu usage, and one or two seconds per folder to get size.

Are you calling this for one folder? or for many results?
Each GetFolderSize call will take about ~0.1 seconds.
I would expect about 1-2 seconds with 99% cpu usage for 20 calls to GetFolderSize.

A better solution will be to do a Everything query with filelist:<filename1;filename2;filename3;...> for all your results.
And then researching inside those results for the folder size.

There needs to be a single call, like the Everything_GetRunCountFromFileName but for sizes, and dates. This would bypass the query and just get the information directly from the Everything database. Added to my TODO list.

[Janus]

@void

In explorer++ using the code I posted, or the adapted code you had me try.
Each folder still takes ~0.5-1.0 seconds to get a size, then the next gets the size set.
Twenty subfolders showing in detail view will take ~10-20 seconds to fill in.
More likely toward the 10, but 20 happens sometimes.

I can post timing samples from the debug window for three variations tomorrow to show what I mean in more detail.

Most of them however, look about like this.

• IPC: execute query: filelist:"C:\Windows"
  search 'filelist:"C:\Windows"' filter '' sort 0
  IPC: wait 2
  term 0078e0d0, flags: 3140, next: 00000000, notnext: 00000000
  folderop: 385, fileop: 385, term: filelist:C:\Windows
  SET SORT 0
  found 1 folders, size 65540, db search time taken: 0.060724 seconds
  found 0 files, size 0, db search time taken: 0.580731 seconds
  set sort 0 1
  already sorted
  finished sort, time taken 0.001389 seconds
  update selection 0.000003 seconds
  DB_WAIT: _db_ready_proc waiting...
  DB_WAIT: _db_ready_proc waited 0.000684 seconds
  when ready 11 0000002d 00000000
  when ready 11 0000002d 00000000
  SIZE CHECK 36 36 diff 0 count 1
  IPC: query complete: 4294967295 max results, offset 0, reply hwnd 12911706
  IPC: watch awake
  IPC: Client disconnected.

or

• IPC: execute query: filelist:"C:\$RECYCLE.BIN"
  search 'filelist:"C:\$RECYCLE.BIN"' filter '' sort 0
  IPC: wait 2
  term 04c483d8, flags: 3140, next: 00000000, notnext: 00000000
  folderop: 385, fileop: 385, term: filelist:C:\$RECYCLE.BIN
  SET SORT 0
  found 1 folders, size 65540, db search time taken: 0.060313 seconds
  found 0 files, size 0, db search time taken: 0.566090 seconds
  set sort 0 1
  already sorted
  finished sort, time taken 0.001465 seconds
  update selection 0.000002 seconds
  DB_WAIT: _db_ready_proc waiting...
  DB_WAIT: _db_ready_proc waited 0.000693 seconds
  when ready 11 0000002d 00000000
  when ready 11 0000002d 00000000
  SIZE CHECK 36 36 diff 0 count 1
  IPC: query complete: 4294967295 max results, offset 0, reply hwnd 15598682
  IPC: watch awake
  IPC: Client disconnected.

When using

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
   TCHAR         InitialPath[MAX_PATH + 11];
   DWORD         LastEvError;
   DWORD         TotIndex;

	LARGE_INTEGER	MatchSize;

   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("filelist:\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

   Everything_SetSearch(InitialPath);
   
   //EVERYTHING_REQUEST_SIZE                                 (0x00000010)
   Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE);

   Everything_QueryW(TRUE);                                                                  

   LastEvError = Everything_GetLastError();

   if (Everything_GetResultSize(0, &MatchSize))
   {
      lTotalFolderSize->QuadPart = MatchSize.QuadPart;
   }
   else
   {
      lTotalFolderSize->QuadPart = 0;
   }
   return S_OK;
}

They are all about like this.
A few are shorter timed, and quite a few are longer.
During that time CPU usage hits ~95-99%

I can compile in debug mode and put it up with symbols if you want to do a profile.
However, the taskmanager clearly shows the high usage happening in everything.

Normal usage of the everything gui does not show any spike.


Janus.

[void]

Thanks for the debug info.
Looks like you have many files.

The main reason for the slowness is Everything is doing a full query for each CalculateFolderSize call.
The full query is completely unnecessary. Unfortunately, I don't have a good solution until Everything 1.5, I will have to implement new API calls for this specific folder size request.

For now, please try searching for:

Code: Select all

   
   TCHAR         InitialPath[MAX_PATH + 24];

   StringCchCopy(InitialPath,SIZEOF_ARRAY(InitialPath),_T("folder: count:1 exact:\""));
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),szPath);
   StringCchCat(InitialPath,SIZEOF_ARRAY(InitialPath),_T("\""));

folder: will only search for folders. This will help performance as the file database is not touched.
count:1 will only find one result, in some cases this will make the query slower as it limits the search to one thread, however here it should provide a good performance increase.
exact: is similar to filelist:, although slightly faster.

For the best results when you want folder sizes for multiple folders, perform only one Everything query with filelist:"filename1";"filename2";"filename3";...

[Janus]

A different approach then.

Since this is being used in a file explorer, it will always be wanting the size of what is to it, a subfolder.
Can I reach upwards to get a wider range of downward content.

For instance, I am browsing C:\
Explorer++ will then request the size of C:\Windows then C:\$Recycle.bin and so forth.

In calculatefoldersize, I have a static variable that is the last query.
When called, it immediately trims a layer off the path.
C:\Windows becomes C:\ for example.
It then compares the new path to the last querypath.
If they match it then loops through the results until the name matches the trimmed name.
In this case it would mean looping through the results of C:\ search for a folder names Windows, than getting its size.

Or I could simply copy the last results to a static cast dynamic array, and if the parent is the same, walk through the array.

Could I use a serch such as 'parent:C: folder:' on a query.
Then loop through the results multiple times?

Or can I do a query on the parent folder of the request, that will give the size of all the subfolders of the parent.
Allowing me to loop through the siblings looking for the one I want?
Then keep that list until the directory is done being looped through?


Janus.

[Janus]

Okay, really stupid question here.

I have been trying to take the passed path from explorer++.
C:\Windows for instance.
Get the foldername, Windows in this case.
The parentfolder, C:\ in this case.
Construct a querystring "parent:C:\ folder:", for instance.
Compare that string with Everything_GetSearch().
If they match, use the existing results, if not, do a fresh query.
This should avoid hitting the database so much.

I do not know if it will however, since windows is completely bonkers on string handling.

I found functions to give me the last dir name no problem, and the parent directory name, no problem.

However, thay all want strings in so many different formats that make no sense at all.
TCHAR, LPWSTR, LPCWSTR, And many more.
That leaves the STRSAFE_ fore labelled garbage out of it.

If I try to just use a = b with the strings, a ends up pointing to the same memory location as b, not with a copy, which is insane.
I watched it in debug from VS, incredible.
If I copy the string, which was fun since it seems there is no almost no way to copy between string types.
Then I get told that such and such is uninitialized, so it throws an exception.
Yet when I try to give an initial value, I get 0xc00005 access violations, after being told that TCHAR is not the same as TCHAR[MAX_PATH], and cannot be converted.
Then there is TCHAR vs LPCWSTR vs LPWSTR vs STRSAFE_LPWSTR vs who knows how many different types.

Is there a sane guide out there that shows what formats are what?
One that does not use jargon.
One that just says, here is what it is, and why?
One that tells me why a text in quotes is uncastable/unusable as LPCWSTR or whatever gibberish it spouted this time, so I can concatenate a simple string to use for a query..

And why does it seem like the calls that naturally go together, all require different types of strings that require more work to convert than the job I am trying to do in the first place.
If I do a memory dump, they are the same, so what is the problem, the only difference is in the stupid labels.

All I want is ParentFolder, FolderName, and a way to compare the name of the indexed item for a match so I can use its size.

Why does C/C++ make that so bloody hard?

Just ranting, not really expecting any sanity out of C/C++, but still holding out hope.


Janus.

[Janus]

Okay, that was fun ... NOT!!!

I brute forced my way through and made a history check version of the code.

This piece of code works with the idea that I am checking a large number of subdirectories in a folder sequentially.
It derives the name of its parent directory.
It then compares it with the last query done.
If they match, then those results are checked, elsewise, a query is done, and those results are looped through.

My logic behind this was hoping that the subdirctories would have their size as part of the results, they don't.

However, here is the code if someone might find a use for it.

As always, please remember, I am not a real C/C++ programmer.
This code is messy, and brute force, and unoptimized.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
	TCHAR			InitialPath[MAX_PATH + 2];
	TCHAR	BasePath[MAX_PATH]		= L"";
	TCHAR	MatchFolder[MAX_PATH]	;
	TCHAR	MatchName[MAX_PATH]		;
	TCHAR	SearchFolder[MAX_PATH]	= L"";
	TCHAR	MatchSearch[MAX_PATH]	;
	TCHAR Match_Search[MAX_PATH];


	DWORD			LastEvError;
	LARGE_INTEGER	MatchSize;
	ULONGLONG		MatchSizeU;
	DWORD			TotResults;
	DWORD			TotIndex;

	DWORD SizeError = 0;

	StringCchCopy(BasePath, MAX_PATH, szPath);
	StringCchCopy(MatchName, MAX_PATH, BasePath);


	PathRemoveFileSpec(BasePath);

	StringCchCopy(InitialPath,MAX_PATH,_T("parent:\""));
	StringCchCat(InitialPath, MAX_PATH ,BasePath);
	
	StringCchCat(InitialPath,MAX_PATH, _T("\" folder:"));

	StringCchCopy(MatchSearch,MAX_PATH, Everything_GetSearch());

	if (wcscmp(MatchSearch, InitialPath) != 0)
		{

		Everything_SetSearch(InitialPath);
		Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE || EVERYTHING_REQUEST_FILE_NAME);
		Everything_QueryW(TRUE);
		LastEvError = Everything_GetLastError();
		}

	TotResults = Everything_GetTotResults();
			SizeError = Everything_GetLastError();

	MatchSizeU	= 0;
	if(TotResults == 0) 
		{
		lTotalFolderSize->QuadPart = MatchSizeU;
		return S_OK;
		}

	lTotalFolderSize->QuadPart = 0;

	for (TotIndex = 0; TotIndex < TotResults; TotIndex++)
		{
		StringCchCopy(SearchFolder, MAX_PATH, BasePath);
		StringCchCat(SearchFolder, MAX_PATH, L"\\");
		StringCchCat(SearchFolder, MAX_PATH, Everything_GetResultFileName(TotIndex));
		SizeError = Everything_GetLastError();

		if (wcscmp(SearchFolder, MatchName) == 0)
			{
			SizeError = Everything_GetResultSize(TotIndex, &MatchSize);
			SizeError = Everything_GetLastError();
				MatchSizeU = MatchSize.QuadPart;
			}
		}

	lTotalFolderSize->QuadPart = MatchSizeU;
		
	return S_OK;
}

So, now on to my next idea.


Janus.

[Janus]

Okay, here is my next attempt.

It generates a parent directory query, then checks to see if the query has changed.
If it has, it updates the query, otherwise, it just proceeds.
Then it loops through the result set.

Response speed is largely determined by total number of files in query.
500K = 1-2 Seconds per directory for updates.
<100K is <1 Sec each.

This one works by comparing file paths, and only adding those in the same tree.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
{
	TCHAR	InitialPath[MAX_PATH + 2];
	TCHAR	BasePath[MAX_PATH]		= L"";
	TCHAR	MatchPath[MAX_PATH]	;
	TCHAR	MatchSearch[MAX_PATH]	;
	TCHAR	FilePath[MAX_PATH]		= L"";
	TCHAR	File_Name[MAX_PATH]		= L"";


	DWORD			SearchOpts;
	DWORD			LastEvError;
	DWORD			TotResults;
	DWORD			TotIndex;
	LARGE_INTEGER	MatchSize;

	DWORD SizeError = 0;
	DWORD LengthCheck = 0;
	DWORD FLengthCheck = 0;
	DWORD MLengthCheck = 0;
	DWORD MFLengthCheck = 0;
	DWORD FileCount = 0;

	StringCchCopy(BasePath, MAX_PATH, szPath);
	StringCchCopy(MatchPath, MAX_PATH, BasePath);

	PathRemoveFileSpec(BasePath);

	StringCchCopy(InitialPath,MAX_PATH,_T("\""));
	StringCchCat(InitialPath, MAX_PATH ,BasePath);
	
	StringCchCat(InitialPath,MAX_PATH, _T("\" file:"));

	StringCchCopy(MatchSearch,MAX_PATH, Everything_GetSearch());

	
	if (wcscmp(MatchSearch, InitialPath) != 0)
		{

		Everything_SetSearch(InitialPath);
		//EVERYTHING_REQUEST_FILE_NAME                            (0x00000001)
		//EVERYTHING_REQUEST_PATH                                 (0x00000002)
		//EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME              (0x00000004)
		//EVERYTHING_REQUEST_EXTENSION                            (0x00000008)
		//EVERYTHING_REQUEST_SIZE                                 (0x00000010)
		//EVERYTHING_REQUEST_DATE_CREATED                         (0x00000020)
		//EVERYTHING_REQUEST_DATE_MODIFIED                        (0x00000040)
		//EVERYTHING_REQUEST_DATE_ACCESSED                        (0x00000080)
		//EVERYTHING_REQUEST_ATTRIBUTES                           (0x00000100)
		//EVERYTHING_REQUEST_FILE_LIST_FILE_NAME                  (0x00000200)
		//EVERYTHING_REQUEST_RUN_COUNT                            (0x00000400)
		//EVERYTHING_REQUEST_DATE_RUN                             (0x00000800)
		//EVERYTHING_REQUEST_DATE_RECENTLY_CHANGED                (0x00001000)
		//EVERYTHING_REQUEST_HIGHLIGHTED_FILE_NAME                (0x00002000)
		//EVERYTHING_REQUEST_HIGHLIGHTED_PATH                     (0x00004000)
		//EVERYTHING_REQUEST_HIGHLIGHTED_FULL_PATH_AND_FILE_NAME  (0x00008000)
		SearchOpts = (EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_PATH | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);
		Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_PATH | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);
		Everything_QueryW(TRUE);
		LastEvError = Everything_GetLastError();
		}

	SearchOpts = Everything_GetResultListRequestFlags();
	TotResults = Everything_GetTotResults();
	SizeError = Everything_GetLastError();

	MatchSize.QuadPart	= 0;
	lTotalFolderSize->QuadPart = 0;
	if(TotResults == 0) 
		{
		lTotalFolderSize->QuadPart = MatchSize.QuadPart;
		return S_OK;
		}

	FileCount = 0;
	StringCchCat(MatchPath, MAX_PATH , L"\\");
	MLengthCheck = wcsspn(MatchPath, MatchPath);
	for (TotIndex = 0; TotIndex < TotResults; TotIndex++)
		{
		StringCchCopy(FilePath, MAX_PATH, Everything_GetResultPath(TotIndex));
		StringCchCat(FilePath, MAX_PATH , L"\\");

		MFLengthCheck = _tcsncmp(MatchPath, FilePath,MLengthCheck);
		FLengthCheck = wcsspn(FilePath, FilePath);
		if ((MFLengthCheck == 0))
			{
			SizeError = Everything_GetResultSize(TotIndex, &MatchSize);
			lTotalFolderSize->QuadPart = lTotalFolderSize->QuadPart +  MatchSize.QuadPart;
			//Everything_GetResultFullPathName(TotIndex, File_Name, MAX_PATH);
			FileCount++;
			FileCount--;
			FileCount++;
			}
		}
	return S_OK;
}

As always, remember, I am not a real C/C++ programmer.
I have no doubt there is a more efficient way to accomplish this, I simply don't know it.
In fact, the more I learn, and learn about C/C++, the more I come to dislike them.

Next I am going to add functions to the DLL.

Everything_MakeCache // Takes a path.
Generates query, and compares it to saved query for Cache.
If it does not match, then it does a query on the path.
Creates an array of child directory structures.
Each entry will have Name, FileCount, FolderCount, TotalFileZise, in an array.

Everything_IsParentCached //Checks to see if childname is in CacheArray.

Everything_GetEntry //Takes ChildName, checks if Cached, then returns Cache entry from mathcing index if it is.

Everything_CacheParent //Which will call makecache after deriving it from supplied path.

Everything_ForceGetEntry //Takes path, then creates Parent Cache if needed to get data.


Exact names and breakdown of function will depend on how hard all of this is.


Janus

[void]

Could I use a serch such as 'parent:C: folder:' on a query.
Then loop through the results multiple times?

Yes, that is my recommendation.

Or can I do a query on the parent folder of the request, that will give the size of all the subfolders of the parent.
Allowing me to loop through the siblings looking for the one I want?
Then keep that list until the directory is done being looped through?

Would this only return one result? I'm not sure how you would loop through any siblings this way.
Unless you searched for parent:c:|exact:c: this would include the parent folder in the results.

Is there a sane guide out there that shows what formats are what?

I don't know a guide of the top of my head..
T = ANSI or UNICODE depending on project settings
LP = Long pointer or just a pointer these days.
C = Const (you can not modify the string contents / read only, without this flag assume you can modify the contents)
W = UNICODE
A = ANSI (usually omitted)
CHAR = a single character.
STR = an array of characters.

If there is no T or W flag, assume the string is ANSI.

If you need to convert between ANSI and Unicode use MultiByteToWideChar / WideCharToMultiByte.
Make sure you use _T() around "text" when working with T strings.
Use L"Text" when working directly with W strings
Use "Text" when working directly with A strings.

Usually both A and W string functions are implemented and then T macros are used to point to either.

My logic behind this was hoping that the subdirctories would have their size as part of the results, they don't.

They should, please make sure you request size information:
Change:
Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE || EVERYTHING_REQUEST_FILE_NAME);
to:
Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_FILE_NAME);

Good luck!

[Janus]

@void

Working in DLL source.

Added a couple of items and two calls.
Take a look at this.

Here is where I am getting results that make no sense to me.
I initially tried with EVERYTHING_REQUEST_PATH, but that gave the directory the folder in question was in.
So I switched to the full path and name, which gave what I wanted.

This is part of my initial attempt, which fails to give folder size.

The point of this is to create a cache of subdirectory names and sizes to a parent.
The reason for this is in explorer++, calls for the sizes sequentially.
So the code checks if the cached directory is the same as its parent.
If not, it makes this call to update the cache.
Once the cache contains the list of the parent's children.
Which is to say the caller and its siblings.
It then looks up the caller, and returns its size.

This allows the first call for directory size to update the cache, and avoid repeated DB queries.
However, I am unable to get folder sizes, so I have added other code to get a file list.
Which I then manually add to the size of the folder where they belong.

Code: Select all

// 180220 Janus Added cachepath and cachedata;
static TCHAR _Everything_CachePath[MAX_PATH] = L" ";

typedef struct EVERYTHING_CacheData
{
	ULARGE_INTEGER FolderSize;
	TCHAR FolderName[MAX_PATH]; 
}EVERYTHING_CacheData;

static EVERYTHING_CacheData *_Everything_CacheData = NULL;

static DWORD CacheFolderCount = 0;

TCHAR BasePath;

// This is the query built from the path;
// BasePath = 'parent:"C:\\windows" folders:';
BasePath = "parent:\"C:\\windows\" folders:";
Everything_SetSearch(BasePath);
Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);
Everything_Query(TRUE);
FeedBack = Everything_GetLastError();

CacheFolderCount = Everything_GetTotResults();
CacheSize = CacheFolderCount * sizeof(EVERYTHING_CacheData);

_Everything_CacheData = _Everything_Alloc(CacheSize);

for (DWORD idx = 0;idx < CacheFolderCount; idx++)
	{
	Everything_GetResultFullPathName( idx, CacheFolderName, MAX_PATH);
	FeedBack1 = Everything_GetResultAttributes(idx);
	FeedBack2 = Everything_GetLastError();
	StringCchCopy(_Everything_CacheData[idx].FolderName, MAX_PATH, CacheFolderName);
	_Everything_CacheData[idx].FolderSize.QuadPart = 0;
	}

When run,

CacheFolderCount is correct.
_Everything_CacheData allocated without error
CacheFolderName is filled in correctly for each loop.

Then the problems start.

FeedBack1 = Everything_GetResultAttributes(idx);

Yields $FFFFFFFF

FeedBack2 = Everything_GetLastError();

Gives Error 8
#define EVERYTHING_ERROR_INVALIDREQUEST 8 // invalid request data, request data first.

Which means the rest is pointless.

My goal is take a path such as C:\windows\inf which is trimmed to its parent, C:\windows
Which is then queried to give all subdirectories, and hopefully their sizes.
I record their names and sizes so when the siblings sizes are queried, it comes from the cache, not the DB.

Doing it this way since the DB query is so time intensive.
Why not just do it once, and save the results for the rest of the calls that will only differ in the final directory name.

If there is something I have missed, please let me know.

If you need to see the whole thing, I can put it an archive on my site.
If you want context, I can include the copy of Explorer++ I am using it with.


Janus.

[Janus]

Okay, I think I have finally got it.

I need to test further, but I think I finally did it.
Response time is about a second for anywhere up to about 150 folders.
Though it did crash a couple of times, but only after several minutes of continuous movement.

I am also now of the opinion that C/C++ string handling has a gaming alignment of psychotic evil.
I joke, but it certainly feels like a way to try filling insane asylums.

Very few of the string routines, especially of the cmp variety, actually match their documentation.
Also, as for M$ string safe routines, mostly they are not.

The more I learn of C/C++, the less I like either of them.

I will post everything tomorrow if holds up under testing.


Janus.

[void]

Please try Everything_GetResultSize to get the folder size.

Everything_GetResultAttributes will fail, unless you request attributes, for example: Everything_SetRequestFlags(EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_ATTRIBUTES | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME);

This information is not usually indexed, so if you only need to check if the result is a file or folder use the Everything_IsFolderResult/Everything_IsFileResult call instead.

Did the Everything client crash?

[Janus]

Okay.

I warn you, this code is messy.
This was structured to make single stepping easier.
I have done no real optimization yet.

The first addition is to everything32.def appended to the end.

Code: Select all

	Everything_GetFolderSize
	Everything_FolderSizeCache

The second is to everything.h

Code: Select all

EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_GetFolderSize(LPCWSTR folderpath, LARGE_INTEGER foldersize );
EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_FolderSizeCache(LPCWSTR cachepath );

The third is in everything.c in two sections.

First:

Code: Select all

// Added to the declarations at the top of the file.

static TCHAR _Everything_CachePath[MAX_PATH] = L" ";

typedef struct EVERYTHING_CacheData
{
	ULARGE_INTEGER FolderSize;
	TCHAR FolderName[MAX_PATH]; 
}EVERYTHING_CacheData;

static EVERYTHING_CacheData *_Everything_CacheData = NULL;
static DWORD CacheFolderCount = 0;

Second: Just after _Everything_window_proc

Code: Select all

EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_FolderSizeCache(LPCWSTR cachepath ) //Janus testing;
	{
	// Record the path in cachepath;

	// Do search with parent:path folders: to get folder list;
	// allocate cache based on number of folders;
	// Record folder paths in _Everything_CacheData[index]->FolderName;

	TCHAR	BasePath[MAX_PATH];//		= {""};
	TCHAR	CacheFolderName[MAX_PATH];

	DWORD	ItemCount = 0;
	DWORD	CacheSize = 0;
	DWORD	ItemDex = 0;
	DWORD	CachePathLen = 0;
	DWORD	ResultPathLen = 0;
	DWORD	CRPathLen = 0;
	DWORD	FeedBack = 0;
	DWORD	Attribs = 0;
	DWORD	LastError = 0;

	LARGE_INTEGER RCSize;	

	_Everything_Free(_Everything_CacheData);

	StringCchCopy(_Everything_CachePath, MAX_PATH, cachepath);

	StringCchCopy(BasePath,MAX_PATH, L"parent:\"");
	//StringCchCopy(BasePath,MAX_PATH, L"\"");
	StringCchCat(BasePath, MAX_PATH, _Everything_CachePath);
	StringCchCat(BasePath,MAX_PATH, L"\" folders:");

	Everything_SetSearch(BasePath);
	Attribs = EVERYTHING_REQUEST_SIZE | EVERYTHING_REQUEST_FULL_PATH_AND_FILE_NAME;
	Everything_SetRequestFlags(Attribs);
	Everything_Query(TRUE);
	LastError = Everything_GetLastError();

	CacheFolderCount = Everything_GetTotResults();
	CacheSize = CacheFolderCount * sizeof(EVERYTHING_CacheData);

	_Everything_CacheData = _Everything_Alloc(CacheSize);
	for (DWORD idx = 0;idx < CacheFolderCount; idx++)
		{
		if (Everything_IsFolderResult(idx))
			{
			Attribs = 0;
			Everything_GetResultFullPathName( idx, CacheFolderName, MAX_PATH);
			LastError = Everything_GetLastError();
			StringCchCopy(_Everything_CacheData[idx].FolderName, MAX_PATH, CacheFolderName);
			FeedBack = Everything_GetResultSize(idx, &RCSize);
			_Everything_CacheData[idx].FolderSize.QuadPart = RCSize.QuadPart;
			}
		}
	}



EVERYTHINGUSERAPI void EVERYTHINGAPI Everything_GetFolderSize(LPCWSTR folderpath, PULARGE_INTEGER foldersize ) //Janus testing;
	{

	TCHAR	CachePath[MAX_PATH];	
	TCHAR	ParentPath[MAX_PATH];	
	TCHAR	FolderPath[MAX_PATH];	
	TCHAR	CacheHit[MAX_PATH];		

	DWORD	ItemDex = 0;
	DWORD	ItemCmp = 0;

	StringCchCopy(CachePath, MAX_PATH, _Everything_CachePath);
	StringCchCopy(FolderPath, MAX_PATH, folderpath);
	StringCchCopy(ParentPath, MAX_PATH, folderpath);
	PathRemoveFileSpec(ParentPath);
	
	if (lstrcmpi(CachePath, ParentPath) != 0)
		{
		Everything_FolderSizeCache(&ParentPath );
		}

	StringCchCopy(CachePath, MAX_PATH, _Everything_CachePath);
	StringCchCat(CachePath,MAX_PATH, L"\\");

	ItemDex = 0;
	while (ItemDex < CacheFolderCount)
		{
		StringCchCopy(CacheHit, MAX_PATH, _Everything_CacheData[ItemDex].FolderName);
		ItemCmp = lstrcmpi(CacheHit, FolderPath);
		if (lstrcmpi(CacheHit, FolderPath) == 0)
			{
			foldersize->QuadPart = _Everything_CacheData[ItemDex].FolderSize.QuadPart;
			ItemDex = CacheFolderCount;
			}
		ItemDex++;
		}
	}

All of the above is called from CalculateFolderSize inside of /explorer++/helper/foldersize.cpp which is part of the explorer++ project.
That function is called for each subfolder within a directory, serially.
Which is why I made the cache.
The call to getfoldersize checks if its parent is the cache base, if not, it updates the cache.
Cache miss is low because of the nature of the calls.
The first in a series will be a miss, followed by however many, that are cache hits.
Since only the first one with a common parent hits the DB, very low penalty.

Code: Select all

HRESULT CalculateFolderSize(TCHAR *szPath,int *nFolders,
int *nFiles,PULARGE_INTEGER lTotalFolderSize)
	{
	TCHAR folderpath[MAX_PATH];
	StringCchCopy(folderpath, MAX_PATH, szPath);
	Everything_GetFolderSize(szPath, lTotalFolderSize);
	return S_OK;
	}

I have also restructured the SLN to properly name everything32.lib, pdb & dll as part of the build.

Still a work in progress.

Compiler is VS2013 with MBCS installed, but I am not sure if the latter is used.
Many of my problems were string issues, which required single stepping and memory dumps to analyze.

@void
If you wish to use any part of this, either directly or as a guide of any sort.
You are free to do so.
I have tried to follow the pattern of how the rest is done, I hope I have done so.

Once I clean up the code and test it fully, in x86 & x64 both, I will be posting it all on my site.
Though if you implement equivalent functionality, I will use it instead.

Thank you for your patience and help both.
Also, thank you for a very helpful and useful program.

I can post binaries for testing if you like.


Janus.

[Janus]

@Void

I am working on cleaning up my code now that I finished a short paying interruption to my hobbies.
I have however run into a question.

Is there a way to determine if a path is in the index?

I use everything_getbuildversion to determine if it is loaded and running.
I have not however, been able to find a way to tell an empty directory, from one that is not in the index.

I have fat32 drives, and once I finish my vdi {virtualbox dynamic allocation drive} extension to the ext2ifs driver, the occasional ext2/3/4 drives as well.
The purpose is to allow me to mount virtualbox drives as local, and most of them are ext2, though there are a few ntfs and fat32 as well.

If there is some way the ext2ifs driver can be extended to allow everything to work with it, that would be great as well.

For now however, I am hoping there is a way see if something is in the index that I have missed.
Also, is there a 'busy' signal for when it is reindexing that I can check for?

Janus.

[void]

Please post a link to your site.

Is there a way to determine if a path is in the index?

Not quickly, I need to add a SDK function: Everything_FileExists(filename) / Everything_FolderExists(filename)
For now, you could search for the path with filelist:filename and check if you get a result.

ext2/3/4 support is on my TODO list.

Also, is there a 'busy' signal for when it is reindexing that I can check for?

There's no event, you would need to poll for a ready state:
check if the index is loaded EVERYTHING_IPC_IS_DB_LOADED. This will return FALSE when the index is being loaded or reindex.
To check if Everything is "busy" use the EVERYTHING_IPC_IS_DB_BUSY call.

[Janus]

@Void

My site is really pathetic, I am not a web designer, and I prefer tables to css.
I have a forum and a blog for my personal fork of Celestia, and that is about it.
Been thinking about reorganizing things since I am making several personal forks of programs.

Here is my modified explorer++ and tweaked EverythingSDK.

http://www.simulatorlabbs.com/forks/

I will clean up stuff and organize it later.
The SDK has been brought up to a VS2013 sln/vcxproj set.
It now makes the everything32.dll & everything64.dll files directly.

I have tried to follow the pattern you set for the SDK.
However, if you decide to keep/use the functions I added, please feel free to customize them as needed.

The explorer++ is in x86 & x64 both, as are the everything dlls.
It now uses the isdbloaded to check if everything is loaded or busy.
If everything is not loaded, or it is busy, it falls back to the old code.

I still have some tweaks to do, and then I will be putting the code I am using for explorer++ up as well.
However, I am still eliminating some irritants.
StrCmpLogicalW is being replaced by StrCmp so it will sort predictably.
I killed the double click on empty space to go up a level.
I replaced the file size 'bytes' with 'B' since I keep details set to show size in bytes.

Also, a question since you know file stuff better than I do.
Can netserverenum and netshareenum be sued to auto scan for shares on a lan?
If so, can everything add the ability to scan the lan for them, with perhaps a hint function?
My thought is it might be a way to lighten setting up lan indexing.
Perhaps allow the ETP to do an auto search over the lan to reduce local DB size if the user chooses.

Just rambling at this point.

I hope the explorer++ with everything integrated works as well for you as it does for me.
I will be posting it on the explorer++ forum as well.


Janus.

[Janus]

Okay, I have now uploaded the explorer++ source.

explorerplusplus-180302.zip

It is setup to use VS2013, Boost 1.64, Pantheios 1.01 Beta16 & Stlsoft 1.9.124 which I am using.

Environmental variables used as follows
Boost, Panthieos, StlSoft.

I will be updating with the last few changes from github for explorer++ in the next few days.
I will post new binaries and archives when I do.


Janus.

[void]

Thanks for sharing your site.

Can netserverenum and netshareenum be sued to auto scan for shares on a lan?

I've never used these calls before, but they appear to be the ones you want for scanning LAN shares.

I don't have immediate plans for crawling the LAN for shares, for Everything 1.5 I do have plans for temporary indexes, for example when you start typing \\server\ Everything will index only the folders and files directly in \\server\
Everything could then start auto suggesting subfolders, eg: \\server\share
Once \\server\share\ is selected or typed, that folder would be indexed.
This would avoid indexing servers with millions of files.

[Janus]

@void

Other than for handing out files, my site isn't anything.

As for servers with millions of files on lan servers, I was actually thinking about something different.

Take advantage of already having the ETP service available.

Have a setting for whether or not to include lan shares in generic searches, or perhaps be able to select them.
Just like you can index based on several file attributes, have an option on each detected \\servername such as 'Include in generic search'.

If the user types in \\servername have everything send a share data request via etp.
Then the copy of everything on the server could respond with a list of either active shares, or of partial matches as the user continues typing.

Or just to be silly, have a separate index per \\servername or for lan servers.

__lan.db for a whole lan, or __servername.db that updates with the screen saver.
Perhaps with one of those map a file to memory tricks, not sure,

There are a lot of ideas, most of which work better than any of the garbage M$ has been foisting off on users the last decade.
As I have said however, I am not a real C/C++ programmer.
All I am doing is following the patterns I see.

If you get a chance to try out my tweak of explorer++ let me know what you think.
I am interested in feedback.


Janus.