RE: locate and updatedb

["Buchbinder, Barry (NIH/NIAID) [E]" <BBuchbinder at niaid dot nih dot gov>]

Byron Boulton sent the following at Wednesday, February 17, 2016 11:21 AM
>On 2/17/2016 11:00 AM, Buchbinder, Barry (NIH/NIAID) [E] wrote: locate
>> Byron Boulton sent the following at Wednesday, February 17, 2016 8:43
>> AM
>>> On 2/16/2016 5:55 PM, Buchbinder, Barry (NIH/NIAID) [E] wrote:
>>>>
>>>> This is technically OT since this involved a non-cygwin tool.
>>>>
>>>> find is slow compared with a non-Cygwin tool, specifically dir (cmd.exe).
>>>>
>>>> Compare find with cmd.exe's dir.  Note that even with the benefit of
>>>> caching (compare the 1st and 3rd times), find takes twice as long as dir.
>>>> Comparing cached times (2nd vs 3rd), dir is 3X faster.
>>>>
>>>> $ time cmd /c dir /s /b 'C:\usr' > /dev/null ; \ time find /c/usr >
>>>> /dev/null ; \ time cmd /c dir /s /b 'C:\usr' > /dev/null
>>>>
>>>> real    0m1.326s
>>>> user    0m0.000s
>>>> sys     0m0.047s
>>>>
>>>> real    0m2.465s
>>>> user    0m0.280s
>>>> sys     0m2.184s
>>>>
>>>> real    0m0.874s
>>>> user    0m0.000s
>>>> sys     0m0.031s
>>>>
>>>> (Note: c:\usr has nothing to do with /usr.)
>>>>
>>>> Here's how I use dir *in the abstract* for drives C: and D:.  (Note:
>>>> the
>>>> /a: option of dir lists all files, including hidden ones; /o:n sorts
>>>> by
>>>> name.)
>>>>
>>>> for D in /c /d
>>>> do
>>>>       "$(cygpath "${COMSPEC}")" /c dir /s /b /a: /o:n "$(cygpath -w "$D")"
>>>> done | \
>>>> tr -s '\r\n' '\n' | \
>>>> cygpath -u -f - | \
>>>> sed -e '/^$/d' -e 's,/\+,/,g' \
>>>> sort -u \
>>>> /usr/libexec/frcode > /tmp/updatedb.tmp chmod --reference
>>>> /var/locatedb /tmp/updatedb.tmp mv /tmp/updatedb.tmp /var/locatedb
>>>>
>>>> What I actually do (attached) is more complicated.  My script
>>>> chooses which directories are scanned, does them in parallel, and
>>>> prints pretty messages.  I get error messages for very long paths (>
>>>> ~250 bytes).  It works well enough for me; YMMV.
>>>
>>> Are you using dir in some sort of custom way to build the database
>>> used by locate? Or are you saying that rather than ever using the
>>> find command to find files, you use a custom script which uses dir?
>>
>> I use dir only to generate the locate database, because scanning the
>> better part of several disks takes so long.  I do not substitute dir
>> for find for other purposes.  One could, but usually locate does what
>> I need, and when it doesn't, I use find.
>
>understands how to read this custom database? If I read you updatedb.sh
>script properly, it produces a file which is just a sorted text file
>with one line per file found by updatedb.sh.

Sorry.  In the example in the email text I forgot a pipe sign after sort
and feeding into /usr/libexec/frcode, which convert to located format.
That fragment should have been as follows.

sort -u | \
/usr/libexec/frcode > /tmp/updatedb.tmp

It's really been so long since I put updated.sh together that I would need
to study it to make detailed comments.  Indeed, my memories of putting it
together are lost in the mists of time.

What I'd advise is to use the script that comes with findutils,
/usr/bin/updated, as your model.  Substitute dir for find, adjust start
Points 9drives or directories), convert line endings, etc., and running
through cygpath, and making other necessary changes before running through
frcode.

Sorry that I cannot be of more help.

Good luck.

- Barry
  Disclaimer: Statements made herein are not made on behalf of NIAID.


--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple