Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance

[Corinna Vinschen <corinna-cygwin at cygwin dot com>]

On Oct  3 23:33, Derry Shribman wrote:
> > I'm missing a comparison using the latest Cygwin from CVS.  That's
> > much more interesting than 1.7.5.
> 
> All the comparisons where done BOTH on 1.7.5 AND on 1.7.7!
> Here are the 1.7.7 results again:

I saw the numbers for 1.7.7 but I was talking about current CVS.  Due to
Yoni's mail to cygwin-patches I already changed quite a few things.  A
comparison should take the latest incarnation of the code into account
since, obviously, development is going into this direction.

> Could you compile and run it and report the results on your PCs?

Compile the latest Cygwin from CVS and do the comparison on the machines
you already used.  What sense does it make to add YA machine with
different hardware?  Especially since all my Windows machines except for
a single netbook are running in VMs.

> > You also never replied to my mail describing the suffix problem when
> > using the NtQueryDirectoryFile function:
> > http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html
> 
> Yoni was busy with preparing the stat performance test program, so I
> will answer this one:
> 
> Calling NtQueryDirectoryFile() is implemented in the kernel by
> IRP_MJ_DIRECTORY_CONTROL, and RestartScan is flags |=
> SL_RESTART_SCAN, ReturnSingleEntry is flags |=
> SL_RETURN_SINGLE_ENTRY.
> 
> What is missing is being able to set flags |= SL_INDEX_SPECIFIED
> when calling from user-mode.

The WDK description does not say that SL_INDEX_SPECIFIED provides
this functionality and it also specifies that using the FileIndex
does not make any sense on NTFS.  Which in turn makes sense, given
that NTFS keeps directories always in name sorted order.

> So - as you wrote: This means that NtQueryDirectoryFile() should be
> called with a 'fresh' handle every time.
> 
> Thats why in the performance test application Yoni wrote, a new
> handle was called for every NtQueryDirectoryFile() - so that this
> was taken into account.

That's not really the same.  The test application does not take .lnk and
.exe suffixes into account.  It just stats what it already got from a
former readdir.  My private test within the Cygwin code itself was not
as positive and in fact slowed down `ls -l' since now there was YA
handle to open.  But anyway.  Maybe the method using a filter with
wildcards is a winner.

> We have NOD32 anti-virus installed on all XP PCs at our work place.
> So that may probably be one of the reasons for the XP slowdown. Can
> you run the test program on an XP at your place and see how results
> differ?

Well, I can try, but it's a VM, too.

> > However, it's easy to make a speed comparsion which ignores the bunch of
> > problems Cygwin is fighting against.  What about the problems of various
> > filesystems, for instance?
> 
> Cygwin anyway today caches the volume type. So it can know whether
> it is NTFS/FAT without a system call.

No, not really.  You can make this point for the drive which hosts the
systemroot, but every other drive (or junction) can change at any given
time.

> > This requires at least one open handle to the volume to be sure to have
> > the right filesystem type.  So, whatever you do, you need to open a
> > handle first.  Either to the parent dir or to the file.  This is kind of
> > a chicken-egg problem.
> 
> This is exactly what Yoni cached in the patch he submitted to cygwin-patches!

I didn't look into the patch for the well-known reason.  The description
in Yoni's mail implies that the caching is based on the path, just like
in 1.5.x.  That's correct only in an environment which you control
yourself, but it's not necessarily correct for any other environment.
That's why I added a new caching algorithm to current CVS which is based
on the results of a single FileFsVolumeInformation result.  This still
requires an open handle, either of the file itself, or of it's parent
dir.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat