This is the mail archive of the cygwin-developers mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance


Hi,

While I was working on the kernel driver for Cygwin, I realized that
there is another way of doing it fast.

The problem is in the way we get the file handle, and by the way, causing NTFS to do a lot of preparation work (e.g. read ahead, and more), since it assume the user is going to read data from the file soon. In our case, in most of the cases this is not true, since we just need the file attributes.

So, What we need to do, is to open the file with FILE_READ_ATTRIBUTES flag, and then use NtQueryInformationFile. in most of the cases, this will be enough.
In Some cases, we will have to reopen the handle, with GENERIC_READ access, in order to read the magic.


The difference in huge. while opening the file with GENERIC_READ took almost 1ms per file, opening with FILE_READ_ATTRIBUTES only took 0.027ms per file:
testing /bin QIFR 3588 files stat() 3312ms, per file: 0.9229ms
testing /bin QIF 3588 files stat() 97.66ms, per file: 0.02722ms


I will start working on a patch doing this, and hope to post it soon.

I copy/paste the code of the test program and the kernel driver I already written below.
Currently we have nothing to do with the driver, but maybe in the future we will find better ways to improve performance, that will require a kernel driver.


Yoni.

Code: cygtest.c
==========================================
#include <stdlib.h>
#include <stdio.h>
#include <dirent.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#include <windows.h>
#include <mmsystem.h>
#include <ntdef.h>
#include <wchar.h>
#include "cygwin_structs.h"

/* to compile: gcc -g cygtest.c -o cygtest.exe -lntdll -lwinmm */
/* results on /bin ~3500 files:
   XP: CYGWIN_NT-5.1 yoni 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
   lstat(1.7.5 unpatched) 3587 files stat() 490.2ms, per file: 0.1367ms
   lstat(1.7.5 patched) 3585 files stat() 78.12ms, per file: 0.02179ms
   lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
   lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
   QFAF 3585 files stat() 38.09ms, per file: 0.01062ms
   QIF 3585 files stat() 3309ms, per file: 0.9229ms
   QDF 3585 files stat() 105.5ms, per file: 0.02942ms

Win7: CYGWIN_NT-6.1 bush-PC 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
lstat(1.7.5 unpatched) 3457 files stat() 934.1ms, per file: 0.2702ms
lstat(1.7.5 patched) 3455 files stat() 634ms, per file: 0.1835ms
lstat(1.7.7 unpatched) 3459 files stat() 777ms, per file: 0.2246ms
lstat(1.7.7 patched) 3459 files stat() 631ms, per file: 0.1824ms
QFAF 3455 files stat() 574ms, per file: 0.1661ms
QIF 3455 files stat() 599ms, per file: 0.1734ms
QDF 3455 files stat() 159ms, per file: 0.04602ms
*/


typedef long long i64;
typedef unsigned long long u64;
typedef unsigned char u8;
typedef unsigned int u32;

#define do_err(...) \
do { \
    printf("%s:%d: ", __FUNCTION__, __LINE__); \
    printf(__VA_ARGS__); \
    printf("\n"); \
    exit(1); \
} while (0)

typedef struct _FILE_NETWORK_OPEN_INFORMATION {
  LARGE_INTEGER  CreationTime;
  LARGE_INTEGER  LastAccessTime;
  LARGE_INTEGER  LastWriteTime;
  LARGE_INTEGER  ChangeTime;
  LARGE_INTEGER  AllocationSize;
  LARGE_INTEGER  EndOfFile;
  ULONG  FileAttributes;
  ULONG  Unknown;
} FILE_NETWORK_OPEN_INFORMATION, *PFILE_NETWORK_OPEN_INFORMATION;

typedef struct _IO_STATUS_BLOCK {
    union {
        NTSTATUS Status;
        PVOID Pointer;
    };
    ULONG_PTR Information;
} IO_STATUS_BLOCK, *PIO_STATUS_BLOCK;

typedef enum _FILE_INFORMATION_CLASS {
  FileDirectoryInformation = 1,
  FileFullDirectoryInformation,
  FileBothDirectoryInformation,
  FileBasicInformation,
  FileStandardInformation,
  FileInternalInformation,
  FileEaInformation,
  FileAccessInformation,
  FileNameInformation,
  FileRenameInformation,
  FileLinkInformation,
  FileNamesInformation,
  FileDispositionInformation,
  FilePositionInformation,
  FileFullEaInformation,
  FileModeInformation,
  FileAlignmentInformation,
  FileAllInformation,
  FileAllocationInformation,
  FileEndOfFileInformation,
  FileAlternateNameInformation,
  FileStreamInformation,
  FilePipeInformation,
  FilePipeLocalInformation,
  FilePipeRemoteInformation,
  FileMailslotQueryInformation,
  FileMailslotSetInformation,
  FileCompressionInformation,
  FileObjectIdInformation,
  FileCompletionInformation,
  FileMoveClusterInformation,
  FileQuotaInformation,
  FileReparsePointInformation,
  FileNetworkOpenInformation,
  FileAttributeTagInformation,
  FileTrackingInformation,
  FileIdBothDirectoryInformation,
  FileIdFullDirectoryInformation,
  FileValidDataLengthInformation,
  FileShortNameInformation,
  FileMaximumInformation
} FILE_INFORMATION_CLASS, *PFILE_INFORMATION_CLASS;

#define NSEC_PER_MS 1000000
#define SHARED_DATA ((KSHARED_USER_DATA *)0x7ffe0000)
#define STATUS_NO_MORE_FILES ((NTSTATUS)0x80000006L)

typedef struct _KUSER_SHARED_DATA {
    unsigned long TickCountLowDeprecated;
    unsigned long TickCountMultiplier;
    volatile unsigned long long InterruptTimeQuad;
} KSHARED_USER_DATA;

static u64 time_monotonic_nsec(void)
{
    u64 t;
    while ((t = SHARED_DATA->InterruptTimeQuad)!=
	SHARED_DATA->InterruptTimeQuad);
    return t*100;
}

static void usage(void)
{
    fprintf(stderr, "stat perf: cygtest METHOD [dir]\n"
	"  lstat: Cygwin's lstat\n"
	"  QFAF: QueryFullAttributesFile\n"
	"  QIF: QueryInformationFile\n"
	"  QIFR: QueryInformationFile without READ permissions\n"
	"  QDF: QueryDirectoryFile\n"
	"  KQIF: QueryInformationFile in kernel\n"
	"  KQDF: QueryDirectoryFile in kernel\n"
	"\n"
	"bash: uname -a; for i in lstat QFAF QIF QDF KQIF KQDF ; do ./cygtest "
	"$i ; done\n");
    exit(1);
}

NTSTATUS NTAPI NtQueryFullAttributesFile(POBJECT_ATTRIBUTES ObjectAttributes,
PFILE_NETWORK_OPEN_INFORMATION FileInformation);
NTSTATUS NTAPI RtlAnsiStringToUnicodeString(
PUNICODE_STRING DestinationString, PANSI_STRING SourceString,
BOOLEAN AllocateDestinationString);
NTSTATUS NTAPI NtQueryAttributesFile(
POBJECT_ATTRIBUTES ObjectAttributes,
PFILE_BASIC_INFORMATION FileAttributes);
NTSTATUS NTAPI NtQueryDirectoryFile(HANDLE FileHandle, HANDLE Event,
void *ApcRoutine, PVOID ApcContext, PIO_STATUS_BLOCK IoStatusBlock,
PVOID FileInformation, ULONG Length,
FILE_INFORMATION_CLASS FileInformationClass, BOOLEAN ReturnSingleEntry,
PUNICODE_STRING FileMask, BOOLEAN RestartScan);
NTSTATUS NTAPI NtOpenFile(PHANDLE FileHandle, ACCESS_MASK DesiredAccess,
POBJECT_ATTRIBUTES ObjectAttributes, PIO_STATUS_BLOCK IoStatusBlock,
ULONG ShareAccess, ULONG OpenOptions);
NTSTATUS NTAPI NtCreateFile(PHANDLE FileHandle, ACCESS_MASK DesiredAccess,
POBJECT_ATTRIBUTES ObjectAttributes, PIO_STATUS_BLOCK IoStatusBlock,
PLARGE_INTEGER AllocationSize, ULONG FileAttributes, ULONG ShareAccess,
ULONG CreateDisposition, ULONG CreateOptions, PVOID EaBuffer OPTIONAL,
ULONG EaLength);
NTSTATUS NTAPI NtQueryInformationFile(HANDLE FileHandle,
PIO_STATUS_BLOCK IoStatusBlock, PVOID FileInformation,
ULONG Length, FILE_INFORMATION_CLASS FileInformationClass);


WCHAR *wstr_from_str(WCHAR *wstr, char *utf)
{
    WCHAR *wc = wstr;
    u8 *u = (u8 *)utf;
    u32 t;
    int i = 0;
    while (*u)
    {
	u8 c = u[0];
	if (c<0Xc0)
	{
	    wc[i++] = c;
	    u++;
	}
	else if (c<0xe0)
	{
	    if ((u[1] & 0xc0)!=0x80)
	        goto Invalid;
	    t = ((c & 0x1f) << 6) | (u[1] & 0x3f);
	    wc[i++] = (WCHAR)t;
	    u += 2;
	}
	else if (c<0xf0)
	{
	    if ((u[1] & 0xc0)!=0x80 || (u[2] & 0xc0)!=0x80)
	        goto Invalid;
	    t = ((c & 0x0f) << 12) | ((u[1] & 0x3f) << 6) | (u[2] & 0x3f);
	    wc[i++] = (WCHAR)t;
	    u += 3;
	}
	else
	    goto Invalid;
	continue;
Invalid:
	wc[i++] = '?'; /* invalid */
	u++;
    }
    wc[i] = 0;
    return wstr;
}

static void get_nt_file_path(WCHAR **wdir, char *dir)
{
    char win_dir[512], *p;
    sprintf(win_dir, "\\??\\c:/cygwin%s", dir);
    for (p = win_dir; *p; p++)
    {
	if (*p=='/')
	    *p = '\\';
    }
    if (*wdir)
	free(*wdir);
    *wdir = malloc(sizeof(WCHAR)*(strlen(win_dir)+1));
    wstr_from_str(*wdir, win_dir);
}

static HANDLE get_dir_handle(char *dir)
{
    WCHAR *wdir = NULL;
    IO_STATUS_BLOCK io;
    UNICODE_STRING udir;
    OBJECT_ATTRIBUTES attr;
    HANDLE hdir;
    int ret;
    get_nt_file_path(&wdir, dir);
    udir.Buffer = wdir;
    udir.Length = wcslen(udir.Buffer) * sizeof(WCHAR);
    udir.MaximumLength = udir.Length + sizeof(WCHAR);
    InitializeObjectAttributes(&attr, &udir, 0, NULL, NULL);
    if ((ret = NtOpenFile(&hdir, SYNCHRONIZE|FILE_LIST_DIRECTORY,
	&attr, &io, FILE_SHARE_VALID_FLAGS,
	FILE_SYNCHRONOUS_IO_NONALERT|FILE_OPEN_FOR_BACKUP_INTENT
	|FILE_DIRECTORY_FILE)))
    {
	do_err("NtOpenFile failed(%ls): %d", wdir, ret);
    }
    free(wdir);
    return hdir;
}

static int query_file_by_dir_handle(char *dir, char *file)
{
    IO_STATUS_BLOCK io;
    UNICODE_STRING file_mask;
    HANDLE hdir;
    WCHAR *wfile;
    struct {
	FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
	WCHAR namebuf[NAME_MAX + 1];
    } fdi;
    hdir = get_dir_handle(dir);
    wfile = malloc(sizeof(WCHAR)*(strlen(file)+1));
    wstr_from_str(wfile, file);
    file_mask.Buffer = wfile;
    file_mask.Length = wcslen(file_mask.Buffer) * sizeof(WCHAR);
    file_mask.MaximumLength = file_mask.Length + sizeof(WCHAR);
    if (NtQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &fdi, sizeof(fdi),
	FileIdBothDirectoryInformation, 1, &file_mask, 1))
    {
	do_err("NtQueryDirectoryFile failed (%s)", file);
    }
    NtClose(hdir);
    return 0;
}

static int query_file_by_handle(char *file, int read)
{
OBJECT_ATTRIBUTES attr;
UNICODE_STRING ufile;
struct {
FILE_ALL_INFORMATION fai;
WCHAR namebuf[NAME_MAX + 1];
} fai;
WCHAR *wfile = NULL;
IO_STATUS_BLOCK io;
HANDLE h;
get_nt_file_path(&wfile, file);
ufile.Buffer = wfile;
ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
if (NtCreateFile(&h, SYNCHRONIZE |
(read ? GENERIC_READ : READ_CONTROL | FILE_READ_ATTRIBUTES),
&attr, &io, NULL, 0,
FILE_SHARE_VALID_FLAGS, FILE_OPEN,
FILE_OPEN_REPARSE_POINT|FILE_OPEN_FOR_BACKUP_INTENT, NULL, 0))
{
do_err("NtCreateFile failed (%s)", file);
}
if (NtQueryInformationFile(h, &io, &fai, sizeof(fai), FileAllInformation))
do_err("NtQueryInformationFile failed (%s)", file);
NtClose(h);
free(wfile);
return 0;
}


static int query_file_by_name(char *file)
{
    int ret;
    OBJECT_ATTRIBUTES attr;
    FILE_NETWORK_OPEN_INFORMATION fi;
    UNICODE_STRING ufile;
    WCHAR *wfile = NULL;
    get_nt_file_path(&wfile, file);
    ufile.Buffer = wfile;
    ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
    ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
    InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
    ret = NtQueryFullAttributesFile(&attr, &fi);
    free(wfile);
    return ret;
}

static HANDLE hdrv;
static int query_file_kernel(char *dir, char *file, int mode)
{
    DWORD bytes_ret;
    query_file_t query_file;
    WCHAR *wdir = NULL;
    file_info_t file_info;
    if (!hdrv)
    {
	if (!(hdrv = CreateFileA("\\\\.\\Cygwin",
	    FILE_GENERIC_READ|FILE_GENERIC_WRITE,
	    FILE_SHARE_READ|FILE_SHARE_WRITE, NULL, OPEN_EXISTING,
	    FILE_ATTRIBUTE_NORMAL, NULL)))
	{
	    do_err("CreateFile \\\\.\\Cygwin failed %x", GetLastError());
	}
    }
    wstr_from_str(query_file.file_name[0], file);
    query_file.mode = mode;
    get_nt_file_path(&wdir, dir);
    wcscpy(query_file.dir_name, wdir);
    if (!DeviceIoControl(hdrv, QUERY_FILE_IOCTL, &query_file,
	sizeof(query_file), &file_info, sizeof(file_info_t), &bytes_ret, NULL))
    {
	do_err("DeviceIoControl failed %x", GetLastError());
    }
    free(wdir);
    return 0;
}

static int verbose = 0;
int main(int argc, char *argv[])
{
    char *dir = "/bin", fullname[512], *method = "";
    struct dirent **namelist, *d;
    int i, n = 0, ret = -1;
    struct stat buf;
    u64 start, end;
    IO_STATUS_BLOCK io;
    HANDLE hdir;
    struct {
	FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
	WCHAR dummy_buf[NAME_MAX + 1];
    } fdi;
    argv++;
    timeBeginPeriod(1);
    usleep(100); /* it takes some time for timeBeginPeriod() to work */
    if (!*argv)
	usage();
    method = *argv++;
    if (*argv)
	dir = *argv++;
    if (*argv)
	usage();
    printf("testing %s %s ", dir, method);
    start = time_monotonic_nsec();
    if (!strcmp(method, "lstat"))
    {
	if ((n = scandir(dir, &namelist, 0, alphasort))>=0)
	    ret = 0;
    }
    else
    {
	hdir = get_dir_handle(dir);
	namelist = calloc(sizeof(*namelist), 100000);
	while (!NtQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &fdi,
	    sizeof(fdi), FileIdBothDirectoryInformation, 1, NULL, 0))
	{
	    d = namelist[n] = calloc(sizeof(**namelist), 1);
	    sprintf(d->d_name, "%.*ls",
		(int)fdi.fdi.FileNameLength/2, fdi.fdi.FileName);
	    n++;
	}
	ret = 0;
	NtClose(hdir);
    }
    if (ret)
	do_err("scandir: %s", strerror(errno));
    end = time_monotonic_nsec();
    printf("%d files ", n);
    if (verbose)
	printf("scandir(%gms) ", ((double)end-start)/NSEC_PER_MS);
    start = time_monotonic_nsec();
    for (i=0; i<n; i++)
    {
	d = namelist[i];
	if (!strcmp(".", d->d_name) || !strcmp("..", d->d_name))
	    continue;
	sprintf(fullname, "%s/%s", dir, d->d_name);
	if (verbose)
	    printf("%s\n", fullname);
	if (!strcmp(method, "QFAF"))
	    ret = query_file_by_name(fullname);
	else if (!strcmp(method, "QIF"))
	    ret = query_file_by_handle(fullname, 0);
	else if (!strcmp(method, "QIFR"))
	    ret = query_file_by_handle(fullname, 1);
	else if (!strcmp(method, "QDF"))
	    ret = query_file_by_dir_handle(dir, d->d_name);
	else if (!strcmp(method, "lstat"))
	    ret = lstat(fullname, &buf);
	else if (!strcmp(method, "KQDF"))
	    ret = query_file_kernel(dir, d->d_name, DIR_HANDLE_MODE);
	else if (!strcmp(method, "KQIF"))
	    ret = query_file_kernel(dir, d->d_name, FILE_HANDLE_MODE);
	else
	    do_err("unknow method %s", method);
	if (ret)
	    do_err("stat %s: %s", d->d_name, strerror(errno));
	free(d);
    }
    free(namelist);
    end = time_monotonic_nsec();
    printf("stat() %4.4gms, per file: %4.4gms ",
	((double)end-start)/NSEC_PER_MS,
	((double)end-start)/n/NSEC_PER_MS);
    printf("\n");
    if (hdrv)
	NtClose(hdrv);
    return 0;
}

============================================================
cygdrv.c
============================================================
#include <wdm.h>
#include "cygwin_structs.h"

#define ALLOC_TAG 'INOY'
#define do_err(...) \
do { \
    DbgPrint(__VA_ARGS__); \
} while (0)

NTSTATUS ZwQueryDirectoryFile(HANDLE FileHandle, HANDLE Event,
    PIO_APC_ROUTINE ApcRoutine, PVOID ApcContext,
    PIO_STATUS_BLOCK IoStatusBlock, PVOID FileInformation, ULONG Length,
    FILE_INFORMATION_CLASS FileInformationClass, BOOLEAN ReturnSingleEntry,
    PUNICODE_STRING FileName, BOOLEAN RestartScan);

static NTSTATUS CygCreate(IN PDEVICE_OBJECT fdo, IN PIRP irp)
{
    irp->IoStatus.Status = STATUS_SUCCESS;
    irp->IoStatus.Information = 0;
    IoCompleteRequest(irp,IO_NO_INCREMENT);
    return STATUS_SUCCESS;
}

static NTSTATUS CygClose(IN PDEVICE_OBJECT fdo, IN PIRP irp)
{
    irp->IoStatus.Status = STATUS_SUCCESS;
    irp->IoStatus.Information = 0;
    IoCompleteRequest(irp,IO_NO_INCREMENT);
    return STATUS_SUCCESS;
}

static NTSTATUS get_dir_handle(HANDLE *hdir, query_file_t *query_file)
{
NTSTATUS ret;
UNICODE_STRING udir;
OBJECT_ATTRIBUTES attr;
IO_STATUS_BLOCK io;
udir.Buffer = query_file->dir_name;
udir.Length = wcslen(udir.Buffer) * sizeof(WCHAR);
udir.MaximumLength = udir.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &udir, 0, NULL, NULL);
if ((ret = ZwOpenFile(hdir, SYNCHRONIZE|FILE_LIST_DIRECTORY, &attr, &io,
FILE_SHARE_VALID_FLAGS, FILE_SYNCHRONOUS_IO_NONALERT
|FILE_OPEN_FOR_BACKUP_INTENT|FILE_DIRECTORY_FILE)))
{
do_err("NtOpenFile failed (%ls): %d", query_file->dir_name, ret);
}
return ret;
}


static NTSTATUS query_file_by_handle(file_info_t *file_info,
query_file_t *query_file)
{
int ret;
OBJECT_ATTRIBUTES attr;
UNICODE_STRING ufile;
WCHAR *wfile;
IO_STATUS_BLOCK io;
HANDLE h = NULL;
wfile = ExAllocatePoolWithTag(NonPagedPool, (wcslen(query_file->dir_name) +
wcslen(query_file->file_name[0])) * sizeof(WCHAR) + 4, ALLOC_TAG);
wcscpy(wfile, query_file->dir_name);
wcscat(wfile, L"\\");
wcscat(wfile, query_file->file_name[0]);
ufile.Buffer = wfile;
ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
if (ZwCreateFile(&h, SYNCHRONIZE | GENERIC_READ, &attr, &io, NULL, 0,
FILE_SHARE_VALID_FLAGS, FILE_OPEN,
FILE_OPEN_REPARSE_POINT|FILE_OPEN_FOR_BACKUP_INTENT, NULL, 0))
{
do_err("NtCreateFile failed (%ls)\n", wfile);
goto Exit;
}
if (ret = ZwQueryInformationFile(h, &io, &file_info->fai,
sizeof(file_info->fai) + sizeof(file_info->namebuf2),
FileAllInformation))
{
do_err("ZwQueryInformationFile failed (%ls): %d\n", wfile, ret);
}
Exit:
ExFreePoolWithTag(wfile, ALLOC_TAG);
if (h)
NtClose(h);
return 0;
}


static NTSTATUS query_file_by_dir_handle(file_info_t *file_info,
query_file_t *query_file)
{
NTSTATUS ret;
IO_STATUS_BLOCK io;
UNICODE_STRING file_mask;
HANDLE hdir;
if ((ret = get_dir_handle(&hdir, query_file)))
return ret;
file_mask.Buffer = query_file->file_name[0];
file_mask.Length = wcslen(file_mask.Buffer) * sizeof(WCHAR);
file_mask.MaximumLength = file_mask.Length + sizeof(WCHAR);
if (ret = ZwQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &file_info->fdi,
sizeof(file_info->fdi) + sizeof(file_info->namebuf),
FileIdBothDirectoryInformation, 1, &file_mask, 1))
{
do_err("NtQueryDirectoryFile failed (%ls)", query_file->file_name[0]);
}
ZwClose(hdir);
return ret;
}


static NTSTATUS CygDeviceControl(IN PDEVICE_OBJECT fdo, IN PIRP irp)
{
IO_STACK_LOCATION *irp_sp = IoGetCurrentIrpStackLocation(irp);
unsigned int code = irp_sp->Parameters.DeviceIoControl.IoControlCode;
void *data = irp->AssociatedIrp.SystemBuffer;
unsigned int len_in = irp_sp->Parameters.DeviceIoControl.InputBufferLength;
unsigned int len_out =
irp_sp->Parameters.DeviceIoControl.OutputBufferLength;
int ret = STATUS_NOT_SUPPORTED, bytes_ret = 0;
file_info_t file_info;
query_file_t *query_file = data;
switch (code)
{
case QUERY_FILE_IOCTL:
if (len_out<sizeof(file_info_t) || len_in<sizeof(query_file_t))
{
ret = STATUS_BUFFER_TOO_SMALL;
goto Exit;
}
if (query_file->mode == DIR_HANDLE_MODE)
ret = query_file_by_dir_handle(&file_info, query_file);
else if (query_file->mode == FILE_HANDLE_MODE)
ret = query_file_by_handle(&file_info, query_file);
else
ret = STATUS_NOT_SUPPORTED;
if (!ret)
{
memcpy(data, &file_info, sizeof(file_info_t));
bytes_ret = sizeof(file_info_t);
}
break;
default:
ret = STATUS_NOT_SUPPORTED;
}
Exit:
irp->IoStatus.Status = ret;
irp->IoStatus.Information = bytes_ret;
IoCompleteRequest(irp,IO_NO_INCREMENT);
return ret;
}


static DEVICE_OBJECT *dev_obj;
static UNICODE_STRING nt_devid, win32_devid;

static VOID CygUnload(IN PDRIVER_OBJECT DriverObject)
{
    IoDeleteSymbolicLink(&win32_devid);
    IoDeleteDevice(dev_obj);
}

NTSTATUS DriverEntry(IN PDRIVER_OBJECT DriverObject,
    IN PUNICODE_STRING RegistryPath)
{
    int ret;
    RtlInitUnicodeString(&nt_devid, L"\\Device\\Cygwin");
    RtlInitUnicodeString(&win32_devid, L"\\DosDevices\\Cygwin");
    if (!NT_SUCCESS(ret = IoCreateDevice(DriverObject, 0,
	&nt_devid, FILE_DEVICE_SMB, 0, 0, &dev_obj)))
    {
	/* Either not enough memory to create a device object or another
	 * device object with the same name exits. This could happen if you
	 * install another instance of this device. */
	do_err("IoCreateDevicefailed %x", ret);
	return -1;
    }
    /* add create sym link */
    if (!NT_SUCCESS(ret = IoCreateSymbolicLink(&win32_devid, &nt_devid)))
    {
	IoDeleteDevice(dev_obj);
	do_err("IoCreateSymbolicLink failed %x", ret);
	return -1;
    }
    DriverObject->DriverUnload = CygUnload;
    DriverObject->MajorFunction[IRP_MJ_CREATE] = CygCreate;
    DriverObject->MajorFunction[IRP_MJ_CLOSE] = CygClose;
    DriverObject->MajorFunction[IRP_MJ_DEVICE_CONTROL] = CygDeviceControl;
    return STATUS_SUCCESS;
}

========================================================
cygwin_structs.h
=======================================================
#ifndef __CYGWIN_STRUCTS__
#define __CYGWIN_STRUCTS__
#ifndef __KERNEL__
#include <winioctl.h>
#endif

#define CYG_CTL_CODE(function, method, access) \
    CTL_CODE(0, function, method, access)
#define IOCTL_CYG(func) \
    CYG_CTL_CODE(func, METHOD_BUFFERED, \
    FILE_READ_ACCESS | FILE_WRITE_ACCESS)

#define QUERY_FILE_IOCTL IOCTL_CYG(0x1)

NTSTATUS NTAPI NtClose(HANDLE Handle);
typedef struct _FILE_INTERNAL_INFORMATION {
    LARGE_INTEGER IndexNumber;
} FILE_INTERNAL_INFORMATION, *PFILE_INTERNAL_INFORMATION;

typedef struct _FILE_EA_INFORMATION {
    ULONG EaSize;
} FILE_EA_INFORMATION, *PFILE_EA_INFORMATION;

typedef struct _FILE_ACCESS_INFORMATION {
    ACCESS_MASK AccessFlags;
} FILE_ACCESS_INFORMATION, *PFILE_ACCESS_INFORMATION;

typedef struct _FILE_MODE_INFORMATION {
    ULONG Mode;
} FILE_MODE_INFORMATION, *PFILE_MODE_INFORMATION;

typedef struct _FILE_ALIGNMENT_INFORMATION {
    ULONG AlignmentRequirement;
} FILE_ALIGNMENT_INFORMATION, *PFILE_ALIGNMENT_INFORMATION;

typedef struct _FILE_NAME_INFORMATION {
    ULONG FileNameLength;
    WCHAR FileName[1];
} FILE_NAME_INFORMATION, *PFILE_NAME_INFORMATION;

typedef struct _FILE_ID_BOTH_DIRECTORY_INFORMATION {
    ULONG         NextEntryOffset;
    ULONG	  FileIndex;
    LARGE_INTEGER CreationTime;
    LARGE_INTEGER LastAccessTime;
    LARGE_INTEGER LastWriteTime;
    LARGE_INTEGER ChangeTime;
    LARGE_INTEGER EndOfFile;
    LARGE_INTEGER AllocationSize;
    ULONG         FileAttributes;
    ULONG         FileNameLength;
    ULONG         EaSize;
    CHAR          ShortNameLength;
    WCHAR         ShortName[12];
    LARGE_INTEGER FileId;
    WCHAR         FileName[1];
} FILE_ID_BOTH_DIRECTORY_INFORMATION, *PFILE_ID_BOTH_DIRECTORY_INFORMATION;

#ifdef __CYGWIN__
typedef struct _FILE_POSITION_INFORMATION {
    LARGE_INTEGER CurrentByteOffset;
} FILE_POSITION_INFORMATION, *PFILE_POSITION_INFORMATION;

typedef struct _FILE_BASIC_INFORMATION {
  LARGE_INTEGER           CreationTime;
  LARGE_INTEGER           LastAccessTime;
  LARGE_INTEGER           LastWriteTime;
  LARGE_INTEGER           ChangeTime;
  ULONG                   FileAttributes;
} FILE_BASIC_INFORMATION, *PFILE_BASIC_INFORMATION;

typedef struct _FILE_STANDARD_INFORMATION {
    LARGE_INTEGER AllocationSize;
    LARGE_INTEGER EndOfFile;
    ULONG NumberOfLinks;
    BOOLEAN DeletePending;
    BOOLEAN Directory;
} FILE_STANDARD_INFORMATION, *PFILE_STANDARD_INFORMATION;
#endif

typedef struct _FILE_ALL_INFORMATION {
  FILE_BASIC_INFORMATION     BasicInformation;
  FILE_STANDARD_INFORMATION  StandardInformation;
  FILE_INTERNAL_INFORMATION  InternalInformation;
  FILE_EA_INFORMATION        EaInformation;
  FILE_ACCESS_INFORMATION    AccessInformation;
  FILE_POSITION_INFORMATION  PositionInformation;
  FILE_MODE_INFORMATION      ModeInformation;
  FILE_ALIGNMENT_INFORMATION AlignmentInformation;
  FILE_NAME_INFORMATION      NameInformation;
} FILE_ALL_INFORMATION, *PFILE_ALL_INFORMATION;

#define DRV_MAX_PATH 512
#define DRV_NAME_MAX 255
#define DIR_HANDLE_MODE 1
#define FILE_HANDLE_MODE 3
typedef struct {
    WCHAR dir_name[DRV_MAX_PATH];
    int nfiles;
    WCHAR file_name[3][DRV_MAX_PATH];
    int mode;
} query_file_t;

typedef struct {
    FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
    WCHAR namebuf[DRV_NAME_MAX + 1];
    FILE_STANDARD_INFORMATION fsi;
    FILE_ALL_INFORMATION fai;
    WCHAR namebuf2[DRV_NAME_MAX + 1];
} file_info_t;

#endif


On 5/10/2010 4:24 PM, Derry Shribman wrote:
Hi,

> I saw the numbers for 1.7.7 but I was talking about current CVS.

Yoni is currently busy working on a "cygwin kernel-mode accelerator"
prototype. Hopefully by tomorrow he will be able to post it, and also
test the performance numbers on the current CVS cygwin1.dll.

Derry

On 10/4/2010 12:23 PM, Corinna Vinschen wrote:
On Oct 3 23:33, Derry Shribman wrote:
I'm missing a comparison using the latest Cygwin from CVS. That's
much more interesting than 1.7.5.

All the comparisons where done BOTH on 1.7.5 AND on 1.7.7! Here are the 1.7.7 results again:

I saw the numbers for 1.7.7 but I was talking about current CVS. Due to Yoni's mail to cygwin-patches I already changed quite a few things. A comparison should take the latest incarnation of the code into account since, obviously, development is going into this direction.

Could you compile and run it and report the results on your PCs?

Compile the latest Cygwin from CVS and do the comparison on the machines you already used. What sense does it make to add YA machine with different hardware? Especially since all my Windows machines except for a single netbook are running in VMs.

You also never replied to my mail describing the suffix problem when
using the NtQueryDirectoryFile function:
http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html

Yoni was busy with preparing the stat performance test program, so I will answer this one:

Calling NtQueryDirectoryFile() is implemented in the kernel by
IRP_MJ_DIRECTORY_CONTROL, and RestartScan is flags |=
SL_RESTART_SCAN, ReturnSingleEntry is flags |=
SL_RETURN_SINGLE_ENTRY.

What is missing is being able to set flags |= SL_INDEX_SPECIFIED
when calling from user-mode.

The WDK description does not say that SL_INDEX_SPECIFIED provides this functionality and it also specifies that using the FileIndex does not make any sense on NTFS. Which in turn makes sense, given that NTFS keeps directories always in name sorted order.

So - as you wrote: This means that NtQueryDirectoryFile() should be
called with a 'fresh' handle every time.

Thats why in the performance test application Yoni wrote, a new
handle was called for every NtQueryDirectoryFile() - so that this
was taken into account.

That's not really the same. The test application does not take .lnk and .exe suffixes into account. It just stats what it already got from a former readdir. My private test within the Cygwin code itself was not as positive and in fact slowed down `ls -l' since now there was YA handle to open. But anyway. Maybe the method using a filter with wildcards is a winner.

We have NOD32 anti-virus installed on all XP PCs at our work place.
So that may probably be one of the reasons for the XP slowdown. Can
you run the test program on an XP at your place and see how results
differ?

Well, I can try, but it's a VM, too.


However, it's easy to make a speed comparsion which ignores the
bunch of
problems Cygwin is fighting against. What about the problems of various
filesystems, for instance?

Cygwin anyway today caches the volume type. So it can know whether it is NTFS/FAT without a system call.

No, not really. You can make this point for the drive which hosts the systemroot, but every other drive (or junction) can change at any given time.

This requires at least one open handle to the volume to be sure to have
the right filesystem type. So, whatever you do, you need to open a
handle first. Either to the parent dir or to the file. This is kind of
a chicken-egg problem.

This is exactly what Yoni cached in the patch he submitted to cygwin-patches!

I didn't look into the patch for the well-known reason. The description in Yoni's mail implies that the caching is based on the path, just like in 1.5.x. That's correct only in an environment which you control yourself, but it's not necessarily correct for any other environment. That's why I added a new caching algorithm to current CVS which is based on the results of a single FileFsVolumeInformation result. This still requires an open handle, either of the file itself, or of it's parent dir.


Corinna





Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]