This is the mail archive of the
cygwin-developers
mailing list for the Cygwin project.
Re: Cygwin Filesystem Performance degradation 1.7.5 vs 1.7.7, and methods for improving performance
Hi,
While I was working on the kernel driver for Cygwin, I realized that
there is another way of doing it fast.
The problem is in the way we get the file handle, and by the way,
causing NTFS to do a lot of preparation work (e.g. read ahead, and
more), since it assume the user is going to read data from the file
soon. In our case, in most of the cases this is not true, since we just
need the file attributes.
So, What we need to do, is to open the file with FILE_READ_ATTRIBUTES
flag, and then use NtQueryInformationFile. in most of the cases, this
will be enough.
In Some cases, we will have to reopen the handle, with GENERIC_READ
access, in order to read the magic.
The difference in huge. while opening the file with GENERIC_READ took
almost 1ms per file, opening with FILE_READ_ATTRIBUTES only took 0.027ms
per file:
testing /bin QIFR 3588 files stat() 3312ms, per file: 0.9229ms
testing /bin QIF 3588 files stat() 97.66ms, per file: 0.02722ms
I will start working on a patch doing this, and hope to post it soon.
I copy/paste the code of the test program and the kernel driver I
already written below.
Currently we have nothing to do with the driver, but maybe in the future
we will find better ways to improve performance, that will require a
kernel driver.
Yoni.
Code: cygtest.c
==========================================
#include <stdlib.h>
#include <stdio.h>
#include <dirent.h>
#include <sys/stat.h>
#include <unistd.h>
#include <errno.h>
#include <time.h>
#include <windows.h>
#include <mmsystem.h>
#include <ntdef.h>
#include <wchar.h>
#include "cygwin_structs.h"
/* to compile: gcc -g cygtest.c -o cygtest.exe -lntdll -lwinmm */
/* results on /bin ~3500 files:
XP: CYGWIN_NT-5.1 yoni 1.7.5(0.228/5/3) 2010-07-18 14:53 i686 Cygwin
lstat(1.7.5 unpatched) 3587 files stat() 490.2ms, per file: 0.1367ms
lstat(1.7.5 patched) 3585 files stat() 78.12ms, per file: 0.02179ms
lstat(1.7.7 unpatched) 3588 files stat() 3570ms, per file: 0.9951ms
lstat(1.7.7 patched) 3588 files stat() 3374ms, per file: 0.9404ms
QFAF 3585 files stat() 38.09ms, per file: 0.01062ms
QIF 3585 files stat() 3309ms, per file: 0.9229ms
QDF 3585 files stat() 105.5ms, per file: 0.02942ms
Win7: CYGWIN_NT-6.1 bush-PC 1.7.5(0.228/5/3) 2010-07-18 14:53 i686
Cygwin
lstat(1.7.5 unpatched) 3457 files stat() 934.1ms, per file: 0.2702ms
lstat(1.7.5 patched) 3455 files stat() 634ms, per file: 0.1835ms
lstat(1.7.7 unpatched) 3459 files stat() 777ms, per file: 0.2246ms
lstat(1.7.7 patched) 3459 files stat() 631ms, per file: 0.1824ms
QFAF 3455 files stat() 574ms, per file: 0.1661ms
QIF 3455 files stat() 599ms, per file: 0.1734ms
QDF 3455 files stat() 159ms, per file: 0.04602ms
*/
typedef long long i64;
typedef unsigned long long u64;
typedef unsigned char u8;
typedef unsigned int u32;
#define do_err(...) \
do { \
printf("%s:%d: ", __FUNCTION__, __LINE__); \
printf(__VA_ARGS__); \
printf("\n"); \
exit(1); \
} while (0)
typedef struct _FILE_NETWORK_OPEN_INFORMATION {
LARGE_INTEGER CreationTime;
LARGE_INTEGER LastAccessTime;
LARGE_INTEGER LastWriteTime;
LARGE_INTEGER ChangeTime;
LARGE_INTEGER AllocationSize;
LARGE_INTEGER EndOfFile;
ULONG FileAttributes;
ULONG Unknown;
} FILE_NETWORK_OPEN_INFORMATION, *PFILE_NETWORK_OPEN_INFORMATION;
typedef struct _IO_STATUS_BLOCK {
union {
NTSTATUS Status;
PVOID Pointer;
};
ULONG_PTR Information;
} IO_STATUS_BLOCK, *PIO_STATUS_BLOCK;
typedef enum _FILE_INFORMATION_CLASS {
FileDirectoryInformation = 1,
FileFullDirectoryInformation,
FileBothDirectoryInformation,
FileBasicInformation,
FileStandardInformation,
FileInternalInformation,
FileEaInformation,
FileAccessInformation,
FileNameInformation,
FileRenameInformation,
FileLinkInformation,
FileNamesInformation,
FileDispositionInformation,
FilePositionInformation,
FileFullEaInformation,
FileModeInformation,
FileAlignmentInformation,
FileAllInformation,
FileAllocationInformation,
FileEndOfFileInformation,
FileAlternateNameInformation,
FileStreamInformation,
FilePipeInformation,
FilePipeLocalInformation,
FilePipeRemoteInformation,
FileMailslotQueryInformation,
FileMailslotSetInformation,
FileCompressionInformation,
FileObjectIdInformation,
FileCompletionInformation,
FileMoveClusterInformation,
FileQuotaInformation,
FileReparsePointInformation,
FileNetworkOpenInformation,
FileAttributeTagInformation,
FileTrackingInformation,
FileIdBothDirectoryInformation,
FileIdFullDirectoryInformation,
FileValidDataLengthInformation,
FileShortNameInformation,
FileMaximumInformation
} FILE_INFORMATION_CLASS, *PFILE_INFORMATION_CLASS;
#define NSEC_PER_MS 1000000
#define SHARED_DATA ((KSHARED_USER_DATA *)0x7ffe0000)
#define STATUS_NO_MORE_FILES ((NTSTATUS)0x80000006L)
typedef struct _KUSER_SHARED_DATA {
unsigned long TickCountLowDeprecated;
unsigned long TickCountMultiplier;
volatile unsigned long long InterruptTimeQuad;
} KSHARED_USER_DATA;
static u64 time_monotonic_nsec(void)
{
u64 t;
while ((t = SHARED_DATA->InterruptTimeQuad)!=
SHARED_DATA->InterruptTimeQuad);
return t*100;
}
static void usage(void)
{
fprintf(stderr, "stat perf: cygtest METHOD [dir]\n"
" lstat: Cygwin's lstat\n"
" QFAF: QueryFullAttributesFile\n"
" QIF: QueryInformationFile\n"
" QIFR: QueryInformationFile without READ permissions\n"
" QDF: QueryDirectoryFile\n"
" KQIF: QueryInformationFile in kernel\n"
" KQDF: QueryDirectoryFile in kernel\n"
"\n"
"bash: uname -a; for i in lstat QFAF QIF QDF KQIF KQDF ; do ./cygtest "
"$i ; done\n");
exit(1);
}
NTSTATUS NTAPI NtQueryFullAttributesFile(POBJECT_ATTRIBUTES
ObjectAttributes,
PFILE_NETWORK_OPEN_INFORMATION FileInformation);
NTSTATUS NTAPI RtlAnsiStringToUnicodeString(
PUNICODE_STRING DestinationString, PANSI_STRING SourceString,
BOOLEAN AllocateDestinationString);
NTSTATUS NTAPI NtQueryAttributesFile(
POBJECT_ATTRIBUTES ObjectAttributes,
PFILE_BASIC_INFORMATION FileAttributes);
NTSTATUS NTAPI NtQueryDirectoryFile(HANDLE FileHandle, HANDLE Event,
void *ApcRoutine, PVOID ApcContext, PIO_STATUS_BLOCK IoStatusBlock,
PVOID FileInformation, ULONG Length,
FILE_INFORMATION_CLASS FileInformationClass, BOOLEAN ReturnSingleEntry,
PUNICODE_STRING FileMask, BOOLEAN RestartScan);
NTSTATUS NTAPI NtOpenFile(PHANDLE FileHandle, ACCESS_MASK DesiredAccess,
POBJECT_ATTRIBUTES ObjectAttributes, PIO_STATUS_BLOCK IoStatusBlock,
ULONG ShareAccess, ULONG OpenOptions);
NTSTATUS NTAPI NtCreateFile(PHANDLE FileHandle, ACCESS_MASK DesiredAccess,
POBJECT_ATTRIBUTES ObjectAttributes, PIO_STATUS_BLOCK IoStatusBlock,
PLARGE_INTEGER AllocationSize, ULONG FileAttributes, ULONG ShareAccess,
ULONG CreateDisposition, ULONG CreateOptions, PVOID EaBuffer OPTIONAL,
ULONG EaLength);
NTSTATUS NTAPI NtQueryInformationFile(HANDLE FileHandle,
PIO_STATUS_BLOCK IoStatusBlock, PVOID FileInformation,
ULONG Length, FILE_INFORMATION_CLASS FileInformationClass);
WCHAR *wstr_from_str(WCHAR *wstr, char *utf)
{
WCHAR *wc = wstr;
u8 *u = (u8 *)utf;
u32 t;
int i = 0;
while (*u)
{
u8 c = u[0];
if (c<0Xc0)
{
wc[i++] = c;
u++;
}
else if (c<0xe0)
{
if ((u[1] & 0xc0)!=0x80)
goto Invalid;
t = ((c & 0x1f) << 6) | (u[1] & 0x3f);
wc[i++] = (WCHAR)t;
u += 2;
}
else if (c<0xf0)
{
if ((u[1] & 0xc0)!=0x80 || (u[2] & 0xc0)!=0x80)
goto Invalid;
t = ((c & 0x0f) << 12) | ((u[1] & 0x3f) << 6) | (u[2] & 0x3f);
wc[i++] = (WCHAR)t;
u += 3;
}
else
goto Invalid;
continue;
Invalid:
wc[i++] = '?'; /* invalid */
u++;
}
wc[i] = 0;
return wstr;
}
static void get_nt_file_path(WCHAR **wdir, char *dir)
{
char win_dir[512], *p;
sprintf(win_dir, "\\??\\c:/cygwin%s", dir);
for (p = win_dir; *p; p++)
{
if (*p=='/')
*p = '\\';
}
if (*wdir)
free(*wdir);
*wdir = malloc(sizeof(WCHAR)*(strlen(win_dir)+1));
wstr_from_str(*wdir, win_dir);
}
static HANDLE get_dir_handle(char *dir)
{
WCHAR *wdir = NULL;
IO_STATUS_BLOCK io;
UNICODE_STRING udir;
OBJECT_ATTRIBUTES attr;
HANDLE hdir;
int ret;
get_nt_file_path(&wdir, dir);
udir.Buffer = wdir;
udir.Length = wcslen(udir.Buffer) * sizeof(WCHAR);
udir.MaximumLength = udir.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &udir, 0, NULL, NULL);
if ((ret = NtOpenFile(&hdir, SYNCHRONIZE|FILE_LIST_DIRECTORY,
&attr, &io, FILE_SHARE_VALID_FLAGS,
FILE_SYNCHRONOUS_IO_NONALERT|FILE_OPEN_FOR_BACKUP_INTENT
|FILE_DIRECTORY_FILE)))
{
do_err("NtOpenFile failed(%ls): %d", wdir, ret);
}
free(wdir);
return hdir;
}
static int query_file_by_dir_handle(char *dir, char *file)
{
IO_STATUS_BLOCK io;
UNICODE_STRING file_mask;
HANDLE hdir;
WCHAR *wfile;
struct {
FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
WCHAR namebuf[NAME_MAX + 1];
} fdi;
hdir = get_dir_handle(dir);
wfile = malloc(sizeof(WCHAR)*(strlen(file)+1));
wstr_from_str(wfile, file);
file_mask.Buffer = wfile;
file_mask.Length = wcslen(file_mask.Buffer) * sizeof(WCHAR);
file_mask.MaximumLength = file_mask.Length + sizeof(WCHAR);
if (NtQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &fdi, sizeof(fdi),
FileIdBothDirectoryInformation, 1, &file_mask, 1))
{
do_err("NtQueryDirectoryFile failed (%s)", file);
}
NtClose(hdir);
return 0;
}
static int query_file_by_handle(char *file, int read)
{
OBJECT_ATTRIBUTES attr;
UNICODE_STRING ufile;
struct {
FILE_ALL_INFORMATION fai;
WCHAR namebuf[NAME_MAX + 1];
} fai;
WCHAR *wfile = NULL;
IO_STATUS_BLOCK io;
HANDLE h;
get_nt_file_path(&wfile, file);
ufile.Buffer = wfile;
ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
if (NtCreateFile(&h, SYNCHRONIZE |
(read ? GENERIC_READ : READ_CONTROL | FILE_READ_ATTRIBUTES),
&attr, &io, NULL, 0,
FILE_SHARE_VALID_FLAGS, FILE_OPEN,
FILE_OPEN_REPARSE_POINT|FILE_OPEN_FOR_BACKUP_INTENT, NULL, 0))
{
do_err("NtCreateFile failed (%s)", file);
}
if (NtQueryInformationFile(h, &io, &fai, sizeof(fai),
FileAllInformation))
do_err("NtQueryInformationFile failed (%s)", file);
NtClose(h);
free(wfile);
return 0;
}
static int query_file_by_name(char *file)
{
int ret;
OBJECT_ATTRIBUTES attr;
FILE_NETWORK_OPEN_INFORMATION fi;
UNICODE_STRING ufile;
WCHAR *wfile = NULL;
get_nt_file_path(&wfile, file);
ufile.Buffer = wfile;
ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
ret = NtQueryFullAttributesFile(&attr, &fi);
free(wfile);
return ret;
}
static HANDLE hdrv;
static int query_file_kernel(char *dir, char *file, int mode)
{
DWORD bytes_ret;
query_file_t query_file;
WCHAR *wdir = NULL;
file_info_t file_info;
if (!hdrv)
{
if (!(hdrv = CreateFileA("\\\\.\\Cygwin",
FILE_GENERIC_READ|FILE_GENERIC_WRITE,
FILE_SHARE_READ|FILE_SHARE_WRITE, NULL, OPEN_EXISTING,
FILE_ATTRIBUTE_NORMAL, NULL)))
{
do_err("CreateFile \\\\.\\Cygwin failed %x", GetLastError());
}
}
wstr_from_str(query_file.file_name[0], file);
query_file.mode = mode;
get_nt_file_path(&wdir, dir);
wcscpy(query_file.dir_name, wdir);
if (!DeviceIoControl(hdrv, QUERY_FILE_IOCTL, &query_file,
sizeof(query_file), &file_info, sizeof(file_info_t), &bytes_ret, NULL))
{
do_err("DeviceIoControl failed %x", GetLastError());
}
free(wdir);
return 0;
}
static int verbose = 0;
int main(int argc, char *argv[])
{
char *dir = "/bin", fullname[512], *method = "";
struct dirent **namelist, *d;
int i, n = 0, ret = -1;
struct stat buf;
u64 start, end;
IO_STATUS_BLOCK io;
HANDLE hdir;
struct {
FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
WCHAR dummy_buf[NAME_MAX + 1];
} fdi;
argv++;
timeBeginPeriod(1);
usleep(100); /* it takes some time for timeBeginPeriod() to work */
if (!*argv)
usage();
method = *argv++;
if (*argv)
dir = *argv++;
if (*argv)
usage();
printf("testing %s %s ", dir, method);
start = time_monotonic_nsec();
if (!strcmp(method, "lstat"))
{
if ((n = scandir(dir, &namelist, 0, alphasort))>=0)
ret = 0;
}
else
{
hdir = get_dir_handle(dir);
namelist = calloc(sizeof(*namelist), 100000);
while (!NtQueryDirectoryFile(hdir, NULL, NULL, 0, &io, &fdi,
sizeof(fdi), FileIdBothDirectoryInformation, 1, NULL, 0))
{
d = namelist[n] = calloc(sizeof(**namelist), 1);
sprintf(d->d_name, "%.*ls",
(int)fdi.fdi.FileNameLength/2, fdi.fdi.FileName);
n++;
}
ret = 0;
NtClose(hdir);
}
if (ret)
do_err("scandir: %s", strerror(errno));
end = time_monotonic_nsec();
printf("%d files ", n);
if (verbose)
printf("scandir(%gms) ", ((double)end-start)/NSEC_PER_MS);
start = time_monotonic_nsec();
for (i=0; i<n; i++)
{
d = namelist[i];
if (!strcmp(".", d->d_name) || !strcmp("..", d->d_name))
continue;
sprintf(fullname, "%s/%s", dir, d->d_name);
if (verbose)
printf("%s\n", fullname);
if (!strcmp(method, "QFAF"))
ret = query_file_by_name(fullname);
else if (!strcmp(method, "QIF"))
ret = query_file_by_handle(fullname, 0);
else if (!strcmp(method, "QIFR"))
ret = query_file_by_handle(fullname, 1);
else if (!strcmp(method, "QDF"))
ret = query_file_by_dir_handle(dir, d->d_name);
else if (!strcmp(method, "lstat"))
ret = lstat(fullname, &buf);
else if (!strcmp(method, "KQDF"))
ret = query_file_kernel(dir, d->d_name, DIR_HANDLE_MODE);
else if (!strcmp(method, "KQIF"))
ret = query_file_kernel(dir, d->d_name, FILE_HANDLE_MODE);
else
do_err("unknow method %s", method);
if (ret)
do_err("stat %s: %s", d->d_name, strerror(errno));
free(d);
}
free(namelist);
end = time_monotonic_nsec();
printf("stat() %4.4gms, per file: %4.4gms ",
((double)end-start)/NSEC_PER_MS,
((double)end-start)/n/NSEC_PER_MS);
printf("\n");
if (hdrv)
NtClose(hdrv);
return 0;
}
============================================================
cygdrv.c
============================================================
#include <wdm.h>
#include "cygwin_structs.h"
#define ALLOC_TAG 'INOY'
#define do_err(...) \
do { \
DbgPrint(__VA_ARGS__); \
} while (0)
NTSTATUS ZwQueryDirectoryFile(HANDLE FileHandle, HANDLE Event,
PIO_APC_ROUTINE ApcRoutine, PVOID ApcContext,
PIO_STATUS_BLOCK IoStatusBlock, PVOID FileInformation, ULONG Length,
FILE_INFORMATION_CLASS FileInformationClass, BOOLEAN ReturnSingleEntry,
PUNICODE_STRING FileName, BOOLEAN RestartScan);
static NTSTATUS CygCreate(IN PDEVICE_OBJECT fdo, IN PIRP irp)
{
irp->IoStatus.Status = STATUS_SUCCESS;
irp->IoStatus.Information = 0;
IoCompleteRequest(irp,IO_NO_INCREMENT);
return STATUS_SUCCESS;
}
static NTSTATUS CygClose(IN PDEVICE_OBJECT fdo, IN PIRP irp)
{
irp->IoStatus.Status = STATUS_SUCCESS;
irp->IoStatus.Information = 0;
IoCompleteRequest(irp,IO_NO_INCREMENT);
return STATUS_SUCCESS;
}
static NTSTATUS get_dir_handle(HANDLE *hdir, query_file_t *query_file)
{
NTSTATUS ret;
UNICODE_STRING udir;
OBJECT_ATTRIBUTES attr;
IO_STATUS_BLOCK io;
udir.Buffer = query_file->dir_name;
udir.Length = wcslen(udir.Buffer) * sizeof(WCHAR);
udir.MaximumLength = udir.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &udir, 0, NULL, NULL);
if ((ret = ZwOpenFile(hdir, SYNCHRONIZE|FILE_LIST_DIRECTORY, &attr,
&io,
FILE_SHARE_VALID_FLAGS, FILE_SYNCHRONOUS_IO_NONALERT
|FILE_OPEN_FOR_BACKUP_INTENT|FILE_DIRECTORY_FILE)))
{
do_err("NtOpenFile failed (%ls): %d", query_file->dir_name, ret);
}
return ret;
}
static NTSTATUS query_file_by_handle(file_info_t *file_info,
query_file_t *query_file)
{
int ret;
OBJECT_ATTRIBUTES attr;
UNICODE_STRING ufile;
WCHAR *wfile;
IO_STATUS_BLOCK io;
HANDLE h = NULL;
wfile = ExAllocatePoolWithTag(NonPagedPool,
(wcslen(query_file->dir_name) +
wcslen(query_file->file_name[0])) * sizeof(WCHAR) + 4, ALLOC_TAG);
wcscpy(wfile, query_file->dir_name);
wcscat(wfile, L"\\");
wcscat(wfile, query_file->file_name[0]);
ufile.Buffer = wfile;
ufile.Length = wcslen(ufile.Buffer) * sizeof(WCHAR);
ufile.MaximumLength = ufile.Length + sizeof(WCHAR);
InitializeObjectAttributes(&attr, &ufile, 0, NULL, NULL);
if (ZwCreateFile(&h, SYNCHRONIZE | GENERIC_READ, &attr, &io, NULL, 0,
FILE_SHARE_VALID_FLAGS, FILE_OPEN,
FILE_OPEN_REPARSE_POINT|FILE_OPEN_FOR_BACKUP_INTENT, NULL, 0))
{
do_err("NtCreateFile failed (%ls)\n", wfile);
goto Exit;
}
if (ret = ZwQueryInformationFile(h, &io, &file_info->fai,
sizeof(file_info->fai) + sizeof(file_info->namebuf2),
FileAllInformation))
{
do_err("ZwQueryInformationFile failed (%ls): %d\n", wfile, ret);
}
Exit:
ExFreePoolWithTag(wfile, ALLOC_TAG);
if (h)
NtClose(h);
return 0;
}
static NTSTATUS query_file_by_dir_handle(file_info_t *file_info,
query_file_t *query_file)
{
NTSTATUS ret;
IO_STATUS_BLOCK io;
UNICODE_STRING file_mask;
HANDLE hdir;
if ((ret = get_dir_handle(&hdir, query_file)))
return ret;
file_mask.Buffer = query_file->file_name[0];
file_mask.Length = wcslen(file_mask.Buffer) * sizeof(WCHAR);
file_mask.MaximumLength = file_mask.Length + sizeof(WCHAR);
if (ret = ZwQueryDirectoryFile(hdir, NULL, NULL, 0, &io,
&file_info->fdi,
sizeof(file_info->fdi) + sizeof(file_info->namebuf),
FileIdBothDirectoryInformation, 1, &file_mask, 1))
{
do_err("NtQueryDirectoryFile failed (%ls)", query_file->file_name[0]);
}
ZwClose(hdir);
return ret;
}
static NTSTATUS CygDeviceControl(IN PDEVICE_OBJECT fdo, IN PIRP irp)
{
IO_STACK_LOCATION *irp_sp = IoGetCurrentIrpStackLocation(irp);
unsigned int code = irp_sp->Parameters.DeviceIoControl.IoControlCode;
void *data = irp->AssociatedIrp.SystemBuffer;
unsigned int len_in =
irp_sp->Parameters.DeviceIoControl.InputBufferLength;
unsigned int len_out =
irp_sp->Parameters.DeviceIoControl.OutputBufferLength;
int ret = STATUS_NOT_SUPPORTED, bytes_ret = 0;
file_info_t file_info;
query_file_t *query_file = data;
switch (code)
{
case QUERY_FILE_IOCTL:
if (len_out<sizeof(file_info_t) || len_in<sizeof(query_file_t))
{
ret = STATUS_BUFFER_TOO_SMALL;
goto Exit;
}
if (query_file->mode == DIR_HANDLE_MODE)
ret = query_file_by_dir_handle(&file_info, query_file);
else if (query_file->mode == FILE_HANDLE_MODE)
ret = query_file_by_handle(&file_info, query_file);
else
ret = STATUS_NOT_SUPPORTED;
if (!ret)
{
memcpy(data, &file_info, sizeof(file_info_t));
bytes_ret = sizeof(file_info_t);
}
break;
default:
ret = STATUS_NOT_SUPPORTED;
}
Exit:
irp->IoStatus.Status = ret;
irp->IoStatus.Information = bytes_ret;
IoCompleteRequest(irp,IO_NO_INCREMENT);
return ret;
}
static DEVICE_OBJECT *dev_obj;
static UNICODE_STRING nt_devid, win32_devid;
static VOID CygUnload(IN PDRIVER_OBJECT DriverObject)
{
IoDeleteSymbolicLink(&win32_devid);
IoDeleteDevice(dev_obj);
}
NTSTATUS DriverEntry(IN PDRIVER_OBJECT DriverObject,
IN PUNICODE_STRING RegistryPath)
{
int ret;
RtlInitUnicodeString(&nt_devid, L"\\Device\\Cygwin");
RtlInitUnicodeString(&win32_devid, L"\\DosDevices\\Cygwin");
if (!NT_SUCCESS(ret = IoCreateDevice(DriverObject, 0,
&nt_devid, FILE_DEVICE_SMB, 0, 0, &dev_obj)))
{
/* Either not enough memory to create a device object or another
* device object with the same name exits. This could happen if you
* install another instance of this device. */
do_err("IoCreateDevicefailed %x", ret);
return -1;
}
/* add create sym link */
if (!NT_SUCCESS(ret = IoCreateSymbolicLink(&win32_devid, &nt_devid)))
{
IoDeleteDevice(dev_obj);
do_err("IoCreateSymbolicLink failed %x", ret);
return -1;
}
DriverObject->DriverUnload = CygUnload;
DriverObject->MajorFunction[IRP_MJ_CREATE] = CygCreate;
DriverObject->MajorFunction[IRP_MJ_CLOSE] = CygClose;
DriverObject->MajorFunction[IRP_MJ_DEVICE_CONTROL] = CygDeviceControl;
return STATUS_SUCCESS;
}
========================================================
cygwin_structs.h
=======================================================
#ifndef __CYGWIN_STRUCTS__
#define __CYGWIN_STRUCTS__
#ifndef __KERNEL__
#include <winioctl.h>
#endif
#define CYG_CTL_CODE(function, method, access) \
CTL_CODE(0, function, method, access)
#define IOCTL_CYG(func) \
CYG_CTL_CODE(func, METHOD_BUFFERED, \
FILE_READ_ACCESS | FILE_WRITE_ACCESS)
#define QUERY_FILE_IOCTL IOCTL_CYG(0x1)
NTSTATUS NTAPI NtClose(HANDLE Handle);
typedef struct _FILE_INTERNAL_INFORMATION {
LARGE_INTEGER IndexNumber;
} FILE_INTERNAL_INFORMATION, *PFILE_INTERNAL_INFORMATION;
typedef struct _FILE_EA_INFORMATION {
ULONG EaSize;
} FILE_EA_INFORMATION, *PFILE_EA_INFORMATION;
typedef struct _FILE_ACCESS_INFORMATION {
ACCESS_MASK AccessFlags;
} FILE_ACCESS_INFORMATION, *PFILE_ACCESS_INFORMATION;
typedef struct _FILE_MODE_INFORMATION {
ULONG Mode;
} FILE_MODE_INFORMATION, *PFILE_MODE_INFORMATION;
typedef struct _FILE_ALIGNMENT_INFORMATION {
ULONG AlignmentRequirement;
} FILE_ALIGNMENT_INFORMATION, *PFILE_ALIGNMENT_INFORMATION;
typedef struct _FILE_NAME_INFORMATION {
ULONG FileNameLength;
WCHAR FileName[1];
} FILE_NAME_INFORMATION, *PFILE_NAME_INFORMATION;
typedef struct _FILE_ID_BOTH_DIRECTORY_INFORMATION {
ULONG NextEntryOffset;
ULONG FileIndex;
LARGE_INTEGER CreationTime;
LARGE_INTEGER LastAccessTime;
LARGE_INTEGER LastWriteTime;
LARGE_INTEGER ChangeTime;
LARGE_INTEGER EndOfFile;
LARGE_INTEGER AllocationSize;
ULONG FileAttributes;
ULONG FileNameLength;
ULONG EaSize;
CHAR ShortNameLength;
WCHAR ShortName[12];
LARGE_INTEGER FileId;
WCHAR FileName[1];
} FILE_ID_BOTH_DIRECTORY_INFORMATION, *PFILE_ID_BOTH_DIRECTORY_INFORMATION;
#ifdef __CYGWIN__
typedef struct _FILE_POSITION_INFORMATION {
LARGE_INTEGER CurrentByteOffset;
} FILE_POSITION_INFORMATION, *PFILE_POSITION_INFORMATION;
typedef struct _FILE_BASIC_INFORMATION {
LARGE_INTEGER CreationTime;
LARGE_INTEGER LastAccessTime;
LARGE_INTEGER LastWriteTime;
LARGE_INTEGER ChangeTime;
ULONG FileAttributes;
} FILE_BASIC_INFORMATION, *PFILE_BASIC_INFORMATION;
typedef struct _FILE_STANDARD_INFORMATION {
LARGE_INTEGER AllocationSize;
LARGE_INTEGER EndOfFile;
ULONG NumberOfLinks;
BOOLEAN DeletePending;
BOOLEAN Directory;
} FILE_STANDARD_INFORMATION, *PFILE_STANDARD_INFORMATION;
#endif
typedef struct _FILE_ALL_INFORMATION {
FILE_BASIC_INFORMATION BasicInformation;
FILE_STANDARD_INFORMATION StandardInformation;
FILE_INTERNAL_INFORMATION InternalInformation;
FILE_EA_INFORMATION EaInformation;
FILE_ACCESS_INFORMATION AccessInformation;
FILE_POSITION_INFORMATION PositionInformation;
FILE_MODE_INFORMATION ModeInformation;
FILE_ALIGNMENT_INFORMATION AlignmentInformation;
FILE_NAME_INFORMATION NameInformation;
} FILE_ALL_INFORMATION, *PFILE_ALL_INFORMATION;
#define DRV_MAX_PATH 512
#define DRV_NAME_MAX 255
#define DIR_HANDLE_MODE 1
#define FILE_HANDLE_MODE 3
typedef struct {
WCHAR dir_name[DRV_MAX_PATH];
int nfiles;
WCHAR file_name[3][DRV_MAX_PATH];
int mode;
} query_file_t;
typedef struct {
FILE_ID_BOTH_DIRECTORY_INFORMATION fdi;
WCHAR namebuf[DRV_NAME_MAX + 1];
FILE_STANDARD_INFORMATION fsi;
FILE_ALL_INFORMATION fai;
WCHAR namebuf2[DRV_NAME_MAX + 1];
} file_info_t;
#endif
On 5/10/2010 4:24 PM, Derry Shribman wrote:
Hi,
> I saw the numbers for 1.7.7 but I was talking about current CVS.
Yoni is currently busy working on a "cygwin kernel-mode accelerator"
prototype. Hopefully by tomorrow he will be able to post it, and also
test the performance numbers on the current CVS cygwin1.dll.
Derry
On 10/4/2010 12:23 PM, Corinna Vinschen wrote:
On Oct 3 23:33, Derry Shribman wrote:
I'm missing a comparison using the latest Cygwin from CVS. That's
much more interesting than 1.7.5.
All the comparisons where done BOTH on 1.7.5 AND on 1.7.7!
Here are the 1.7.7 results again:
I saw the numbers for 1.7.7 but I was talking about current CVS. Due to
Yoni's mail to cygwin-patches I already changed quite a few things. A
comparison should take the latest incarnation of the code into account
since, obviously, development is going into this direction.
Could you compile and run it and report the results on your PCs?
Compile the latest Cygwin from CVS and do the comparison on the machines
you already used. What sense does it make to add YA machine with
different hardware? Especially since all my Windows machines except for
a single netbook are running in VMs.
You also never replied to my mail describing the suffix problem when
using the NtQueryDirectoryFile function:
http://cygwin.com/ml/cygwin-patches/2010-q3/msg00073.html
Yoni was busy with preparing the stat performance test program, so I
will answer this one:
Calling NtQueryDirectoryFile() is implemented in the kernel by
IRP_MJ_DIRECTORY_CONTROL, and RestartScan is flags |=
SL_RESTART_SCAN, ReturnSingleEntry is flags |=
SL_RETURN_SINGLE_ENTRY.
What is missing is being able to set flags |= SL_INDEX_SPECIFIED
when calling from user-mode.
The WDK description does not say that SL_INDEX_SPECIFIED provides
this functionality and it also specifies that using the FileIndex
does not make any sense on NTFS. Which in turn makes sense, given
that NTFS keeps directories always in name sorted order.
So - as you wrote: This means that NtQueryDirectoryFile() should be
called with a 'fresh' handle every time.
Thats why in the performance test application Yoni wrote, a new
handle was called for every NtQueryDirectoryFile() - so that this
was taken into account.
That's not really the same. The test application does not take .lnk and
.exe suffixes into account. It just stats what it already got from a
former readdir. My private test within the Cygwin code itself was not
as positive and in fact slowed down `ls -l' since now there was YA
handle to open. But anyway. Maybe the method using a filter with
wildcards is a winner.
We have NOD32 anti-virus installed on all XP PCs at our work place.
So that may probably be one of the reasons for the XP slowdown. Can
you run the test program on an XP at your place and see how results
differ?
Well, I can try, but it's a VM, too.
However, it's easy to make a speed comparsion which ignores the
bunch of
problems Cygwin is fighting against. What about the problems of various
filesystems, for instance?
Cygwin anyway today caches the volume type. So it can know whether
it is NTFS/FAT without a system call.
No, not really. You can make this point for the drive which hosts the
systemroot, but every other drive (or junction) can change at any given
time.
This requires at least one open handle to the volume to be sure to have
the right filesystem type. So, whatever you do, you need to open a
handle first. Either to the parent dir or to the file. This is kind of
a chicken-egg problem.
This is exactly what Yoni cached in the patch he submitted to
cygwin-patches!
I didn't look into the patch for the well-known reason. The description
in Yoni's mail implies that the caching is based on the path, just like
in 1.5.x. That's correct only in an environment which you control
yourself, but it's not necessarily correct for any other environment.
That's why I added a new caching algorithm to current CVS which is based
on the results of a single FileFsVolumeInformation result. This still
requires an open handle, either of the file itself, or of it's parent
dir.
Corinna