Re: [1.7] Invalid UTF8 while creating a file -> cannot delete?

[Corinna Vinschen <corinna-cygwin at cygwin dot com>]

On Sep 22 17:12, Andy Koppe wrote:
> 2009/9/22 Corinna Vinschen:
> >> Therefore, when converting a UTF-16 Windows filename to the current
> >> charset, 0xDC?? words should be treated like any other UTF-16 word
> >> that can't be represented in the current charset: it should be encoded
> >> as a ^N sequence.
> >
> > How? ?Just like the incoming multibyte character didn't represent a valid
> > UTF-8 char, a single U+DCxx value does not represent a valid UTF-16 char.
> > Therefore, the ^N conversion will fail since U+DCxx can't be converted
> > to valid UTF-8.
> 
> True, but that's an implementation issue rather than a design issue,
> i.e. the ^N conversion needs to do the UTF-8 conversion itself rather
> than invoke the __utf8 functions. Shall I look into creating a patch?

Well, sure I'm interested to see that patch (lazy me), but please note
that we need a snail mailed copyright assignment per
http://cygwin.com/assign.txt from you before we can apply any significant
patches.  Sorry for the hassle.

Hmm... maybe it's not that complicated.  The ^N case checks for a valid
UTF-8 lead byte right now.  The U+DCxx case could be handled by
generating (in sys_cp_wcstombs) and recognizing (in sys_cp_mbstowcs) a
non-valid lead byte, like 0xff.

> >> This won't work correctly, because different POSIX filenames will map
> >> to the same Windows filename. For example, the filenames "\xC3\xA4"
> >> (valid UTF-8 for a-umlaut) and "\xC4" (invalid UTF-8 sequence that
> >> represents a-umlaut in 8859-1), will both map to Windows filename
> >> "U+00C4", i.e a-umlaut in UTF-16. Furthermore, after creating a file
> >> called "\xC4", a readdir() would show that file as "\xC3\xA4".
> >
> > Right, but using your above suggestion will also lead to another filename
> > in readdir, it would just be \x0e\xsome\xthing.
> 
> I don't think the suggestion above is directly relevant to the problem
> I tried to highlight here.
> 
> Currently, with UTF-8 filename encodings, "\xC3xA4" turns into U+00C4
> on disk, while "\xC4" turns into U+DCC4, and converting back yields
> the original separate filenames.

Well, right now it doesn't exactly.

> If I understand your proposal
> correctly, both "\xC3\xA4" and "\xC4" would turn into U+00C4, hence
> converting back would yield "\xC3\xA4" for both. This is wrong. Those
> filenames shouldn't be clobbering each other, and a filename shouldn't
> change between open() and readdir(), certainly not without switching
> charset inbetween.

I see your point.  I was more thinking along the lines of how likely
that clobbering is, apart from pathological testcases.

> Having said that, if you did switch charset from UTF-8 e.g. to
> ISO-8859-1, the on-disk U+DCC4 would indeed turn into
> "\x0E\xsome\xthing". However, that issue applies to any UTF-16

You don't have to switch the charset.  Assume you're using any
non-singlebyte charset in which \xC4 is the start of a double- or
multibyte sequence.  open ("\xC4"); close; readdir(); will return
"\x0E\xsome\xthing" on readdir.

Only singlebyte charsets are off the hook.  So, your proposal to switch
to the default ANSI codepage for the C locale would be good for most
western languages, but it would still leave the eastern language users
with double-byte charsets behind.

Note that I'm not as opposed to your proposal to use the ANSI codepage
as before this discussion.  But I would like to see that the solution
works for most eastern language users as well.


Corinna

-- 
Corinna Vinschen                  Please, send mails regarding Cygwin to
Cygwin Project Co-Leader          cygwin AT cygwin DOT com
Red Hat

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple