This is the mail archive of the cygwin mailing list for the Cygwin project.


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]
Other format: [Raw text]

Re: [1.7.0-50] scp progress counter flies through first 175 MB or so


On Thu, Jun 25, 2009 at 04:36:51PM +0200, Corinna Vinschen wrote:
>On Jun 25 10:05, Christopher Faylor wrote:
>> On Thu, Jun 25, 2009 at 12:10:39PM +0200, Corinna Vinschen wrote:
>> >On Jun 24 13:17, Andrew Schulman wrote:
>> >> Here's an odd one.
>> >> 
>> >> Using openssh 5.2p1-2 with Cygwin 1.7.0-50, when I scp any file, the
>> >> progress counter appears to show ridiculously fast transfer rates, e.g.
>> >> about 50 MB/s over a 750 KB/s connection, for the first 175 MB or so. After
>> >> that the counter settles down to normal speed.  Then when the counter
>> >> reaches the end, it "hangs" at 100% for the remaining time while the copy
>> >> finishes.
>> >> 
>> >> At first I thought that the copy itself was being corrupted in the first
>> >> 175 MB, but I'm no longer able to reproduce that.  I believe now that the
>> >> copy is good and it's only the progress counter that's wrong.
>> >> 
>> >> When I revert to Cygwin 1.7.0-49, this problem doesn't occur.
>> >
>> >I can reproduce that copying a file via scp from a Windows machine to
>> >a Linux box.
>> >
>> >It looks like the pipes between the local scp and the local ssh are now
>> >exchanging the data quicker at the start than the ssh socket can send
>> >them to the remote machine.  On my XP machine, scp advances quickly by
>> >about 260 Megs (hard to tell, maybe it's exaclty 256 Megs for some
>> >reason?), then keeps the advance roughly at that value until scp
>> >finished.  At the end scp is just waiting for ssh which still has to
>> >send the 256/260 Megs of data.
>> >
>> >This is really weird, given that Cygwin does not create such a big
>> >buffer for the pipe.  Consequentially Task Manager claims that the
>> >memory is neither taken by scp, nor by ssh.  Both processes have normal
>> >VM sizes < 10 Megs.  Per Task manager the memory is paged Kernel Memory.
>> >A strange side effect is that the entire time taken by the data 
>> >transmission is longer than with -49, by almost exactly the time it
>> >takes to empty the big kernel cache.
>> >
>> >Puzzeling.
>> 
>> Is ssh using non-blocking pipes opened for write?  Until a week or two
>> ago, Cygwin didn't support those and treated the non-blocking write as a
>> blocking write.
>
>scp switches the pipes to non-blocking and then tries to do blocking io
>on its own, using the poll() function.  It calls a function called scpio
>which basically work like this:
>
>  scpio (io_function, fd, buf, size)
>  {
>    for (offset = 0; offset < size;) {
>      r = io_function (fd, buf + offset, size - offset);
>      [...]
>      if (r < 0) {
>	if (errno == EINTR)
>	  continue;
>	if (errno == EAGAIN || errno == EWOULDBLOCK) {
>	  poll (fd, 1, -1) /* Use poll() for blocking */
>	  continue;
>	}
>	[...]
>      }
>      offset += r;
>    }
>  }
>
>Looks like scp now stumbles over the pipe select() implementation.

Yes.  Grumble.  That's a bad interaction between non-blocking writes and
our stupid-thanks-to-Microsoft select implementation.  I think I can
work around this particular problem though.  I'll get to that this
weekend.

cgf

--
Problem reports:       http://cygwin.com/problems.html
FAQ:                   http://cygwin.com/faq/
Documentation:         http://cygwin.com/docs.html
Unsubscribe info:      http://cygwin.com/ml/#unsubscribe-simple


Index Nav: [Date Index] [Subject Index] [Author Index] [Thread Index]
Message Nav: [Date Prev] [Date Next] [Thread Prev] [Thread Next]