Twisted: Ticket #3454: FilePath operations which are atomic on POSIX should be atomic on Windowshttps://twistedmatrix.com/trac/ticket/3454
<p>
<tt>os.rename</tt> doesn't overwrite on Windows, and so in <tt>FilePath.setContent</tt> we do a crummy platform check and delete the file in advance of moving the new one over it.
</p>
<p>
However, <a class="ext-link" href="http://msdn.microsoft.com/en-us/library/aa365512(VS.85).aspx"><span class="icon">​</span>ReplaceFile</a> and <a class="ext-link" href="http://msdn.microsoft.com/en-us/library/aa365240(VS.85).aspx"><span class="icon">​</span>MoveFileEx with the MOVEFILE_REPLACE_EXISTING flag</a> <em>do</em> overwrite, and we could use one of those in the implementation of <tt>moveTo</tt> instead.
</p>
en-usTwistedhttps://twistedmatrix.com/trac/chrome/common/trac_banner.pnghttps://twistedmatrix.com/trac/ticket/3454
Trac 1.0.1exarkunWed, 01 Oct 2008 02:31:09 GMThttps://twistedmatrix.com/trac/ticket/3454#comment:1
https://twistedmatrix.com/trac/ticket/3454#comment:1
<p>
Snag: there are currently no test coverage for the atomicity of any operations on any platforms. It would be simpler to provide consistency by guaranteeing nothing. :)
</p>
TicketteratornWed, 01 Oct 2008 03:22:09 GMTcc sethttps://twistedmatrix.com/trac/ticket/3454#comment:2
https://twistedmatrix.com/trac/ticket/3454#comment:2
<ul>
<li><strong>cc</strong>
<em>teratorn</em> added
</li>
</ul>
<p>
I'm pretty sure than an atomic rename is impossible on win32 without using the transactional filesystem APIs available in Vista and Server 2008.
</p>
<p>
Some time ago I researched this question very deeply. See <a class="closed ticket" href="https://twistedmatrix.com/trac/ticket/1705" title="defect: Test failure on win32 (FilePathTestCase.testGetAndSet) (closed: fixed)">#1705</a>
</p>
<p>
Using a move-and-replace API doesn't really buy us anything over using the current delete-then-rename scheme. It can still fail and leave the filesystem in an inconsistent state.
</p>
<p>
Providing atomic semantics on Vista and Server 2008 would be *nice*, I guess, but perhaps impractical. Is there any way to call MoveFileTransacted without introducing dependencies? On Python 2.3 and up?
</p>
TicketspivWed, 01 Oct 2008 05:08:23 GMThttps://twistedmatrix.com/trac/ticket/3454#comment:3
https://twistedmatrix.com/trac/ticket/3454#comment:3
<p>
I think teratorn is right: this isn't feasible.
</p>
<p>
AFAIK, these APIs can all fail if the destination file to be overwritten is in use. That's fundamentally unlike POSIX, and that part of POSIX is a big part of what's so useful about atomic rename.
</p>
<p>
If you find a way to do this, I'd love to know about it; it'd be useful for bzr... we'd be able to get rid of bzrlib.osutils.fancy_rename. But I don't believe it is possible.
</p>
TicketspivWed, 01 Oct 2008 05:08:32 GMTcc changedhttps://twistedmatrix.com/trac/ticket/3454#comment:4
https://twistedmatrix.com/trac/ticket/3454#comment:4
<ul>
<li><strong>cc</strong>
<em>spiv</em> added
</li>
</ul>
TicketexarkunFri, 03 Oct 2008 14:54:21 GMTcc changedhttps://twistedmatrix.com/trac/ticket/3454#comment:5
https://twistedmatrix.com/trac/ticket/3454#comment:5
<ul>
<li><strong>cc</strong>
<em>exarkun</em> added
</li>
</ul>
<p>
We can use pywin32 to call pretty much any Windows APIs we like. Twisted on Windows requires pywin32 for most things already, so it's not even a new dependency. ctypes is another possibility.
</p>
<p>
Of course, it still <em>may</em> be impossible, but I know that there's room for improvement from the current situation.
</p>
TicketteratornFri, 03 Oct 2008 21:10:41 GMThttps://twistedmatrix.com/trac/ticket/3454#comment:6
https://twistedmatrix.com/trac/ticket/3454#comment:6
<p>
Replying to <a class="closed" href="https://twistedmatrix.com/trac/ticket/3454#comment:5" title="Comment 5 for Ticket #3454">exarkun</a>:
</p>
<blockquote class="citation">
<p>
We can use pywin32 to call pretty much any Windows APIs we like. Twisted on Windows requires pywin32 for most things already, so it's not even a new dependency.
</p>
</blockquote>
<p>
Yep, we can use it to call MoveFileEx, but it doesn't have a binding for MoveFileTransacted.
</p>
<blockquote class="citation">
<p>
ctypes is another possibility.
</p>
</blockquote>
<p>
Only problem is that it's only available in the py 2.5 stdlib and up.
</p>
<blockquote class="citation">
<p>
Of course, it still <em>may</em> be impossible, but I know that there's room for improvement from the current situation.
</p>
</blockquote>
<p>
For fun and future reference:
<a class="ext-link" href="http://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows"><span class="icon">​</span>http://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows</a>
</p>
<p>
I'm convinced that it's impossible to have a real atomic operation without MoveFileTransacted.. I think the only reasonable thing to do is to assume that this is the case until proven otherwise.
</p>
<p>
So, should we use MoveFileEx with MOVEFILE_REPLACE_EXISTING as an improvement over separate delete and rename calls? If so, should this be a hard dependency on pywin32 or should it fall back to delete/rename silently?
</p>
<p>
My vote is for doing this, and making it a hard dependency. I see pywin32 as a standard library that just wasn't included... and as you say it's already depended on for core functionality (spawning processes, etc). Also, I've often wanted to use pywin32 for things, by shyed away from it because it wasn't an official dependency.
</p>
<p>
Should we check for ctypes and Vista/2008, and in that case use a transactional move? If so, how can we do the platform check? Or perhaps ctypes can just tell us if the function exists.
</p>
<p>
This would be a pretty simple block of code, with no maintenance requirements, assuming we don't screw up the platform check. And it would provide a valuable improvement for folks targetting up-and-coming Windows platforms. Any objections?
</p>
TicketkhornWed, 13 Jan 2010 23:10:28 GMThttps://twistedmatrix.com/trac/ticket/3454#comment:7
https://twistedmatrix.com/trac/ticket/3454#comment:7
<p>
This appears to be a duplicate of <a class="new ticket" href="https://twistedmatrix.com/trac/ticket/3004" title="defect: FilePath.setContent should do an atomic rename on Windows where possible (new)">#3004</a>
</p>
TicketglyphMon, 24 May 2010 02:06:11 GMTstatus changed; resolution sethttps://twistedmatrix.com/trac/ticket/3454#comment:8
https://twistedmatrix.com/trac/ticket/3454#comment:8
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>duplicate</em>
</li>
</ul>
<p>
Replying to <a class="closed" href="https://twistedmatrix.com/trac/ticket/3454#comment:7" title="Comment 7 for Ticket #3454">khorn</a>:
</p>
<blockquote class="citation">
<p>
This appears to be a duplicate of <a class="new ticket" href="https://twistedmatrix.com/trac/ticket/3004" title="defect: FilePath.setContent should do an atomic rename on Windows where possible (new)">#3004</a>
</p>
</blockquote>
<p>
At first I thought maybe not because this is about everything and that's just about <tt>setContent</tt>, but really whatever addresses this is going to need to deal with multiple methods. So yeah, let's just have one ticket.
</p>
Ticket<automation>Mon, 14 Feb 2011 04:30:49 GMTowner deletedhttps://twistedmatrix.com/trac/ticket/3454#comment:9
https://twistedmatrix.com/trac/ticket/3454#comment:9
<ul>
<li><strong>owner</strong>
<em>glyph</em> deleted
</li>
</ul>
Ticket