Django: Ticket #16926: Custom SQL with Windows or Mac end-of-lines fail with Python 3https://code.djangoproject.com/ticket/16926
<p>
In <code>core.managment.sql.custom_sql_for_model</code>. The execution of custom sql fails when the sql file has windows or mac line endings. The sql file is opened with universal newline so that windows or mac end-of-lines are automatically translated to unix end-of-lines. In python 2.7 if the universal newline mode passed to open() the file is opened in text mode, even if binary mode is set, as is the case here. In Python 3 universal newline only works if the file is opened in text mode. Specifying b and U in the mode apparently opens it in binary mode and thus ignores the universal newline mode.
</p>
<p>
The python re module only uses \n as end-of-line. So the \r in the window or mac sql file causes the regex to fail. Subsequently multiple SQL statements are passed to the backend. This fails in e.g. SQLite.
</p>
<p>
The attached patch changes the regex which splits the files.
</p>
<p>
Another option would be to use different open() calls depending on python the version. If the file is opened in text mode in Python 3 the decode method is missing on the returned string. In python 3 the encoding parameter should probably be used in that case.
</p>
en-usDjangohttps://www.djangoproject.com/s/img/site/hdr_logo.gifhttps://code.djangoproject.com/ticket/16926
Trac 1.2.2adsworthSat, 24 Sep 2011 21:54:42 GMTattachment sethttps://code.djangoproject.com/ticket/16926
https://code.djangoproject.com/ticket/16926
<ul>
<li><strong>attachment</strong>
set to <em>fix-also-split-on-carrige-return.diff</em>
</li>
</ul>
TicketadsworthSat, 24 Sep 2011 21:59:04 GMTowner changedhttps://code.djangoproject.com/ticket/16926#comment:1
https://code.djangoproject.com/ticket/16926#comment:1
<ul>
<li><strong>owner</strong>
changed from <em>nobody</em> to <em>adsworth</em>
</li>
</ul>
TicketAymeric AugustinSun, 25 Sep 2011 06:15:34 GMTneeds_docs, stage changedhttps://code.djangoproject.com/ticket/16926#comment:2
https://code.djangoproject.com/ticket/16926#comment:2
<ul>
<li><strong>needs_docs</strong>
set
</li>
<li><strong>stage</strong>
changed from <em>Unreviewed</em> to <em>Accepted</em>
</li>
</ul>
<p>
I think this is an artifact of the 2to3 conversion.
</p>
<p>
Trunk has:
</p>
<pre class="wiki"> fp = open(sql_file, 'U')
</pre><p>
while the py3k branch has:
</p>
<pre class="wiki"> fp = open(sql_file, 'rbU')
</pre><p>
As far as I can tell, it should be enough to change the mode flag to <code>'U'</code> in the py3k branch.
</p>
TicketadsworthSun, 25 Sep 2011 08:15:07 GMThttps://code.djangoproject.com/ticket/16926#comment:3
https://code.djangoproject.com/ticket/16926#comment:3
<p>
In theory, yes.
A py3k string, does not have a decode method though. So we need to pass the encoding parameter in case of py3k and call decode in case of py2. I'll create a new patch.
</p>
TicketAymeric AugustinSun, 25 Sep 2011 08:32:38 GMThttps://code.djangoproject.com/ticket/16926#comment:4
https://code.djangoproject.com/ticket/16926#comment:4
<p>
In fact, I was surprised by the <code>'rbU'</code> pattern that appears nowhere else in Django, because the <code>b</code> and <code>U</code> flags are contradictory.
</p>
<p>
It is intentional but apparently hackish: see the commit message of <a class="ext-link" href="https://bitbucket.org/loewis/django-3k-old/changeset/0d7900201511"><span class="icon">​</span>https://bitbucket.org/loewis/django-3k-old/changeset/0d7900201511</a>
</p>
<p>
It would be a good idea to check with MvL before changing this code again.
</p>
TicketadsworthMon, 26 Sep 2011 10:16:10 GMTattachment sethttps://code.djangoproject.com/ticket/16926
https://code.djangoproject.com/ticket/16926
<ul>
<li><strong>attachment</strong>
set to <em>16926-fix-custom_sql_for_model.patch</em>
</li>
</ul>
TicketadsworthMon, 26 Sep 2011 10:19:44 GMThttps://code.djangoproject.com/ticket/16926#comment:5
https://code.djangoproject.com/ticket/16926#comment:5
<p>
The new patch 16926-fix-custom_sql_for_model.patch now first tries to open the custom SQL file with the Python3 encoding parameter failing that it opens the file without the encoding parameter and then decodes the resulting string.
</p>
<p>
I'll mail MvL and ask him to comment on this patch.
</p>
TicketanonymousSun, 09 Oct 2011 11:57:51 GMThttps://code.djangoproject.com/ticket/16926#comment:6
https://code.djangoproject.com/ticket/16926#comment:6
<p>
Thanks for the patch. I have adopted it slightly as <a class="ext-link" href="https://bitbucket.org/loewis/django-3k/changeset/b61b9900f687"><span class="icon">​</span>https://bitbucket.org/loewis/django-3k/changeset/b61b9900f687</a>
</p>
<p>
I'll merge those back into the branch shortly.
</p>
TicketAymeric AugustinMon, 20 Aug 2012 21:03:35 GMTstatus changed; resolution sethttps://code.djangoproject.com/ticket/16926#comment:7
https://code.djangoproject.com/ticket/16926#comment:7
<ul>
<li><strong>status</strong>
changed from <em>new</em> to <em>closed</em>
</li>
<li><strong>resolution</strong>
set to <em>fixed</em>
</li>
</ul>
<p>
This was fixed in <a class="changeset" href="https://code.djangoproject.com/changeset/4e7f04cd" title="[py3] Fixed file.read().decode(), used codecs.open() instead">[4e7f04cd]</a>.
</p>
Ticket