Planning to port cqlsh to Python 3 (CASSANDRA-10190)

I propose porting cqlsh and cqlshlib to Python 3. End-of-life for Python 2.7
<https://legacy.python.org/dev/peps/pep-0373/> is currently planned for 1
January 2020. We should prepare to port the tool to a version of Python
that will be officially supported.
I'm seeking input on three questions:
- Should we port it to straight Python 3, or Python 2/3 cross compatible?
- How much more testing is needed?
- Can we wait until after 4.0 for this?
I have an implementation
<https://github.com/ptbannister/cassandra/tree/cqlshlib3> to go with my
proposal. In parallel with getting the dtest cqlsh_tests working again, I
ported cqlsh.py and cqlshlib to Python 3. It passes with almost all of the
dtests and the unittests, so it's in pretty good shape, although it's not
100% done (more on that below).
*Python 3 or 2/3 cross compatible?* There are plenty of examples of Python
libraries that are compatible with both Python 2 and Python 3 (notably the
Cassandra Python driver), so I think this is achievable. The question is,
do we want to pay the price of cross compatibility? If we write cqlsh to be
2/3 cross compatible, we'll carry a long term technical debt to maintain
that feature. The value of continuing to support Python 2 will diminish
over time. However, a cross compatible implementation may ease the
transition for some users, especially if there are users who have made
significant custom modifications to the Python 2.7 implementation of cqlsh,
so I think we must at least consider the question.
*What additional testing is needed before we could release it?* I used
coverage.py to check on the code coverage of our existing dtest cqlsh_tests
and cqlshlib unittests. There are several blind spots in our current
testing that should be addressed before we release a port of cqlsh. Details
of this are available on JIRA ticket CASSANDRA-10190
<https://issues.apache.org/jira/browse/CASSANDRA-10190> in the attachment
coverage_notes.txt
<https://issues.apache.org/jira/secure/attachment/12926015/coverage_notes.txt>.
Beyond that, I've made no efforts to test on platforms other than Ubuntu
and CentOS, so Windows testing is needed if we're making efforts to support
Windows. It would also be preferable for some real users to try out the
port before it replaces the Python 2.7 cqlsh in a release.
Besides this, there are a couple of test failures I'm still trying to
figure out, notably tests involving user defined map types (a task made
more interesting by Python's general lack of support for immutable map
types).
*Can we wait until after 4.0 for this?* I don't think it's reasonable to
try to release this with 4.0 given the current consensus around a feature
freeze in the next few months. My feeling is that our testers and
committers are already very busy with the currently planned changes for
4.0. I recommend planning toward a release to occur after 4.0. If we run up
against Python 2.7 EOL before we can cut the next release, we could
consider releasing a ported cqlsh independently, for installation through
distutils or pip.
Patrick Bannister