{"url":"https://api.github.com/repos/apache/spark/pulls/26194","id":330391306,"node_id":"MDExOlB1bGxSZXF1ZXN0MzMwMzkxMzA2","html_url":"https://github.com/apache/spark/pull/26194","diff_url":"https://github.com/apache/spark/pull/26194.diff","patch_url":"https://github.com/apache/spark/pull/26194.patch","issue_url":"https://api.github.com/repos/apache/spark/issues/26194","number":26194,"state":"closed","locked":false,"title":"[SPARK-29536][PYTHON] Upgrade cloudpickle to 1.1.1 to support Python 3.8","user":{"login":"HyukjinKwon","id":6477701,"node_id":"MDQ6VXNlcjY0Nzc3MDE=","avatar_url":"https://avatars0.githubusercontent.com/u/6477701?v=4","gravatar_id":"","url":"https://api.github.com/users/HyukjinKwon","html_url":"https://github.com/HyukjinKwon","followers_url":"https://api.github.com/users/HyukjinKwon/followers","following_url":"https://api.github.com/users/HyukjinKwon/following{/other_user}","gists_url":"https://api.github.com/users/HyukjinKwon/gists{/gist_id}","starred_url":"https://api.github.com/users/HyukjinKwon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/HyukjinKwon/subscriptions","organizations_url":"https://api.github.com/users/HyukjinKwon/orgs","repos_url":"https://api.github.com/users/HyukjinKwon/repos","events_url":"https://api.github.com/users/HyukjinKwon/events{/privacy}","received_events_url":"https://api.github.com/users/HyukjinKwon/received_events","type":"User","site_admin":false},"body":"### What changes were proposed in this pull request?\r\n\r\nInline cloudpickle in PySpark to cloudpickle 1.1.1. See https://github.com/cloudpipe/cloudpickle/blob/v1.1.1/cloudpickle/cloudpickle.py\r\n\r\nhttps://github.com/cloudpipe/cloudpickle/pull/269 was added for Python 3.8 support (fixed from 1.1.0). Using 1.2.2 seems breaking PyPy 2 due to cloudpipe/cloudpickle#278 so this PR currently uses 1.1.1.\r\n\r\nOnce we drop Python 2, we can switch to the highest version.\r\n\r\n### Why are the changes needed?\r\n\r\npositional-only arguments was newly introduced from Python 3.8 (see https://docs.python.org/3/whatsnew/3.8.html#positional-only-parameters)\r\n\r\nParticularly the newly added argument to `types.CodeType` was the problem (https://docs.python.org/3/whatsnew/3.8.html#changes-in-the-python-api):\r\n\r\n> `types.CodeType` has a new parameter in the second position of the constructor (posonlyargcount) to support positional-only arguments defined in **PEP 570**. The first argument (argcount) now represents the total number of positional arguments (including positional-only arguments). The new `replace()` method of `types.CodeType` can be used to make the code future-proof.\r\n\r\n### Does this PR introduce any user-facing change?\r\n\r\nNo.\r\n\r\n### How was this patch tested?\r\n\r\nManually tested. Note that the optional dependency PyArrow looks not yet supporting Python 3.8; therefore, it was not tested. See \"Details\" below.\r\n\r\n\r\n