CI: add_version enhancements (#3475)

- Prevent OpenSSL from bumping across major versions
- Skip adding versions that already have pending PRs
- Copy/move patches when adding new releases in the same series

README.md

@@ -69,6 +69,7 @@ This project was forked from [rbenv](https://github.com/rbenv/rbenv) and
   * [Using Pyenv without shims](#using-pyenv-without-shims)
   * [Running nested shells from Python-based programs](#running-nested-shells-from-python-based-programs)
   * [Environment variables](#environment-variables)
+  * [Manual shell setup](#manual-shell-setup)
 * **[Development](#development)**
   * [Contributing](#contributing)
   * [Version History](#version-history)
@@ -177,134 +178,29 @@ which does install native Windows Python versions.
 ----
 
 The below setup should work for the vast majority of users for common use cases.
-See [Advanced configuration](#advanced-configuration) for details and more configuration options.
+See [Advanced configuration](#advanced-configuration)
+and specifically [Manual shell setup](#manual-shell-setup) for details and more configuration options.
 
-If `pyenv` is already on `PATH`, you can configure the relevant shell startup
+To add the suggested setup code to the startup files of the running shell,
-files automatically:
+run `<path/to/pyenv> --install`.
+Specifically:
+
+* If you installed Pyenv with the installer script:
+
+```sh
+~/.pyenv/bin/pyenv init --install
+```
+
+* If you installed Pyenv with Homebrew:
 
 ```sh
 pyenv init --install
 ```
 
-If `pyenv` is not on `PATH` yet, run the same command through the `pyenv`
-executable in your chosen installation directory.
-
-This uses the same shell detection as `pyenv init`. If a startup file already
-contains Pyenv-related configuration, the command refuses to edit it; review the
-file manually and run `pyenv init <shell>` to see the suggested setup.
-
 For Bash, avoid the automatic `--install` path if your `BASH_ENV` points to
 `.bashrc`; use the manual Bash instructions below so the `eval "$(pyenv init - bash)"`
 line only goes in your login startup file.
 
-#### Bash
-  <details>
-
-  Stock Bash startup files vary widely between distributions in which of them source
-  which, under what circumstances, in what order and what additional configuration they perform.
-  As such, the most reliable way to get Pyenv in all environments is to append Pyenv
-  configuration commands to both `.bashrc` (for interactive shells)
-  and the profile file that Bash would use (for login shells).
-
-  1. First, add the commands to `~/.bashrc` by running the following in your terminal:
-
-      ```bash
-      echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
-      echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
-      echo 'eval "$(pyenv init - bash)"' >> ~/.bashrc
-      ```
-  2. Then, if you have `~/.profile`, `~/.bash_profile` or `~/.bash_login`, add the commands there as well.
-     If you have none of these, create a `~/.profile` and add the commands there.
-
-     * to add to `~/.profile`:
-       ``` bash
-       echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.profile
-       echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.profile
-       echo 'eval "$(pyenv init - bash)"' >> ~/.profile
-       ```
-     * to add to `~/.bash_profile`:
-       ```bash
-       echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
-       echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
-       echo 'eval "$(pyenv init - bash)"' >> ~/.bash_profile
-       ```
-
-   **Bash warning**: There are some systems where the `BASH_ENV` variable is configured
-   to point to `.bashrc`. On such systems, you should almost certainly put the
-   `eval "$(pyenv init - bash)"` line into `.bash_profile`, and **not** into `.bashrc`. Otherwise, you
-   may observe strange behaviour, such as `pyenv` getting into an infinite loop.
-   See [#264](https://github.com/pyenv/pyenv/issues/264) for details.
-   
-   </details>
-   
-#### Zsh
-  
-  <details>
-  Add Pyenv startup commands to `~/.zshrc` by running the following in your terminal:
-  
-  ```zsh
-  echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
-  echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
-  echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
-  ```
-  
-  If you wish to get Pyenv in noninteractive login shells as well, also add the commands to `~/.zprofile` or `~/.zlogin`.
-  </details>
-
-#### Fish
-
-  <details>
-
-  1. If you have Fish 3.2.0 or newer, execute this interactively:
-     ```fish
-     set -Ux PYENV_ROOT $HOME/.pyenv
-     test -d $PYENV_ROOT/bin; and fish_add_path $PYENV_ROOT/bin
-     ```
-
-  2. Otherwise, execute the snippet below:
-     ```fish
-     set -Ux PYENV_ROOT $HOME/.pyenv
-     test -d $PYENV_ROOT/bin; and set -U fish_user_paths $PYENV_ROOT/bin $fish_user_paths
-     ```
-
-  3. Now, add this to `~/.config/fish/config.fish`:
-     ```fish
-     pyenv init - fish | source
-     ```
-  </details>
-
-#### Nushell
-
-  <details>
-
-  Add the following lines to your `config.nu` to add Pyenv and its shims to your `PATH`.
-  Shell integration (completions and subcommands changing the shell's state)
-  isn't currently supported.
-
-  ~~~ nu
-  $env.PYENV_ROOT = "~/.pyenv" | path expand
-  if (( $"($env.PYENV_ROOT)/bin" | path type ) == "dir") {
-    $env.PATH = $env.PATH | prepend $"($env.PYENV_ROOT)/bin" }
-  $env.PATH = $env.PATH | prepend $"(pyenv root)/shims"
-  ~~~
-
-  </details>
-
-#### Microsoft PowerShell
-
-  <details>
-
-  Add the commands to `$profile.CurrentUserAllHosts` by running the following in your terminal:
-
-  ~~~ pwsh
-  echo '$Env:PYENV_ROOT="$Env:HOME/.pyenv"' >> $profile.CurrentUserAllHosts
-  echo 'if (Test-Path -LP "$Env:PYENV_ROOT/bin" -PathType Container) { 
-    $Env:PATH="$Env:PYENV_ROOT/bin:$Env:PATH" }' >> $profile.CurrentUserAllHosts
-  echo 'iex ((pyenv init -) -join "`n")' >> $profile.CurrentUserAllHosts
-  ~~~
-
-  </details>
-
 ### C. Restart your shell
 ----
 
@@ -815,6 +711,127 @@ name | default | description
 See also [_Special environment variables_ in Python-Build's README](plugins/python-build/README.md#special-environment-variables)
 for environment variables that can be used to customize the build.
 
+
+### Manual shell setup
+
+Below is the suggested shell setup added to shell startup files by `pyenv init --install`.
+
+* To automatically install Pyenv for a shell different than the running shell, run
+
+```sh
+path/to/pyenv --install <shell executable name>
+```
+
+e.g. `~/.pyenv --install bash`.
+
+#### Bash
+  <details>
+
+  Stock Bash startup files vary widely between distributions in which of them source
+  which, under what circumstances, in what order and what additional configuration they perform.
+  As such, the most reliable way to get Pyenv in all environments is to append Pyenv
+  configuration commands to both `.bashrc` (for interactive shells)
+  and the profile file that Bash would use (for login shells).
+
+  1. First, add the commands to `~/.bashrc` by running the following in your terminal:
+
+      ```bash
+      echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bashrc
+      echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bashrc
+      echo 'eval "$(pyenv init - bash)"' >> ~/.bashrc
+      ```
+  2. Then, if you have `~/.profile`, `~/.bash_profile` or `~/.bash_login`, add the commands there as well.
+     If you have none of these, create a `~/.profile` and add the commands there.
+
+     * to add to `~/.profile`:
+       ``` bash
+       echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.profile
+       echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.profile
+       echo 'eval "$(pyenv init - bash)"' >> ~/.profile
+       ```
+     * to add to `~/.bash_profile`:
+       ```bash
+       echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
+       echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile
+       echo 'eval "$(pyenv init - bash)"' >> ~/.bash_profile
+       ```
+
+   **Bash warning**: There are some systems where the `BASH_ENV` variable is configured
+   to point to `.bashrc`. On such systems, you should almost certainly put the
+   `eval "$(pyenv init - bash)"` line into `.bash_profile`, and **not** into `.bashrc`. Otherwise, you
+   may observe strange behaviour, such as `pyenv` getting into an infinite loop.
+   See [#264](https://github.com/pyenv/pyenv/issues/264) for details.
+   
+   </details>
+   
+#### Zsh
+  
+  <details>
+  Add Pyenv startup commands to `~/.zshrc` by running the following in your terminal:
+  
+  ```zsh
+  echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.zshrc
+  echo '[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.zshrc
+  echo 'eval "$(pyenv init - zsh)"' >> ~/.zshrc
+  ```
+  
+  If you wish to get Pyenv in noninteractive login shells as well, also add the commands to `~/.zprofile` or `~/.zlogin`.
+  </details>
+
+#### Fish
+
+  <details>
+
+  1. If you have Fish 3.2.0 or newer, execute this interactively:
+     ```fish
+     set -Ux PYENV_ROOT $HOME/.pyenv
+     test -d $PYENV_ROOT/bin; and fish_add_path $PYENV_ROOT/bin
+     ```
+
+  2. Otherwise, execute the snippet below:
+     ```fish
+     set -Ux PYENV_ROOT $HOME/.pyenv
+     test -d $PYENV_ROOT/bin; and set -U fish_user_paths $PYENV_ROOT/bin $fish_user_paths
+     ```
+
+  3. Now, add this to `~/.config/fish/config.fish`:
+     ```fish
+     pyenv init - fish | source
+     ```
+  </details>
+
+#### Nushell
+
+  <details>
+
+  Add the following lines to your `config.nu` to add Pyenv and its shims to your `PATH`.
+  Shell integration (completions and subcommands changing the shell's state)
+  isn't currently supported.
+
+  ~~~ nu
+  $env.PYENV_ROOT = "~/.pyenv" | path expand
+  if (( $"($env.PYENV_ROOT)/bin" | path type ) == "dir") {
+    $env.PATH = $env.PATH | prepend $"($env.PYENV_ROOT)/bin" }
+  $env.PATH = $env.PATH | prepend $"(pyenv root)/shims"
+  ~~~
+
+  </details>
+
+#### Microsoft PowerShell
+
+  <details>
+
+  Add the commands to `$profile.CurrentUserAllHosts` by running the following in your terminal:
+
+  ~~~ pwsh
+  echo '$Env:PYENV_ROOT="$Env:HOME/.pyenv"' >> $profile.CurrentUserAllHosts
+  echo 'if (Test-Path -LP "$Env:PYENV_ROOT/bin" -PathType Container) { 
+    $Env:PATH="$Env:PYENV_ROOT/bin:$Env:PATH" }' >> $profile.CurrentUserAllHosts
+  echo 'iex ((pyenv init -) -join "`n")' >> $profile.CurrentUserAllHosts
+  ~~~
+
+  </details>
+
 ----
 
 ## Development

plugins/python-build/scripts/add_cpython.py

@@ -17,6 +17,7 @@ import os.path
 import pathlib
 import pprint
 import re
+import shutil
 import subprocess
 import sys
 import typing
@@ -46,6 +47,14 @@ EXCLUDED_VERSIONS= {
 here = pathlib.Path(__file__).resolve()
 OUT_DIR: pathlib.Path = here.parent.parent / "share" / "python-build"
 
+AUTO_ADD_VERSION_REF_RE = re.compile(
+    r"^(?P<object_id>[0-9a-f]{40,64})\t"
+    r"refs/heads/auto_add_version/(?P<versions>\S+)$"
+)
+OPENSSL_RELEASE_TAG_RE = re.compile(
+    r"^openssl-(?P<major>\d+)\.\d+(?:\.\d+)*(?:[a-z]+\d*)?$"
+)
+
 T_THUNK=\
 '''export PYTHON_BUILD_FREE_THREADING=1
 source "${BASH_SOURCE[0]%t}"
@@ -83,10 +92,13 @@ def adapt_script(version: packaging.version.Version,
                                  url=new_package_url+'#'+new_package_hash,
                                  verify_py_suffix=verify_py_suffix)
 
-        elif m:=re.match(r'\s*install_package\s+"(?P<package>openssl-\S+)"\s+'
+        elif m:=re.match(r'\s*install_package\s+'
-                       r'"(?P<url>\S+)"\s.*$',
+                         r'"(?P<package>openssl-(?P<openssl_major>\d+)\.\S+)"\s+'
+                         r'"(?P<url>\S+)"\s.*$',
                          line):
-            item = VersionDirectory.openssl.get_store_latest_release()
+            item = VersionDirectory.openssl.get_store_latest_release(
+                int(m.group('openssl_major'))
+            )
 
             line = Re.sub_groups(m,
                                package=item.package_name,
@@ -129,6 +141,8 @@ def add_version(version: packaging.version.Version):
         return False
     VersionDirectory.existing.append(_CPythonExistingScriptInfo(version,str(new_path)))
 
+    handle_version_patches(version, previous_version, is_prerelease_upgrade)
+
     cleanup_prerelease_upgrade(is_prerelease_upgrade, previous_version, version)
 
     handle_t_thunks(version, previous_version, is_prerelease_upgrade)
@@ -137,6 +151,51 @@ def add_version(version: packaging.version.Version):
     return True
 
 
+def handle_version_patches(
+        version: packaging.version.Version,
+        previous_version: packaging.version.Version,
+        is_prerelease_upgrade: bool)\
+        -> None:
+    if (previous_version.major, previous_version.minor) != (version.major, version.minor):
+        return
+
+    patches_dir = OUT_DIR / "patches"
+    previous_patches = patches_dir / str(previous_version)
+    if not previous_patches.exists():
+        return
+
+    new_patches = patches_dir / str(version)
+    if is_prerelease_upgrade:
+        logger.info(f"Git moving patches from {previous_version} to {version}")
+        subprocess.check_call((
+            "git", "-C", OUT_DIR, "mv",
+            f"patches/{previous_version}",
+            f"patches/{version}",
+        ))
+    else:
+        logger.info(f"Copying patches from {previous_version} to {version}")
+        shutil.copytree(previous_patches, new_patches)
+
+    previous_package_patches = new_patches / f"Python-{previous_version}"
+    new_package_patches = new_patches / f"Python-{version}"
+    if is_prerelease_upgrade:
+        subprocess.check_call((
+            "git", "-C", OUT_DIR, "mv",
+            f"patches/{version}/Python-{previous_version}",
+            f"patches/{version}/Python-{version}",
+        ))
+    else:
+        previous_package_patches.rename(new_package_patches)
+
+    previous_t_patches = patches_dir / f"{previous_version}t"
+    if previous_t_patches.exists() or previous_t_patches.is_symlink():
+        if is_prerelease_upgrade:
+            previous_t_patches.unlink()
+        (patches_dir / f"{version}t").symlink_to(
+            str(version), target_is_directory=True
+        )
+
+
 def cleanup_prerelease_upgrade(
         is_prerelease_upgrade: bool,
         previous_version: packaging.version.Version,
@@ -211,7 +270,11 @@ def main():
         VersionDirectory.available.get_store_available_source_downloads(release, True)
         del release
 
-    versions_to_add = sorted(VersionDirectory.available.keys() - VersionDirectory.existing.keys())
+    versions_to_add = sorted(
+        VersionDirectory.available.keys()
+        - VersionDirectory.existing.keys()
+        - get_pending_versions()
+    )
 
     logger.info("Versions to add:\n"+pprint.pformat(versions_to_add))
     result = False
@@ -219,6 +282,27 @@ def main():
         result = add_version(version_to_add) or result
     return int(not result)
 
+
+def get_pending_versions() -> typing.Set[packaging.version.Version]:
+    ls_remote = subprocess.check_output(
+        ("git", "-C", OUT_DIR, "ls-remote", "origin",
+         "refs/heads/auto_add_version/*"),
+        text=True,
+        timeout=30,
+    )
+
+    pending_versions = set()
+    for line in ls_remote.splitlines():
+        match = AUTO_ADD_VERSION_REF_RE.fullmatch(line)
+        if not match:
+            raise ValueError(f"Unexpected git ls-remote output: {line!r}")
+        pending_versions.update(
+            packaging.version.Version(version)
+            for version in match.group("versions").split("_")
+        )
+    return pending_versions
+
+
 def parse_args():
     parser = argparse.ArgumentParser(description=__doc__)
     parser.add_argument(
@@ -440,18 +524,41 @@ class _OpenSSLVersionInfo(typing.NamedTuple):
 class OpenSSLVersionsDirectory(KeyedList[_OpenSSLVersionInfo, packaging.version.Version]):
     key_field = "version"
 
-    def get_store_latest_release(self) \
+    def get_store_latest_release(self, major: int) \
             -> _OpenSSLVersionInfo:
-        if self:
+        matching = [
-            #already retrieved
+            release for release in self
-            return self[max(self.keys())]
+            if release.version.major == major
+        ]
+        if matching:
+            return max(matching, key=lambda release: release.version)
+
+        url = "https://api.github.com/repos/openssl/openssl/releases?per_page=100"
+        while url:
+            response = requests.get(url, timeout=30)
+            response.raise_for_status()
+            matching = [
+                release
+                for release in response.json()
+                if not release['draft']
+                and not release['prerelease']
+                and (match := OPENSSL_RELEASE_TAG_RE.fullmatch(
+                    release['tag_name']
+                ))
+                and int(match.group('major')) == major
+            ]
+            if matching:
+                j_release = matching[0]
+                break
+            url = response.links.get('next', {}).get('url')
+        else:
+            raise ValueError(f"No OpenSSL {major}.x release found")
 
-        j = requests.get("https://api.github.com/repos/openssl/openssl/releases/latest", timeout=30).json()
         # noinspection PyTypeChecker
         # urlparse can parse str as well as bytes
         shasum_url = more_itertools.one(
             asset['browser_download_url']
-            for asset in j['assets']
+            for asset in j_release['assets']
             if urllib.parse.urlparse(asset['browser_download_url']).path.split('/')[-1].endswith('.sha256')
         )
         shasum_text = requests.get(shasum_url, timeout=30).text
@@ -467,7 +574,7 @@ class OpenSSLVersionsDirectory(KeyedList[_OpenSSLVersionInfo, packaging.version.
 
         package_url = more_itertools.one(
             asset['browser_download_url']
-            for asset in j['assets']
+            for asset in j_release['assets']
             if urllib.parse.urlparse(asset['browser_download_url']).path.split('/')[-1] == package_filename
         )