Added dependency upgrades for python3.12 support

[shivakrishnaah]

This PR adds support for python 3.11 and 3.12 to mlserver.

As alibi-explain and alibi-detect runtimes required to be also upgraded to allow this support (and not yet released), we currently point to the master branches instead for installation.

Before a proper release of mlserver, we need to revert this change. This also required a proper release of the Alibi libraries.

The main changes were on TensorFlow and Pytorch upgrades to support this transition.

Also the CI now contains tests for python 3.9-3.12 which we officially support.

[CLAassistant]

All committers have signed the CLA.

[shivakrishnaah]

Pull Request

fixes #1926

Description

Upgraded most of the dependencies to support python3.12 version

Changes Made

Upgraded the dependencies and the python docker images

Checklist

• Code follows the project's style guidelines
• All tests related to the changes pass successfully
• Documentation is updated (if necessary)
• Code is reviewed by at least one other team member
• Any breaking changes are communicated and documented

[sakoush]

@shivakrishnaah many thanks for your great contribution to add support for python 3.12.

To keep the changes for this PR as minimal as possible, lets just add support for python 3.12 and keep the default version of mlserver at python 3.10 for now. We cannot upgrade everything to python 3.12 at the moment as this will break other workflows that we have.

So lets revert the changes to the CI / dockerfile.

I have triggered the CI anyway so that we can look at the tests status.

[sakoush]

@shivakrishnaah Many thanks for your recent changes. There are some issues with the CI (check here as an example):

pyproject.toml changed significantly since poetry.lock was last generated. Run poetry lock [--no-update] to fix the lock file.

The first thing that needs to be fixed perhaps on your PR is to run make lock at the top level to refresh poetry lock files for mlserver.

There are a few things to note:

• Torch min version required for py 3.12 support is 2.4, check here. This needs to be updated.
• Some tests are failing in my local env. They will need to be addressed.
• Some internal libraries (e.g. alibi and alibi-detect pin their TF dependencies to be <2.15 and therefore we will also have to upgrade these internal libraries before accepting this change (we will upgrade these libraries ourselves). Perhaps then do not look at changes / tests for these runtimes for now: mlserver-alibi-detect and mlserver-alibi-explain.

note: I did push a few commits for reference, I hope this is fine with you.

[shivakrishnaah]

@sakoush Thank you for taking the time to review my PR. I have pulled the changes from your branch and merged them with mine. Please let me know if there's anything else I need to resolve or review.

[sakoush]

LGTM. Thanks for the initial work @shivakrishnaah I have done some extra work to get it over the line in the last few days.

Some of the CI tasks still fail which I will investigate before merging.

[sakoush]

failing tests are all isolated in test_lib_path, we will look into it in a follow up PR.