[PULP-1243] High memory consumption when syncing a Oracle remote repository.

[decko]

Version
"rpm": "3.31.2"
"core": "3.84.1"

Describe the bug
When syncing the https://yum.oracle.com/repo/OracleLinux/OL9/developer/x86_64/ repository, the memory consumption can reach 10GB.

To Reproduce
Create a remote pointing to the Oracle repository, associate it with a local repository, and ask Pulp to sync it.

Additional context
I'm attaching a memray flamegraph profile in the hope it helps to deal with the issue.

memray-flamegraph-memray_profile.html

https://issues.redhat.com/browse/PULP-1243

[dralley]

A separate instance of the same issue, but relating to a Google repository:

Describe the bug
When syncing the https://packages.cloud.google.com/yum/repos/cloud-sdk-el9-x86_64/ repository, the memory consumption can reach 6GB.

To Reproduce
Create a remote pointing to the Google repository, associate it with a repository, and ask Pulp to sync it.

Additional context
I'm attaching a memray flamegraph profile in the hope it helps to deal with the issue.
memray-flamegraph-memray_profile.html

[dralley]

A separate instance of the same issue, but relating to an ElasticSearch repository:

Describe the bug
When syncing the https://artifacts.elastic.co/packages/8.x/yum repository, the memory consumption can reach 6GB.

[dralley]

A workaround that will work in most cases: Instead of using a mirror sync mode, use additive sync in combination with the retain_package_versions option. Example: with retain_package_versions=3

A document on various possible strategies for sync memory usage reduction: https://hackmd.io/6fwkDMOXRamBz27d5CM-IQ

[ianballou]

A Katello community user is running into (likely) this issue with the Elastic repository.

When I synced the 3 Elastic repositories by themselves, I noticed 10+ GB memory consumption, which is strange given the small size of the repositories. Syncing the 8.x repo took as long as RHEL 9 BaseOS, which is over 4x its size in RPMs.

Afterwards, I noticed also that the Pulpcore API was hanging onto 30% of my systems 23 GB of memory. Perhaps related to how Katello calls the Pulp API at sync time, but I'm not sure why the usage remained high.

I then synced the 8.x repo alone, and the OOM killer was invoked. The repository did sync, but my system became unstable.

The community user noticed memory usage jumping up to 80 GB when syncing their smart proxy. The smart proxy is likely syncing multiple different versions of the same Elastic 8.x repo at the same time, which could be exacerbating the issue.

https://community.theforeman.org/t/content-proxy-complete-sync-out-of-memory/42891/13

[dralley]

good god, oof. Well, I will take another look at this

[dralley]

So, looking at the comment on Sept 13, the "workaround" is highly effective. There's a lot of old, duplicate packages which don't need to be processed. Setting retain_package_versions to 3 results in syncs and publishes getting 4-10x faster and never going over 500mb of memory.

(rpmrepo) dalley@dalley-thinkpadt14gen5:~/devel/repos$ rpmrepo download elastic8 https://artifacts.elastic.co/packages/8.x/yum/ --only-metadata
error: Macro %add_sysuser has unterminated body
(rpmrepo) dalley@dalley-thinkpadt14gen5:~/devel/repos$ rpmrepo details elastic8
error: Macro %add_sysuser has unterminated body
========
Packages
========

Number of packages:                          2597     
Number of unique packages (latest version):  30       
Number of packages (latest 3 versions):      90       
Packages total size:                         464.98 GB
Packages total size (latest version):        5.47 GB  
Packages total size (latest 3 versions):     16.45 GB 

I still think we need to handle such cases better, but really you don't want users to be dealing with all of those old versions in the first place.