feature: kcp-operator overhaul

[xrstf]

Feature Description

Over the last year(s) we have identified a number of shortcomings and design mistakes in the kcp-operator. This is to a large extent because we designed it when we (well, I) knew very little about kcp's architecture.

The Problems

Here's a non-exhaustive list of things that are sub optimal:

• Client CAs: The kcp-operator deploys separate CAs for the front-proxy and the shards. This makes kubeconfigs not re-usable, which is super mega annoying when you want to use your Kubeconfig to watch an EndpointSlice and then connect to the URLs in it. We did this because we thought "more CAs = more security", but maybe we went overboard here and should have just deployed one single client CA.
• Kubernetes naming scheme: The scheme (or kind of lack thereof) of how the operator names deployments, certificates etc. is inconsistent and can easily lead to naming conflicts. We should overhaul this and now, with the knowledge of what resources we actually use, clean it up.
• URL configuration: kcp has 3 different CLI flags for controlling the URLs on shards, and the kcp-operator currently does a (and this is directed at me) piss-poor job at abstracting this away from the enduser. Even when I look through the flags today, I am unsure how to use the kcp-operator to deploy my kcp. We should completely rethink how URLs are configured. One idea @ntnn and me had sketched out was that FPs, Shards and VWs all only have a single spec.url field and nothing else. And the operator will figure out which value to use where.

Proposed Solution

The Goal

Fix all of these issues.

We must ensure that existing kcp setups can be migrated somehow to whatever our new operator will be. etcd is standalone, so the data should be fine, but we must also keep all PKIs alive so that we do not invalidate existing kubeconfigs. Telling folks to just start over is not an option anymore for this project.

The only real change I can currently see for our CRDs is the introduction of spec.url, so we settled for now on the plan of keeping v1alpha1, deprecating the old fields and introducing the new spec.url field. This way we prevent all the excessive work that is part of introducing your first 2nd API version.

Alternative Solutions

No response

Want to contribute?

• I would like to work on this issue.

Additional Context

No response

[ntnn]

On spec.url: The problem is that what happens if the user deploys a single RootShard and later adds a FrontProxy? If the FrontProxy configures a different URL the RootShard will have to be deployed even though nothing on the CR itself changed.

I could also see something like this:

kind: RootShard
metadata:
	name: root
spec:
	shard:
		base: https://root.kcp.platform/
		external: https://kcp.platform/
		vw: https://vw.root.kcp.platform/
---
kind: FrontProxy
spec:
	rootShard:
		ref:
			name: root
---
kind: Shard
metadata:
	name: second
spec:
	rootShard:
		ref:
			name: root
	shard:
		base: https://second.kcp.platform/
		vw: https://vw.second.kcp.platform/
		# Shard gets external/front-proxy URL from root shard

Though the shard property feels too nondescript :D

[xrstf]

I am really struggeling with the term "base". A "base URL" to me indicates "oh, this is what all the other strings will be appended to, like a prefix", but in reality it seems to be kind of sort of the internal URL, that we refuse to call "internal" because everything is meant to be externally public.

[ntnn]

I agree, the naming is not good and probably came from a different idea.
We could rename them in kcp. I think --shard-virtual-workspace-url is fine.
Instead of --shard-base-url we could just name it --shard-url? Should be easy enough to add that and mark --shard-base-url deprecated.
We could deprecate --shard-external-url and add --front-proxy-url to make it explicit and default that to --shard-url.
wdyt?

[xrstf]

How exactly kcp names its flags, I do not care tooo much about. IMHO the operator should/can abstract these flags away and offer a nice CRD to the admins. And that's where I would, if necessary, rename things to make them better.

Fixing upstream is of course also nice.

[ntnn]

Maybe something like this:

kind: RootShard
metadata:
  name: root
spec:
  urls:
    shard: https://root.kcp.platform/
    front-proxy: https://kcp.platform/
    virtual-workspace: https://vw.root.kcp.platform/
---
kind: FrontProxy
spec:
  rootShard:
    ref:
      name: root
---
kind: Shard
metadata:
  name: second
spec:
  rootShard:
    ref:
      name: root
  urls:
    shard: https://second.kcp.platform/
    virtual-workspace: https://vw.second.kcp.platform/
    # Shard gets front-proxy URL from root shard

I feel like the core config should stay with the root shard and downstream should intuitively know that when they are adding a front proxy they will also change the root shard.

[gman0]

Just a small thing I ran into (since there is an overhaul anyway):

The DeploymentSpecTemplate is missing Replicas:

kcp-operator/sdk/apis/operator/v1alpha1/common.go

Lines 328 to 331 in 11ec4bc

	type DeploymentSpecTemplate struct {
	// Template describes the pods that will be created.
	Template *PodTemplateSpec `json:"template,omitempty"`
	}

Instead, there are a couple of resources that have number of replicas as a dedicated field:

• FrontProxySpec:
  

  kcp-operator/sdk/apis/operator/v1alpha1/frontproxy_types.go

  Lines 24 to 30 in 11ec4bc

  	// FrontProxySpec defines the desired state of FrontProxy.
  	// +kubebuilder:validation:XValidation:rule="!(has(self.externalHostname) && has(self.external.hostname))",message="Cannot set both ExternalHostname (deprecated) and External.hostname. Use External.hostname only."
  	type FrontProxySpec struct {
  	// RootShard configures the kcp root shard that this front-proxy instance should connect to.
  	RootShard RootShardConfig `json:"rootShard"`
  	// Optional: Replicas configures the replica count for the front-proxy Deployment.
  	Replicas *int32 `json:"replicas,omitempty"`

• RootShardProxySpec:
  

  kcp-operator/sdk/apis/operator/v1alpha1/rootshard_types.go

  Lines 43 to 48 in 11ec4bc

  	type RootShardProxySpec struct {
  	// Optional: Image allows to override the container image used for this proxy.
  	Image *ImageSpec `json:"image,omitempty"`
  	// Optional: Replicas configures how many instances of this proxy run in parallel. Defaults to 2 if not set.
  	Replicas *int32 `json:"replicas,omitempty"`

• CommonShardSpec:
  

  kcp-operator/sdk/apis/operator/v1alpha1/shard_types.go

  Lines 35 to 51 in 11ec4bc

  	type CommonShardSpec struct {
  	// ClusterDomain is the DNS domain for services in the cluster. Defaults to "cluster.local" if not set.
  	// +optional
  	ClusterDomain string `json:"clusterDomain,omitempty"`
  	// ShardBaseURL is the base URL under which this shard should be reachable. This is used to configure
  	// the external URL. If not provided, the operator will use kubernetes service address to generate it.
  	// +optional
  	ShardBaseURL string `json:"shardBaseURL,omitempty"`
  	// Etcd configures the etcd cluster that this shard should be using.
  	Etcd EtcdConfig `json:"etcd"`
  	Image *ImageSpec `json:"image,omitempty"`
  	// Replicas configures how many instances of this shard run in parallel. Defaults to 2 if not set.
  	Replicas *int32 `json:"replicas,omitempty"`

I understand this may be an intentional design, in which case sorry for the noise!

[xrstf]

Yeah this was intentional, the explicit replicas field was first and only later did we add the DeploymentTemplates.

I have no good strong reasons to still have the dedicated fields, except that scaling a Deployment is such a common thing, I would not want to force users to resort to tweaking the Deployment via the template. I would consider the templates to be for "advanced usescases" only.

But I'm open to feedback how this feels in production :-)

[ntnn]

My thoughts on CAs:

I think we should have a "global" client CA that is recognized by front-proxy and shards to allow users to create a kubeconfig to manager their kcp instance without a hassle.
I also think it would be good to have shard-specific client CAs so if downstream wishes they can have operators that should only work on a single shard (e.g. VW deployed alongside the shard) and give them a kubeconfig that really is only valid for that shard.

Not sure though how often the per-shard client CA would be something people actually use.

[xrstf]

So you mean configuring a CA bundle for the --client-ca on each shard?

[ntnn]

Yeah; so each shard would have it's own client CA and it's own CA bundle with the "global" client CA.
I can see uses for that but as said - not sure if anyone would actually use it.

We could also just leave that as an option to be implemented by interested parties.