refactor(discovery): use systemd D-Bus API

[maxbrunet]

This replaces the invocations of systemdctl CLI by the systemd D-Bus API.

This is still a polling approach, the module conveniently remembers the previous set of units and returns the changes only, allowing to trigger minimal discovery updates.

The D-Bus signals that would allow for this to be event-driven are unreliable, and although there is a workaround, I have observe many duplicated updates and easily fall into infinite loops of updates.

Using the D-Bus API permits to mount the D-Bus socket into the agent container to make it aware of the host systemd units. (note: I have also removed the /proc mount which is not needed, hostPID: true is enough)

Closes #760

[javierhonduco]

Nice work, will review it later today 🙇 .

(note: I have also removed the /proc mount which is not needed, hostPID: true is enough)

I think we still need it for the metadata providers, no?

[kakkoyun]

It always awesome to have one less external dependency 👍

[maxbrunet]

(note: I have also removed the /proc mount which is not needed, hostPID: true is enough)

I think we still need it for the metadata providers, no?

It was done to expose the kernel config, but after some testing, it has been proven unecessary: #892 (comment)

hostPID: true tells to run the pod in the host's PID namespace (same as --pid=host with docker run), so all information about the host's processes should be available to the agent.

From what I have documented here, everything gathered in /proc is under a PID, but we can check, which metadata provider are you worried?

[kakkoyun]

Let's merge and test it further.

[javierhonduco]

FYI, this feature uses ~15% of the CPU cycles as of 99d1c7927379775e328c9db0222dd6169db6a116 . Not sure how much we spent before spawning to systemctl but wanted to surface the current cost

https://pprof.me/8b20774

[kakkoyun]

Shall we open an issue for this?

[javierhonduco]

Might be worth it, as I am not sure of the previous performance implications so this might be a win, perf-wise, but I am worried about the extra GC pressure this is adding