WHAT SONOBUOY IS ? and more...

DEFINITION : Sonobuoy is a tool which makes it easier to understand the state of your Kubernetes cluster by running a set of plugins (plugins are set of tests, also Kubernetes conformance tests are a part of these plugins). It generates clear and informative reports of your cluster.
Conformant Cluster : When a cluster passes all the conformance tests suite ( subset of e2e tests which defines the core set of interoperable features that all conformant Kubernetes clusters must support) it is called conformant. And by that we can assure that our cluster is ready to handle production workloads, also if a workload is works on a conformant cluster then it should also work on any other cluster which is also conformant. Sonobuoy runs tests based on the version of kubernetes to prove conformance.
Cluster operators use sonobuoy to run entire conformance suite to verify that the cluster in a good state and that it is ready for production workload. And if the tests fails, sonobuoy can help to point out what is the cause of problem.
Service providers (like AKS, GKE, EKS etc.) can also run sonobuoy to find out that the clusters within their services are working as expected and passing the conformance tests.

Structure

There are mainly 4 components in sonobuoy. CLI is the part from which we interact. It creates an aggregator pod made with sonobuoy image, so we can say sonobuoy is running in your cluster and it is figuring out which plugins do we need to run for this cluster to prove conformace. Now this aggregator pod creates a pod for each plugin which contains two components which is your plugin (it do all the work of running tests and gathering the logs) and worker (it is responsible for backward communication with aggregator pod by https).
Sonobuoy provides two types of plugins. First type of plugin is Job. These are the plugins which only need to run once. The Sonobuoy aggregator will create a single pod for this type of plugin. The Kubernetes E2E plugin is a job-type plugin.

Note : conformance tests about which we talked about before are the subset of e2e tests, conformance tests are for checking the basic functionality but e2e tests take into account more complex scenarios like in case of third party tools, online vendors etc.
Second type of plugin is Daemonset. These are the plugins which need to run on every node, even control-plane nodes. The systemd-logs gatherer is a daemonset-type plugin.

The main command is sonobuoy run which creates and throws a yaml into your cluster for running the plugins by creating a bunch of objects. We can see what it creates by kubectl get all --namespace sonobuoy. By this command sonobuoy runs e2e tests in a way it can prove the conformance, these tests take upto an hour. It also runs systemd-logs gatherer to see what is happening on each node.
Command sonobuoy status lists all the plugins running in the cluster and also their status, results and count.
Command sonobuoy logs looks for logs for each container which we have launched in that namespace. As we know that e2e tests take upto hours to complete, so by this command we can see how our plugins are running, we can also see how many tests will be running in this particular cluster to prove conformance (which means conformance tests), we can see that in this case 337 tests will be running out of 5770 to prove conformance.

Screenshot (507).png

Command sonobuoy delete deletes all the resources made by sonobuoy in your cluster.

Screenshot (505).png

Command sonobuoy gen shows the yaml which was executed by sonobuoy in your cluster when we did sonobuoy run.

Screenshot (506).png

We can see also the configuration which describes our run. This configuration is customizable by the user.

Screenshot (508).png

We can also see here the full description of our plugin here and also observe that to specify a plugin their is not so much to specify. This whole plugin section of the yaml is what executed by the aggregator pod.

Screenshot (509).png

We can add flags : --mode quick and --plugin e2e to run only a single e2e test quickly. By this we can observe what happens when a plugin gets completed and passed.

Screenshot (512).png

Now we can see in the status complete and passed in the results. When we see status in json format we can see that it only runs 1 test and skipped 5769. We can also see result from tarball file by sonobuoy results (name of tarball we get by sonobuoy retrieve).

Each plugin is a container inside a pod with an another worker container, a mount is shared between them for the sharing of information. After a plugin is executed, it produces an output. After that it creates a result file which have the result from which aggregator interprets at the end. It also creates a done file which have path of the result file. Worker reads that file and sends the result file to the aggregator pod securely.
Sonobuoy can supports result file in formats JUnit and Raw. Aggregator pod understands the result file produced by the plugin and in return gives useful information about the run through commands like sonobuoy status, sonobuoy results etc.