Limited Time Offer!

For Less Than the Cost of a Starbucks Coffee, Access All DevOpsSchool Videos on YouTube Unlimitedly.
Master DevOps, SRE, DevSecOps Skills!

Enroll Now
Uncategorised

Knative: Troubleshooting & Debugging Guide

December 21, 2022 Leave a Comment

Check if all containers are running, ready and healthy

Knative Serving Components

$ kubectl get pods -n knative-serving

NAME                                      READY   STATUS    RESTARTS   AGE
activator-6b9dc4c9db-cl56b                1/1     Running   0          2m
autoscaler-77f9b75856-f88qw               1/1     Running   0          2m
controller-7dcb56fdb6-dbzrp               1/1     Running   0          2m
domain-mapping-6bb8f95654-c575d           1/1     Running   0          2m
domainmapping-webhook-c77dcfcfb-hg2wv     1/1     Running   0          2m
webhook-78dc6ddddb-6868n                  1/1     Running   0          2m

Knative Serving Networking Layer

$ kubectl get pods -n knative-serving

NAME                                    READY   STATUS    RESTARTS   AGE
net-istio-controller-ccc455b58-f98ld    1/1     Running   0          19s
net-istio-webhook-7558dbfc64-5jmt6      1/1     Running   0          19s
$ kubectl get pods -n istio-system

NAME                                   READY   STATUS    RESTARTS   AGE
istio-ingressgateway-c7b9f6477-bgr6q   1/1     Running   0          44s
istiod-79d65bf5f4-5zvtj                1/1     Running   0          29s
$ kubectl get pods -n knative-serving

NAME                                      READY   STATUS    RESTARTS   AGE
net-kourier-controller-5fcbb6d996-fprpd   1/1     Running   0          103s

$ kubectl get pods -n kourier-system
NAME                                      READY   STATUS    RESTARTS   AGE
3scale-kourier-gateway-86b9f6dc44-xpn6h   1/1     Running   0          2m22s

$ kubectl get pods -n contour-external

NAME                            READY   STATUS      RESTARTS   AGE
contour-7b995cdb68-jg5s8        1/1     Running     0          41s
contour-certgen-v1.24.2-zmr9r   0/1     Completed   0          41s
envoy-xkzck                     2/2     Running     0          41s

$ kubectl get pods -n contour-external

NAME                            READY   STATUS      RESTARTS   AGE
contour-7b995cdb68-jg5s8        1/1     Running     0          41s
contour-certgen-v1.24.2-zmr9r   0/1     Completed   0          41s
envoy-xkzck                     2/2     Running     0          41s

$ kubectl get pods -n contour-internal

NAME                            READY   STATUS      RESTARTS   AGE
contour-57fcf576fd-wb57c        1/1     Running     0          55s
contour-certgen-v1.24.2-gqgrx   0/1     Completed   0          55s
envoy-rht69                     2/2     Running     0          55s

Knative Eventing

$ kubectl get pods -n knative-eventing

NAME                                  READY   STATUS    RESTARTS   AGE
eventing-controller-bb8b689c4-lk6pq   1/1     Running   0          41s
eventing-webhook-577bb88ccd-hcz5p     1/1     Running   0          41s

Check if there are any errors logged in the Knative components

$ kubectl logs -n knative-serving <pod-name>
$ kubectl logs -n knative-eventing <pod-name>
$ kubectl logs -n <ingress-namespaces> <pod-namespaces> # see above for the relevant namespaces

For example

$ kubectl logs -n knative-serving activator-6b9dc4c9db-cl56b
2023/05/01 11:52:51 Registering 3 clients
2023/05/01 11:52:51 Registering 3 informer factories
2023/05/01 11:52:51 Registering 4 informers

Check the status of the Knative Resources

$ kubectl describe -n <namespace> kservice
$ kubectl describe -n <namespace> config
$ kubectl describe -n <namespace> revision
$ kubectl describe -n <namespace> sks # Serverless Service
$ kubectl describe -n <namespace> kingress # Knative Ingress
$ kubectl describe -n <namespace> rt # Knative Route
$ kubectl describe -n <namespace> dm # Domain-Mapping

Check the status at the end. For example

Knative Serving

$ kubectl describe -n default kservice

... omitted ...
Status:
  Address:
    URL:  http://hello.default.svc.cluster.local
  Conditions:
    Last Transition Time:        2023-05-01T12:08:18Z
    Status:                      True
    Type:                        ConfigurationsReady
    Last Transition Time:        2023-05-01T12:08:18Z
    Status:                      True
    Type:                        Ready
    Last Transition Time:        2023-05-01T12:08:18Z
    Status:                      True
    Type:                        RoutesReady
  Latest Created Revision Name:  hello-00001
  Latest Ready Revision Name:    hello-00001
  Observed Generation:           1
  Traffic:
    Latest Revision:  true
    Percent:          100
    Revision Name:    hello-00001
  URL:                http://hello.default.10.89.0.200.sslip.io
Events:
  Type    Reason   Age   From                Message
  ----    ------   ----  ----                -------
  Normal  Created  45s   service-controller  Created Configuration "hello"
  Normal  Created  45s   service-controller  Created Route "hello"

Knative Eventing

$ kubectl describe -n <namespace> brokers
$ kubectl describe -n <namespace> eventtypes
$ kubectl describe -n <namespace> triggers
$ kubectl describe -n <namespace> channels
$ kubectl describe -n <namespace> subscriptions
$ kubectl describe -n <namespace> apiserversources
$ kubectl describe -n <namespace> containersources
$ kubectl describe -n <namespace> pingsources
$ kubectl describe -n <namespace> sinkbindings

Check the status at the end. For example

$ kubectl describe -n default brokers

... omitted ...
Status:
  Annotations:
    bootstrap.servers:                 my-cluster-kafka-bootstrap.kafka:9092
    default.topic.partitions:          10
    default.topic.replication.factor:  3

KServe Debugging Guide

  • https://kserve.github.io/website/developer/debug/

Debugging application issues

  • https://knative.dev/docs/serving/troubleshooting/debugging-application-issues/

Check Istio-Ingressgateway Pod Status:

Knative Serving Troubleshooting: Autoscaler Component

Reference

  • https://knative.dev/docs/install/troubleshooting/
  • https://knative.dev/docs/serving/troubleshooting/debugging-application-issues/
  • https://knative.dev/docs/eventing/troubleshooting/
  • https://knative.dev/docs/serving/troubleshooting/debugging-application-issues/
  • https://github.com/dewitt/knative-docs/blob/master/serving/debugging-application-issues.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/debugging-performance-issues.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/accessing-logs.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/accessing-metrics.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/accessing-traces.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/debugging-application-issues.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/debugging-performance-issues.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/gke-assigning-static-ip-address.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/installing-logging-metrics-traces.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/outbound-network-access.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/setting-up-a-logging-plugin.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/using-a-custom-domain.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/using-an-ssl-cert.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/using-cert-manager-on-gcp.md
  • https://github.com/dewitt/knative-docs/blob/master/serving/using-external-dns.md
  • https://github.com/knative/serving/issues
  • https://github.com/knative/eventing/issues
Rajesh Kumar
Follow me
Latest posts by Rajesh Kumar (see all)
Subscribe
Notify of
guest
0 Comments
Newest
Oldest Most Voted
Inline Feedbacks
View all comments
0
Would love your thoughts, please comment.x
()
x