lifelog and tech

Sample of error: 
Unable to connect to the server: getting credentials: decoding stdout: no kind "ExecCredential" is registered for version "client.authentication.k8s.io/v1alpha1 " in scheme "pkg/client/auth/exec/exec.go:62"

Preferred check list:

-. Need to verify if the communication with the endpoint is normal or not.

-. The version of AWS CLI should be latest.

-. Please check if there's any difference in kubectl version for client and server. this gap can cause this kind of error.

The command of checking each version:

kubectl version --short

-. Please update apiVersion value in User section of .kube/config file to client.authentication.k8s.io/v1beta1
or run "aws eks update-kubeconfig --region region-code --name my-cluster" after updating kubectl version.

Reference:

https://github.com/aws/aws-cli/issues/6920
https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html
https://kubernetes.io/releases/version-skew-policy/
https://docs.aws.amazon.com/eks/latest/userguide/install-kubectl.html
https://stackoverflow.com/questions/73744199/unable-to-connect-to-the-server-getting-credentials-decoding-stdout-no-kind

CLI를 이용한 kubectl 명령어의 apply 와 create 차이점은 apply 는 선언적(declaritive)이라는 데 반해 create 은 명령적(imperative)이라는 것이다.

Declarative: 목적지를 알려주고 step by step 으로 도달하기 위한 단계를 생략

Imperative: 무엇을 어떻게 목적지까지 갈지 명확한 가이드 존재

위 내용을 좀 더 자세히 설명하자면 다음과 같다.

보통, 리소스의 상태가 manifest 파일에 기록되면 kubectl apply로 해당 상태를 배포한다.

이에 반해, kubectl create 명령어는 바로 CLI를 통해 쿠버네티스 리소스를 생성한다.

kubectl create도 manifest 파일을 이용해서 리소스의 새 인스턴스를 생성할 수 있으나, 만약 이미 해당 인스턴스가 존재하고 있었다면 에러가 발생한다.

• Example of kubectl apply

예를 들어서 아래 yml 파일은 2개의 nginx 리플리카를 배포한다고 기술하고 있다.

newdeployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: newdeployment
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        ports:
        - containerPort: 80

kubectl apply로 해당 파일을 배포하면 metadata의 이름 newdeployment로 기술된 내용이 배포된다.

kubectl apply -f newdeployment.yml

result:

deployment/newdeployment created

kubectl get deployment 명령어로 배포된 내용을 확인한다.

NAME           READY   UP-TO-DATE   AVAILABLE   AGE
newdeployment   2/2     2            2           5m10s

해당 결과로 newdeployment라는 배포로 2개의 pod가 준비된 것을 확인할 수 있다.

• Example of kubectl create

kubectl create을 사용하여 명령적으로 새 배포를 생성할 수 있다.

kubectl create deployment directdeployment --image=nginx
deployment.apps/directdeployment created

만약 방금 생성한 존재하는 배포이름과 동일하게 생성을 시도하면 이미 존재한다며 오류가 발생한다.

kubectl create deployment newdeployment --image=nginx
Error from server (AlreadyExists): deployments.apps "newdeployment" already exists

• 차이점 정리

만약 replica 갯수를 2에서 3으로 증가시켜야 한다면, 위에 기술했던 yml 파일의 replica 갯수를 2에서 3으로 수정한 후,

kubectl apply로 배포하기만 하면 된다.

kubectl apply가 수정된 manifest파일을 참조해서 리소스를 알아서 업데이트하는 반면,

kubectl create 은 이미 존재하는 배포 이름이 아닌 오직 새 리소스를 생성할 때 명시적으로 사용할 수 있다.

• 이외 해당 명령어의 부가적인 옵션

https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands

추가 참조: https://kubernetes.io/docs/tasks/manage-kubernetes-objects/imperative-command/

Create and run a particular image

kubectl run NAME --image=image [--env="key=value"] [--port=port] [--replicas=replicas] [--dry-run=bool] [--overrides=inline-json] [--command] -- [COMMAND] [args...]

Create a namespace called 'dev' and a pod with image nginx called p on the namespace:

kubectl create namespace dev
kubectl run p --image=nginx --restart=Never -n dev

• 기본 네임스페이스 변경

현재 컨텍스트 정보 확인

$ kubectl config current-context
docker-desktop

현재 컨텍스트의 정보 확인

$ kubectl config get-contexts docker-desktop
CURRENT	NAME			CLUSTER			AUTHINFO		NAMESPACE
*		docker-desktop	docker-desktop	docker-desktop

NAMESPACE가 비어있는 경우 기본 네임스페이스는 default

기본 네임스페이스를 kube-system으로 변경

$ kubectl config set-context docker-desktop --namespace=kube-system
Context "docker-desktop" modified.

재확인

$ kubectl config get-contexts $(kubectl config current-context) --namespace=kube-system
CURRENT	NAME			CLUSTER			AUTHINFO		NAMESPACE
*		docker-desktop	docker-desktop	docker-desktop	kube-system

Window

> kubectl config view | findstr namespace
	namespace: kube-system

MacOS & Linux

$ kubectl config view | grep namespace
	namespace: kube-system

전체 네임스페이스 확인

$ kubectl get pods --all-namespaces
NAMESPACE		NAME		READY	STATUS	RESTARTS	AGE
docker			compose-909	1/1		Running	0			46m
..

다시 기본 네임스페이스를 default로 변경시

kubectl config set-context contextname --namespace=default

혹은

kubectl config set-context contextname --namespace-""

사용

또는 kubens 라는 툴을 이용하여 간단하게 네임스페이스 변경

#macOS
$ brew install kubectx

#linux
$ wget https://raw.githubusercontent.com/ahmetb/kubectx/master/kubens

$ kubens
default
docker
...

# change default namespace with kubens tool
$ kubens kube-system
Context "docker-desktop" modified.
Active namespace is "kube-system".

-. Storage 10%

Understand storage classes, persistent volumes
Understand volume mode, access modes and reclaim policies for volumes
Understand persistent volume claims primitive
Know how to configure applications with persistent storage

-. Troubleshooting 30%

Understand storage classes, persistent volumes
Understand volume mode, access modes and reclaim policies for volumes
Understand persistent volume claims primitive
Know how to configure applications with persistent storage

-. Workloads & Scheduling 15%

Understand deployments and how to perform rolling update and rollbacks
Use ConfigMaps and Secrets to configure applications
Know how to scale applications
Understand the primitives used to create robust, self-healing, application deployments
Understand how resource limits can affect Pod scheduling
Awareness of manifest management and common templating tools

-. Cluster Architecture, Installation & Configuration 25%

Manage role based access control (RBAC)
Use Kubeadm to install a basic cluster
Manage a highly-available Kubernetes cluster
Provision underlying infrastructure to deploy a Kubernetes cluster
Perform a version upgrade on a Kubernetes cluster using Kubeadm
Implement etcd backup and restore

-. Services & Networking 20%

Understand host networking configuration on the cluster nodes
Understand connectivity between Pods
Understand ClusterIP, NodePort, LoadBalancer service types and endpoints
Know how to use Ingress controllers and Ingress resources
Know how to configure and use CoreDNS
Choose an appropriate container network interface plugin

kubectl get pods

생성된 pod 정보 확인

kubectl describe pods

자세한 pod 정보 확인. IP address, the ports used and a list of events related to the lifecycle of the Pod.

$ sleep 1; launch.sh
Starting Kubernetes. This is expected to take less than a minute.....
Kubernetes Started

$ kubectl get pods
NAME                                  READY   STATUS              RESTARTS   AGE
kubernetes-bootcamp-fb5c67579-xxdfc   0/1     ContainerCreating   0          4s
$ kubectl get pods
NAME                                  READY   STATUS    RESTARTS   AGE
kubernetes-bootcamp-fb5c67579-xxdfc   1/1     Running   0          52s
$ kubectl describe pods
Name:         kubernetes-bootcamp-fb5c67579-xxdfc
Namespace:    default
Priority:     0
Node:         minikube/10.0.0.17
Start Time:   Tue, 12 Apr 2022 07:15:00 +0000
Labels:       app=kubernetes-bootcamp
              pod-template-hash=fb5c67579
Annotations:  <none>
Status:       Running
IP:           172.18.0.3
IPs:
  IP:           172.18.0.3
Controlled By:  ReplicaSet/kubernetes-bootcamp-fb5c67579
Containers:
  kubernetes-bootcamp:
    Container ID:   docker://b525713ba1e40d0358493591c84aaa2444cc2bf13beb481a88aa27dc438e8d58
    Image:          gcr.io/google-samples/kubernetes-bootcamp:v1
    Image ID:       docker-pullable://jocatalin/kubernetes-bootcamp@sha256:0d6b8ee63bb57c5f5b6156f446b3bc3b3c143d233037f3a2f00e279c8fcc64af
    Port:           8080/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 12 Apr 2022 07:15:02 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-6vrph (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-6vrph:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-6vrph
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  74s   default-scheduler  Successfully assigned default/kubernetes-bootcamp-fb5c67579-xxdfc to minikube
  Normal  Pulled     72s   kubelet            Container image "gcr.io/google-samples/kubernetes-bootcamp:v1" already present on machine
  Normal  Created    72s   kubelet            Created container kubernetes-bootcamp
  Normal  Started    72s   kubelet            Started container kubernetes-bootcamp

-. proxy를 통해 pod에 접근하여 pod 이름을 구하여 POD_NAME 환경변수로 export하기

먼저, 새로운 터미널에서 proxy start. 아웃풋을 확인하려면 다른 터미널에서 확인

$ echo -e "\n\n\n\e[92mStarting Proxy. After starting it will not output a response. e first Terminal Tab\n"; kubectl proxy

Starting Proxy. After starting it will not output a response. Please click the first Terminal Tab

Starting to serve on 127.0.0.1:8001

기존 터미널에서 kubectl 로 파드 이름을 구하여 환경변수로 지정하여 호출하기.

아웃풋 확인은 curl을 통해 호출한다.

$ export POD_NAME=$(kubectl get pods -o go-template --template '{{range .items}}{{.metadata.name}}{{"\n"}}{{end}}')
$ echo Name of the Pod: $POD_NAME
Name of the Pod: kubernetes-bootcamp-fb5c67579-xxdfc
$ curl http://localhost:8001/api/v1/namespaces/default/pods/$POD_NAME/proxy/
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-xxdfc | v=1

해당 파드 내의 컨테이너에서 어플리케이션이 STDOUT으로 보내는 기록이 로그로 남고, 확인은 다음과 같이 한다.

$ kubectl logs $POD_NAME
Kubernetes Bootcamp App Started At: 2022-04-12T07:15:03.138Z | Running On:  kubernetes-bootcamp-fb5c67579-xxdfc 

Running On: kubernetes-bootcamp-fb5c67579-xxdfc | Total Requests: 1 | App Uptime: 1320.845 seconds | Log Time: 2022-04-12T07:37:03.983Z
Running On: kubernetes-bootcamp-fb5c67579-xxdfc | Total Requests: 2 | App Uptime: 1335.31 seconds | Log Time: 2022-04-12T07:37:18.448Z
Running On: kubernetes-bootcamp-fb5c67579-xxdfc | Total Requests: 3 | App Uptime: 1445.257 seconds | Log Time: 2022-04-12T07:39:08.395Z

pod가 up and running 상태이면 exec 명령어를 사용하여 

kubectl exec 의 옵션에 대한 정보:

$ kubectl exec --help
Execute a command in a container.

Examples:
  # Get output from running 'date' command from pod mypod, using the first
container by default
  kubectl exec mypod -- date
  
  # Get output from running 'date' command in ruby-container from pod mypod
  kubectl exec mypod -c ruby-container -- date
  
  # Switch to raw terminal mode, sends stdin to 'bash' in ruby-container from
pod mypod
  # and sends stdout/stderr from 'bash' back to the client
  kubectl exec mypod -c ruby-container -i -t -- bash -il
  
  # List contents of /usr from the first container of pod mypod and sort by
modification time.
  # If the command you want to execute in the pod has any flags in common (e.g.
-i),
  # you must use two dashes (--) to separate your command's flags/arguments.
  # Also note, do not surround your command and its flags/arguments with quotes
  # unless that is how you would execute it normally (i.e., do ls -t /usr, not
"ls -t /usr").
  kubectl exec mypod -i -t -- ls -t /usr
  
  # Get output from running 'date' command from the first pod of the deployment
mydeployment, using the first container by default
  kubectl exec deploy/mydeployment -- date
  
  # Get output from running 'date' command from the first pod of the service
myservice, using the first container by default
  kubectl exec svc/myservice -- date

Options:
  -c, --container='': Container name. If omitted, the first container in the pod
will be chosen
  -f, --filename=[]: to use to exec into the resource
      --pod-running-timeout=1m0s: The length of time (like 5s, 2m, or 3h, higher
than zero) to wait until at least one pod is running
  -i, --stdin=false: Pass stdin to the container
  -t, --tty=false: Stdin is a TTY

Usage:
  kubectl exec (POD | TYPE/NAME) [-c CONTAINER] [flags] -- COMMAND [args...]
[options]

Use "kubectl options" for a list of global command-line options (applies to all
commands).

pod의 환경변수 확인: kubectl exec $POD_NAME -- env

pod의 컨테이너에서 bash 세션 생성 : kubectl exec -ti $POD_NAME -- bash

bash 세션으로 들어온 후, 컨테이너 안의 실제 어플리케이션 코드에 접근 가능.

어플리케이션이 정상 동작하는 상태인지 curl localhost로 확인.

root@kubernetes-bootcamp-fb5c67579-xxdfc:/# curl localhost:8080
Hello Kubernetes bootcamp! | Running on: kubernetes-bootcamp-fb5c67579-xxdfc | v=1

• K8s pod : 컨테이너들의 그룹. 사설 네트워크에서 독립적으로 동작하여 proxy를 통해 접근하여 디버그 등 확인.
• K8s node : 파드를 포함한 워커머신, 컨트롤 플레인에 의해 관리됨

https://kubernetes.io/ko/docs/tutorials/kubernetes-basics/explore/explore-intro/

• What is a Kubernetes taint : 

Node affinity is a property of Pods that attracts them to a set of nodes (either as a preference or a hard requirement). Taints are the opposite -- they allow a node to repel a set of pods. Tolerations are applied to pods, and allow (but do not require) the pods to schedule onto nodes with matching taints.

- From Taints and Tolerations https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/

• What is Kubespray :
  • open source project to deploy Kubernetes cluster with high security and availability.
  • automatically configure cluster network with simple setting. support network plug-in such as:

flannel, calico, canal, cilium, contiv, weavenet, kube-router, multus..

calico - is mostly used for public cloud

• Control Plane: 

워커 노드와 클러스터 내 파드를 관리한다.

전체 개념: https://kubernetes.io/ko/docs/concepts/overview/components/

• 파드(Pod) : 어플리케이션의 구성요소
• 워커노드(Worker Nodes) : 파드를 호스트함.
• kubelet : 클러스터 각 노드에서 실행되는 에이전트. 파드에서 컨테이너가 동작하도록 관리.  pod spec에 따라 컨테이너가 healthy하게 동작하도록 관리. captain on the ship who creates the pod on the ships (metaphor)
• kube-scheduler : responsible for deciding which pod goes on which node("where"). it doesn't actually place the pod on the nodes but kublet will take in charge. right container to the right ship. (metaphor)
  • phases: 1. Filter Nodes (sufficient CPU & menory resources) 2. Rank Nodes (priority. calculate resources that would be free after placing the pod on them)
• kube-proxy : 클러스터 각 노드의 네트워크 프록시. 서비스 구현부. 클러스터 외부에서 파드로 통신하는 네트워크 규칙 관리. Looks for new services and create appropriate rules (iptable rule) on each node to forward traffic to those services to the backend pods, whenever new service is created.
• difference in Kubespray and kubeadm : 쿠베스프레이는 kubeadm과는 다르게, 별도의 로드밸런서를 사용하지 않고 각각의 nginx가 리버스 프록시로 실행됨. nginx-proxy가 전체 마스터 노드 바라봄. 쿠베adm은 워커노드들이 로드밸런서를 바라보고, 로드밸런서가 각 스택 etcd 클러스터를 가리키는 구조.
• etcd : 데이터 저장소.
• What is Kubeadm : cluster creation/management tool provided by Kubernetes officially.