Kubeflow와 KFServing을 GCP에 설치한다.
공식 홈페이지는 다음과 같다.
https://www.kubeflow.org/docs/started/installing-kubeflow/
Installing Kubeflow
Deployment options for Kubeflow
www.kubeflow.org
설치 버전 정리
- Google Clould Compute Engine
- Ubuntu 18.04.5 LTS
- docker-CE 20.10.14
- kubernetes v1.20.5
- cilium (coredns 용 network addon)
- kubeflow 1.3
1. 서버 Spec
| 구분 | CPU | RAM | Storage | 내부 IP |
| Master | 8 vCore | 32 GB | 200 GB | 10.138.10.10 |
| Node-1 | 8 vCore | 32 GB | 200 GB | 10.138.10.13 |
| Node-2 | 8 vCore | 32 GB | 200 GB | 10.138.10.14 |
OS : Ubuntu Ubuntu 18.04.5 LTS (Bionic Beaver)
(20.04 버전은 kubeflow가 아직 지원을 안하는 듯 함... 구성시 문제 발생함)
2. Docker 설치
# root 권한으로 실행
$ sudo su -
# 패키지 관리 도구 업데이트
$ apt update
$ apt-get update
$ apt upgrade -y
# 기존 docker 설치된 리소스 확인 후 발견되면 삭제
$ apt-get remove docker docker-engine docker.io
# docker를 설치하기 위한 각종 라이브러리 설치
$ apt-get install apt-transport-https ca-certificates curl software-properties-common -y
# curl 명령어를 통해 gpg key 내려받기
$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# 키를 잘 내려받았는지 확인
$ apt-key fingerprint 0EBFCD88
# 패키지 관리 도구에 도커 다운로드 링크 추가
$ add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
# 패키지 관리 도구 업데이트
$ apt-get update
# Docker Engine은 최신 버전을 사용함.
$ apt-get install docker-ce docker-ce-cli containerd.io
$ apt-mark hold docker-ce docker-ce-cliDocker 설치 확인
# docker version
Client: Docker Engine - Community
Version: 20.10.14
API version: 1.41
Go version: go1.16.15
Git commit: a224086
Built: Thu Mar 24 01:47:57 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.14
API version: 1.41 (minimum version 1.12)
Go version: go1.16.15
Git commit: 87a90dc
Built: Thu Mar 24 01:45:46 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.11
GitCommit: 3df54a852345ae127d1fa3092b95168e4a88e2f8
runc:
Version: 1.0.3
GitCommit: v1.0.3-0-gf46b6ba
docker-init:
Version: 0.19.0
GitCommit: de40ad0
3. Kubernetes 설치
Kubeflow version은 현재 1.4 이며 Kubernetes와의 호환성은 다음과 같음
Kubeflow 1.3 이후, 각 Component 별로 Kubernetes 및 istio 환경에 대한 호환성을 확인해야 함에 주의
# root 권한으로 실행 (이미 했으면 생략)
$ sudo su -
# update package repository
$ apt update && apt upgrade -y
# Network 설정
$ swapoff -a
# google k8s 패키지 소스 등록
$ curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
$ cat <<EOF | sudo tee /etc/apt/sources.list.d/kubernetes.list
#deb https://apt.kubernetes.io/ kubernetes-xenial main
deb https://mirrors.aliyun.com/kubernetes/apt kubernetes-xenial main
EOF
$ apt-get update
$ apt install linux-image-extra-virtual ca-certificates curl software-properties-common -y
# 버전 설치
#$ apt-get install -y kubelet=1.16.15-00 kubeadm=1.16.15-00 kubectl=1.16.15-00
$ apt-get install -y kubelet=1.20.5-00 kubeadm=1.20.5-00 kubectl=1.20.5-00 --allow-downgrades --allow-change-held-packages
#$ apt-mark hold kubelet=1.16.15-00 kubeadm=1.16.15-00 kubectl=1.16.15-00
$ sudo su
cat <<EOF > /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
EOF
$ sysctl --system
$ systemctl daemon-reload
$ systemctl restart kubelet
Kubernetes 설치버전 확인
# kubectl version
Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.5", GitCommit:"6b1d87acf3c8253c123756b9e61dac642678305f", GitTreeState:"clean", BuildDate:"2021-03-18T01:10:43Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
여기까지는 설치하려는 모든 노드(Master/Woker Node)에서 실행함.
3-1. Master 노드 설치
sudo kubeadm init --pod-network-cidr=192.168.0.0/16
* 발생한 에러
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
에러시
echo '1' > /proc/sys/net/ipv4/ip_forward한 후 다시실행
(실행이 종료되면 화면에 출력된 값을 별도 저장한다.)
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubenetes system 관련 추가 실행 (addon 설치)
$ kubectl taint nodes --all node-role.kubernetes.io/master-
$ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1.6/install/kubernetes/quick-install.yaml
$ kubectl get pods -n kube-system --selector=k8s-app=ciliumhttps://kubernetes.io/docs/concepts/cluster-administration/addons/
Installing Addons
Note: This section links to third party projects that provide functionality required by Kubernetes. The Kubernetes project authors aren't responsible for these projects, which are listed alphabetically. To add a project to this list, read the content guide
kubernetes.io
3-2. Worker 노드 설치
master 구성시 생성된, token을 worker node 에서 실행함.
kubeadm join 10.138.10.10:6443 --token s7au79.zhzn0s5fx1f1x6ax \
--discovery-token-ca-cert-hash sha256:ec079a7e630a558f11bb8c4a7f956f59ebec1ba35c16a84133b4c9axxxxxxxx* 발생한 에러
[ERROR FileContent--proc-sys-net-ipv4-ip_forward]: /proc/sys/net/ipv4/ip_forward contents are not set to 1
에러시
echo '1' > /proc/sys/net/ipv4/ip_forward한 후 다시실행
다음과 같이 표시되면 worker설치 성공.
This node has joined the cluster:
* Certificate signing request was sent to apiserver and a response was received.
* The Kubelet was informed of the new secure connection details.
Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
3-3. Master 에서 Cluster 구성 확인
Master node에서 구성된 Kubernetes Cluster의 정보를 확인한다.
# Master node에서 실행
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master Ready control-plane,master 8m50s v1.20.5
node1 Ready <none> 76s v1.20.5
node2 Ready <none> 72s v1.20.5Cluster node의 상세 정보 확인시
$ kubectl get nodes -o wide
3-4. Kubenetes Dashboard 설치
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.4.0/aio/deploy/recommended.yamlDashboard 외부접근을 위한 NodePort 수정
$ kubectl patch svc -n kubernetes-dashboard kubernetes-dashboard --type='json' -p '[
{"op":"replace","path":"/spec/type", "value":"NodePort"},
{"op":"replace","path":"/spec/ports/0/nodePort","value":30003}
]'NodePort 확인
$ kubectl get svc -n kubernetes-dashboard kubernetes-dashboard
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes-dashboard NodePort 10.103.211.111 <none> 443:30003/TCP 29h
* Web UI를 통한 대시보드 접속
https://{Node_IP}:30003
* 쿠버네티스 로그인 토큰 생성
serviceaccount 생성
cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
name: admin-user
namespace: kube-system
EOF* ClusterRoleBinding 생성
cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: admin-user
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cluster-admin
subjects:
- kind: ServiceAccount
name: admin-user
namespace: kube-system
EOF* 사용자 계정의 토큰 호출
$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')결과
$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret |
grep admin-user | awk '{print $1}')
Name: admin-user-token-kcj5k
Namespace: kube-system
Labels: <none>
Annotations: kubernetes.io/service-account.name: admin-user
kubernetes.io/service-account.uid: 24f3d8b7-fd90-4356-b955-b19b4fb85f27
Type: kubernetes.io/service-account-token
Data
====
token: eyJhbGciOiJSUzI1NiIsImtpZCI6IkpkMGFiMmpld0VPT1daMFpNTUVJWW50Y1RxczRKV0FWMzZZbHlXSHpYbVUifQ.eyJpc3MiOiJr
dWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmV
ybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLWtjajVrIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYW
Njb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3Vud
C51aWQiOiIyNGYzZDhiNy1mZDkwLTQzNTYtYjk1NS1iMTliNGZiODVmMjciLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06
YWRtaW4tdXNlciJ9.MFlfSlK1i8iUYpk4g7mWvn5pdVDYkeFEh0Z9wtcSzmnUoHwHh3JUSejcCIwViqydkS1-S2bB4pdIW_G-v0BUCHBKO00GNMVHGq
SzXox3Q_GFBuKryqBBoKLLtIfMb2k9HunnL4HJFcmG4Nc_jH-R7ehGVcBnp8a1x4xF11Lyo_fI5BzkV4kKIT7XlOXCvzZKMwhz1BL5-D190FbIft9Bs
ALDgjYUBE6qXxMJsMFgKYgp3ztzhuI59Esto9-mDRhWBSZ_E60TaDm9zmi49QV1F1IJObW6-2sTW_xZZHfchZNtX5Esbual1J9KcLFzjPHJIEM-az8X
tVG2Fk_aAl0g0Q
ca.crt: 1066 bytes
namespace: 11 bytes웹(UI) 접속시 Token 사용
kubernetes pods 확인
$ kubectl get pods --all-namespaces
k9s (Console Dashboard) 설치
wget https://github.com/derailed/k9s/releases/download/v0.25.18/k9s_Linux_x86_64.tar.gz
tar xzf k9s_Linux_x86_64.tar.gz
mv -f k9s /usr/bin실행은 k9s , 종료는 ctrl + c .
4. Kubeflow 설치
4-1. PV(Persistent Volume) 설치
kubeflow를 설치전에 storage를 먼저 설치해야 한다.
pv 생성을 위해서 NFS(Network File System)을 설치 함.
[Master Node]
$ sudo apt install -y nfs-common nfs-kernel-server portmap
$ sudo mkdir /nfs # 스토리지 폴더로 사용
$ sudo chmod 777 /nfs
$ sudo cat > /etc/exports << EOF
/nfs 10.138.10.10(rw,sync,insecure,no_root_squash,no_subtree_check) # master node 내부 ip
/nfs 10.138.10.13(rw,sync,insecure,no_root_squash,no_subtree_check) # worker node 1 내부 ip
/nfs 10.138.10.14(rw,sync,insecure,no_root_squash,no_subtree_check) # worker node 2 내부 ip
EOF
$ /etc/init.d/nfs-kernel-server restart # 서버 재시작
$ sudo exportfs -a[Worker Node]
# Worker node nfs 클라이언트 설치 ( 2대 모두 설치 )
$ sudo apt install nfs-common
$ sudo mkdir /nfs
$ sudo chmod 777 /nfs
$ sudo mount 10.138.10.10:/nfs /nfsMount 할 vm과의 방화벽이 오픈되어 있어야 한다.
방화벽이 오픈되면 file이 공유된다.
4-2. nfs-client Storage Class 설치
[Master Node]
$ sudo curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 > get_helm.sh
$ sudo chmod 700 get_helm.sh
$ sudo ./get_helm.sh
$ sudo kubectl -n kube-system create sa tiller
$ sudo kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller
$ sudo helm init --service-account tiller
$ sudo helm repo update
$ kubectl apply -f https://raw.githubusercontent.com/rancher/local-path-provisioner/master/deploy/local-path-storage.yaml
$ kubectl -n local-path-storage get pod
$ kubectl get storageclass
# katib-mysql error 처리용
$ mkdir -p /var/lib/mysql-files
$ chmod 777 /var/lib/mysql-files/
# storage provisiner 추가
$ helm repo add raphael https://raphaelmonrouzeau.github.io/charts/repository/
$ helm repo update
$ helm install nfs-provisioner \
--set nfs.server=[nfs 서버 ip] \
--set nfs.path=/nfsroot \
--set storageClass.defaultClass=true \
--set storageClass.name=nfs-provisioner \
raphael/nfs-server-provisioner
$ kubectl get sc
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMOD
E ALLOWVOLUMEEXPANSION AGE
local-path rancher.io/local-path Delete WaitForFirstCons
umer false 164m
nfs-provisioner (default) cluster.local/nfs-provisioner-nfs-server-provisioner Delete Immediate
true 15m볼륨 PVC 확인
$ kubectl get pvc
$ kubectl get pvpv가 제대로 설정되면 다음과 깉이 표시됨 (kubeflow 설치 시 ..보여짐)
$ kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-3dca96cb-35cf-4f51-bca1-8687582dfd28 20Gi RWO Delete Bound kubeflow/mysql-pv-claim nfs-client 27m
pvc-55404915-608e-4e94-a33f-c8603c122dea 10Gi RWO Delete Bound istio-system/authservice-pvc nfs-client 27m
pvc-ad316a8e-ab5b-4beb-a2a5-fdf362d2beb5 10Gi RWO Delete Bound kubeflow/katib-mysql nfs-client 27m
pvc-f080541d-1a4d-43cb-847d-b1ec66301667 20Gi RWO Delete Bound kubeflow/minio-pvc nfs-client 27m
4-3. Private Docker Registry 설치
# private Registry 배포
$ wget https://raw.githubusercontent.com/mojokb/handson-kubeflow/master/registry/kubeflow-registry-deploy.yaml
# private Registry Service
$ wget https://raw.githubusercontent.com/mojokb/handson-kubeflow/master/registry/kubeflow-registry-svc.yaml
$ kubectl apply -f kubeflow-registry-deploy.yaml
$ kubectl apply -f kubeflow-registry-svc.yaml
# 배포 확인
$ kubectl get pod --all-namespaces -o wide# /etc/hosts에 private registry 등록
K8s에서 pod 생성 시 private registry를 lookup할 수 있도록 /etc/hosts에 등록 master 주소 등록
[master/ worker 모두 실행]
$ cat << EO_HOSTS >> /etc/hosts
10.138.10.10 kubeflow-registry.default.svc.cluster.local
EO_HOSTS
cat /etc/hosts
4-4. kustomize 설치
Kubeflow 1.3 부터는 kfctl 을 사용하지않고 kustomize 를 사용한다.
Kubeflow 1.3 은 최신 Kustomize 4.x 와 호환이 안된다. 3.2 버전을 다운로드 받아야한다.
wget https://github.com/kubernetes-sigs/kustomize/releases/download/v3.2.0/kustomize_3.2.0_linux_amd64
mv kustomize_3.2.0_linux_amd64 kustomize
chmod 777 kustomize
sudo mv kustomize /usr/local/bin
4-5. Kubeflow 설치
kubeflow/manifest 를 clone 받은 후 kustomize 를 사용해 설치를 진행하자
git clone https://github.com/kubeflow/manifests.git
cd manifests
while ! kustomize build example | kubectl apply -f -; do echo "Retrying to apply resources"; sleep 10; done
* Pipeline 추가 설치
# pipeline 추가 설치
kustomize build apps/pipeline/upstream/env/platform-agnostic-multi-user-pns | kubectl apply -f -
4-5-1. Kubeflow용 tls 인증서 구성
kubeflow 는 기본적으로 https 환경에서 정상 동작합니다. 이를 위해 cert-manager 를 통해 tls 인증서를 생성하고, kubeflow gateway 에 적용
* tls 인증서 생성
DNS_NAME=<외부 접속IP>
kubectl apply -f - << EOF
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: kubeflow-tls
namespace: istio-system
spec:
commonName: ${DNS_NAME}
dnsNames:
- ${DNS_NAME}
issuerRef:
kind: ClusterIssuer
name: kubeflow-self-signing-issuer
secretName: kubeflow-tls
EOF
* kubeflow gateway 에 https host 추가
kubectl patch gateway -n kubeflow kubeflow-gateway --type='json' -p '[
{"op":"add", "path":"/spec/servers/1", "value":{"hosts":["*"], "port":{"name":"https", "number":443, "protocol":"HTTPS"}, "tls":{"credentialName":"kubeflow-tls","mode":"SIMPLE"}}}
]'
* kubeflow gateway service 에 고정 nodeport 사용 설정
KUBEFLOW_DASHBOARD_PORT=32001
kubectl apply -f - << EOF
apiVersion: v1
kind: Service
metadata:
name: istio-gateway-extra-service
namespace: istio-system
spec:
externalTrafficPolicy: Cluster
ports:
- name: https
nodePort: ${KUBEFLOW_DASHBOARD_PORT}
port: 443
protocol: TCP
targetPort: 8443
selector:
app: istio-ingressgateway
istio: ingressgateway
sessionAffinity: None
type: NodePort
EOF
* Minio 접속용 virtual service 추가
kubectl apply -f - << EOF
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: minio-web
namespace: kubeflow
spec:
gateways:
- kubeflow-gateway
hosts:
- '*'
http:
- match:
- uri:
prefix: /minio
rewrite:
uri: /minio
route:
- destination:
host: minio-service.kubeflow.svc.cluster.local
port:
number: 9000
EOF
설치가 완료되면 port forwarding 를 사용하여 centraldashboard 에 접속해보자
https://{node_ip}:32001/
계정은 user@example.com / 12341234 로 로그인
* Minio 서비스 포트 변경
$ kubectl patch svc -n kubeflow minio-service --type='json' -p '[
{"op":"replace","path":"/spec/type", "value":"NodePort"},
{"op":"replace","path":"/spec/ports/0/nodePort","value":32001}
]'* Mino UI 접속
http://{node_ip}:32001/minio
id : minio / pw : minio123
* minio key
cat << EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
name: minio-secret # Secret 명
namespace: myspace # Secret이 설치될 Namespace
annotations:
serving.kubeflow.org/s3-endpoint: minio-service.kubeflow:9000 # 설치된 Minio 서비스 endpoint
serving.kubeflow.org/s3-usehttps: "0" # http를 사용하지 않음 (설치된 Minio 설정)
serving.kubeflow.org/s3-verifyssl: "0" # ssy verify 하지 않음 (설치된 Minio 설정)
serving.kubeflow.org/s3-region: us-east-0
type: Opaque
stringData:
AWS_ACCESS_KEY_ID: minio
AWS_SECRET_ACCESS_KEY: minio123
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kfserving-sa # ServiceAccount 명
namespace: myspace # ServiceAccount가 설치될 Namespace
secrets:
- name: minio-secret # 이 ServiceAccount가 사용할 Secret (위에서 생성한 Secret)
EOF* InferenceService 배포
kubectl을 통한 배포
cat << EOF | kubectl apply -f -
apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
name: covid-19 # KFServing InferenceService 명
namespace: kubeflow-user-example-com # KFServing InferenceService가 배포될 Namespace
spec:
predictor:
serviceAccountName: kfserving-sa # s3:// 레파지토리(Minio)를 사용하기 위해 이 KFServing InferenceService가 사용할 serviceAccount (Step1.에서 생성함)
tensorflow:
storageUri: 's3://saved-model/covid-19/' # 모델을 저장한 경로 s3://<bucket명>/<저장경로>
EOF
배포확인
$ kubectl get inferenceservices -n kubeflow-user-example-com covid-19
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
covid-19 http://covid-19.kubeflow-user-example-com.example.com True 100 covid-19-predictor-default-00007 4h8m
kubeflow UI를 통해서도 Model 배포 및 확인이 가능함
* InferenceService 테스트
$ kubectl get svc -n istio-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
authservice ClusterIP 10.98.91.118 <none> 8080/TCP 2d5h
cluster-local-gateway ClusterIP 10.104.55.71 <none> 15020/TCP,80/TCP 2d5h
istio-gateway-extra-service NodePort 10.98.218.236 <none> 443:32001/TCP 30h
istio-ingressgateway NodePort 10.110.142.177 <none> 15021:32310/TCP,80:32304/TCP,443:32128/TCP,31400:32680/TCP,15443:30446/TCP 2d5h
istiod ClusterIP 10.106.171.130 <none> 15010/TCP,15012/TCP,443/TCP,15014/TCP 2d5h
knative-local-gateway ClusterIP 10.103.191.26 <none> 80/TCP 2d5h
- istio-ingressgateway
$ kubectl get svc -n istio-system istio-ingressgateway
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
istio-ingressgateway NodePort 10.110.142.177 <none> 15021:32310/TCP,80:32304/TCP,443:32128/TCP,31400:32680/TCP,15443:30446/TCP 4d4h
- inferenceservices 확인
$ kubectl get inferenceservices -n kubeflow-user-example-com
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
covid-19 http://covid-19.kubeflow-user-example-com.example.com True 100 covid-19-predictor-default-00015 2d3h
local 접근 통해서 Inferenceservice의 접속을 확인해본다.
$ curl -H "Host: covid-19.kubeflow-user-example-com.example.com" http://10.110.142.177<a href="/dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY0OTM5Mjk5NnxFd3dBRUdocE0wVTJiMnhKWkRsSlRESkhhSGc9fOxZqJd0nevI75Ct_qiRptrAg6Dm0uxGixA0HrxaaSgg">Found</a>.Host: covid-19.kubeflow-user-example-com.example.com
SERVING_URL: http://104.198.13.157:32304/v1/models/covid-19:predict
NAME URL READY PREV LATEST PREVROLLEDOUTREVISION LATESTREADYREVISION AGE
covid-19 http://covid-19.kubeflow-user-example-com.example.com True 100 covid-19-predictor-default-00015 2d3h
* Trying 104.198.13.157...
* TCP_NODELAY set
* Connected to 104.198.13.157 (104.198.13.157) port 32304 (#0)
> POST /v1/models/covid-19:predict HTTP/1.1
> Host: covid-19.kubeflow-user-example-com.example.com
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Length: 4030478
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 302 Found
< location: /dex/auth?client_id=kubeflow-oidc-authservice&redirect_uri=%2Flogin%2Foidc&response_type=code&scope=profile+email+groups+openid&state=MTY0OTM5MzA0MXxFd3dBRURkelUwZEhTR2R0UzNkVlRXOWhOVk09fE2WhP33fjloEJVD0M6-4fJKY9DaEMi9kC5gT9E3AjRN
< date: Fri, 08 Apr 2022 04:44:01 GMT
< x-envoy-upstream-service-time: 6
< server: istio-envoy
< connection: close
< content-length: 0
<
* we are done reading and this is set to close, stop send
* Closing connection 0** HTTP/1.1 302 Found
/dex/auth로 인하여 인증용 Redirect가 발생한다.
** Dex 인증 우회
1) 적용된 Auth filter 확인
$ kubectl get envoyfilters.networking.istio.io authn-filter -n istio-system
NAME AGE
authn-filter 4d5hKFServing, Knative serving, Istio Virtual service 등를 호출할 경우 '/dex/auth'로 리다이텍트 되면서 인증을 요구 한다.
✓ 요청할 URL에 대하여 Dex 인증을 우회하거나,
✓ 사전에 인증 과정을 거쳐서 authservice_session 값을 얻은 후, 요청시 Token을 전달하면 된다.
* Dex 인증을 제외할 URL를 아래와 같이 EnvoyFilter에 적용한다
$ cat bypass-auth-flowers-sample.yaml
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: bypass-auth-flowers-sample
namespace: istio-system
spec:
workloadSelector:
labels:
istio: ingressgateway
configPatches:
- applyTo: VIRTUAL_HOST
match:
routeConfiguration:
vhost:
name: 104.198.13.157:32304
patch:
operation: MERGE
value:
per_filter_config:
envoy.ext_authz:
disabled: true* Auth Filter 적용
$ kubectl apply -f bypass-auth-flowers-sample.yaml
custom image : reddiana/jupyterlab-elyra
kubeflow 삭제는 못찾았고.
결국 Kubenetes를 재구성 하는 방법으로 진행한다.
[master]
kubeadm reset cleanup-node
[worker]
kubeadm reset
실행