With the latest release of Pure Service Orchestrator, we added support for a non-Helm installation for environments that do not allow Helm. This new method uses an Operator to setup and install PSO. The result is the same exact functionality but uses a security model more agreeable to some K8s distro vendors.
I do live demos of PSO a handful of times a day. Even though I use Terraform and Ansible to automate the creation of my lab K8s clusters I don’t want to do this many times a day. I usually just tear down PSO and leave my cluster ready for the next demo.
Removing the CRD and the Namespace created when installing the Operator has a couple of issues. One small issue is the Operator method creates a new namespace “pso-operator”. This is the default name, and you can choose your own namespace name during install time. I often choose “pso” for simplicity. As we have discovered, deleting a namespace that had a CRD installed into hangs in the status “Terminating”, for like, forever. FOR-EV-ER. This seems to be an issue dating back quite a ways in K8s land.
https://github.com/kubernetes/kubernetes/issues/60807#issuecomment-448120772
From a couple of GitHub issues and the help of Simon “I don’t do the twitter” Dodsley This is the process for deleting the CRD first and the Namespace. This method keeps the namespace form hanging in the state “Terminating”.
# Removing the pso-operator
kubectl delete all --all -n pso-operator
# If you haven't don't it already don't delete the namespace yet.
kubectl get ns
NAME STATUS AGE
default Active 2d21h
kube-public Active 2d21h
kube-system Active 2d21h
pso-operator Active 14h
kubectl get crd
NAME CREATED AT
psoplugins.purestorage.com 2019-04-17T01:37:31Z
# ok so...
kubectl delete crd psoplugins.purestorage.com
customresourcedefinition.apiextensions.k8s.io "psoplugins.purestorage.com" deleted
# does it hang? yeah it does
^C
# stuck terminating?
kubectl describe crd psoplugins.purestorage.com
# snipping non-relevant output
...
Conditions:
Last Transition Time: 2019-04-17T01:37:31Z
Message: no conflicts found
Reason: NoConflicts
Status: True
Type: NamesAccepted
Last Transition Time: <nil>
Message: the initial names have been accepted
Reason: InitialNamesAccepted
Status: True
Type: Established
Last Transition Time: 2019-04-18T13:54:36Z
Message: CustomResource deletion is in progress
Reason: InstanceDeletionInProgress
Status: True
Type: Terminating
Stored Versions:
v1
# Run this command to allow it to delete
kubectl patch crd/psoplugins.purestorage.com -p '{"metadata":{"finalizers":[]}}' --type=merge
customresourcedefinition.apiextensions.k8s.io/psoplugins.purestorage.com patched
# Re-run the crd delete
kubectl delete crd psoplugins.purestorage.com
# Confirm it is gone
kubectl get crd
No resources found.
# Remove the Namespace
kubectl delete ns pso-operator
namespace "pso-operator" deleted
#Verify removal
kubectl get ns
NAME STATUS AGE
default Active 2d21h
kube-public Active 2d21h
kube-system Active 2d21hIf you sort of ignored my warning above and tried to remove the namespace BEFORE successfully removing the CRD follow the following procedure.
Namespace Removal
# Find that pesky 'Terminating' namespace
kubectl get ns
NAME STATUS AGE
default Active 2d20h
kube-public Active 2d20h
kube-system Active 2d20h
pso Active 13h
pso-operator Terminating 35h
kubectl cluster-info
# run the kube-proxy
kubectl proxy &
# output the namespace to json
kubectl get namespace pso-operator -o json >tmp.json
# Edit the tmp.json to remove the finalizer the spec: should look like this:
"spec": {
"finalizers": [
]
},
# Now send that tmp.json to the API server
curl -k -H "Content-Type: application/json" -X PUT --data-binary @tmp.json http://127.0.0.1:8001/api/v1/namespaces/pso-operator/finalize
# Check your namespaces
kubectl get ns
NAME STATUS AGE
default Active 2d20h
kube-public Active 2d20h
kube-system Active 2d20h
pso Active 13h
# disable the kube-proxy, bring it back to the foreground and ctrl-C
fg
^C