PSO and “Failed to Log in to Any iSCSI Targets.”

So I create and destroy Kubernetes clusters on vSphere on a pretty regular basis. Some I create with Terraform and Ansible. Some I use PKS. I have a plumbing test for Pure Service Orchestrator that mounts a single volume to a pod on each node.

Every once in a while I get an error like this, on just one node:

Failed to log in to any iSCSI targets! Will not be able to attach volume

In order to make sure it isn’t PSO with the error and it shouldn’t be since the other nodes are working. Run this command:

iscsiadm -m discovery -t st -p 192.168.230.24
iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
 iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.230.24,3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479]
 iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
 iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.230.25,3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479]
 iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
 iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.230.26,3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479]
 iscsiadm: Could not stat /etc/iscsi/nodes//,3260,-1/default to delete node: No such file or directory
 iscsiadm: Could not add/update [tcp:[hw=,ip=,net_if=,iscsi_if=default] 192.168.230.27,3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479]
 192.168.230.24:3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479
 192.168.230.25:3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479
 192.168.230.26:3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479
 192.168.230.27:3260,1 iqn.2010-06.com.purestorage:flasharray.4ca976f28eb0d479

Now I that isn’t what should be the result. So I thought at first to restart iscsi and that didn’t help. Then I thought, well this is a lab so lets just…

#cd /etc/iscsi
#rm -r nodes

Do not try this if you have other iSCSI targets for other storage. Not sure you will be happy. At first, I thought I should stop iSCSI before doing this. It doesn’t seem to have any effect. Now every node is able to mount and start the pod. Pure Service Orchestrator is trying to mount that volume over and over so it didn’t take long to see everything showing the way I wanted.

NAME                                        READY   STATUS    RESTARTS   AGE
 pure-flex-4zlcq                             1/1     Running   0          12m
 pure-flex-7stfb                             1/1     Running   0          12m
 pure-flex-g2kt2                             1/1     Running   0          12m
 pure-flex-jg5cz                             1/1     Running   0          12m
 pure-flex-n8wkw                             1/1     Running   0          6m34s
 pure-flex-rtsv7                             1/1     Running   0          12m
 pure-flex-vtph2                             1/1     Running   0          12m
 pure-flex-w8x22                             1/1     Running   0          12m
 pure-flex-wqr9k                             1/1     Running   0          12m
 pure-flex-xwbww                             1/1     Running   0          12m
 pure-provisioner-9c8dc9f79-xrq6d            1/1     Running   1          12m
 redis-master-demolocal-1-779f74876c-9k24t    1/1     Running   0          12m
 redis-master-demolocal-10-6695b56f47-zgqc7   1/1     Running   0          12m
 redis-master-demolocal-2-778666b57-5xdh8     1/1     Running   0          6m3s
 redis-master-demolocal-3-84848dfb87-fhj6n    1/1     Running   0          12m
 redis-master-demolocal-4-7c9dfdffb9-6cjv5    1/1     Running   0          12m
 redis-master-demolocal-5-65b555fc79-jjdkl    1/1     Running   0          12m
 redis-master-demolocal-6-6d495bfdf-cb5r2     1/1     Running   0          12m
 redis-master-demolocal-7-5c5db655-fx2qd      1/1     Running   0          12m
 redis-master-demolocal-8-74bc65b8d9-2bt8h    1/1     Running   0          12m
 redis-master-demolocal-9-65dd54c587-zb9p2    1/1     Running   0          12m

Leave a Reply

Your email address will not be published. Required fields are marked *