Tag: swarm

My SSH Issue Docker Swarm hosts

That one time you all of sudden could not SSH into your Docker Swarm hosts?

I am writing this so I will remember to be smarter next time.

Ever Get this?

minas-tirith:~ jowings$ ssh scarif
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

I started to flip out and wondered why this just all of sudden happened on all four host in my swarm cluster. Was something actually nasty happening? Probably not, but you never know. I thought I broke the pub key on my mac. because I went into .ssh/known_hosts and removed the entry for my hosts as I quite commonly see this because I rebuild vm’s and hosts all the time. Then I got something different and got the same exact error from my Windows 10 machine.

Permission denied (publickey).

Pretty sure I didn’t break 2 different ssh clients on 2 different computers.
What did I do?

$docker stack deploy -c gitlab.yml gitlab

So I am keeping local git copies and thoughs I would be smart to have Gitlab to run this service in my home lab.

Problem in my zeal to have git use stander ssh tcp port 22 to push my repos up to the server I did this:

version: '3'
services:
web:
image: 'gitlab/gitlab-ce:latest'
restart: always
hostname: 'gitlab1'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab.2vcps.local'
ports:
- '80:80'
- '443:443'
- '22:22'

So basically my gitlab service was now available using tcp/22 on my entire cluster. Even though the container is only on one host they way Docker overlay networking works is any host in that cluster will forward the request for tcp/22 to that container. The container without my public key, the container that no matter my hostname does not have the same SSH “ID” as my actual hosts.
Bad move JO.
So don’t do that and stuff.

To fix:

version: '3'
services:
web:
image: 'gitlab/gitlab-ce:latest'
restart: always
hostname: 'gitlab1'
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab.2vcps.local'

ports:
- '80:80'
- '443:443'
- '12022:22'

I changed the port mapping for now. I can use HAPROXY later to use the virtual hostname and point traffic to the container.

$docker stack deploy -c gitlab.yml gitlab

and it updates the service with the new port mapping.

Using the Docker Volume Plugin with Docker Swarm

Remember the prerequisites. Check the official README for the latest information. Official README

Platform and Software Dependencies

Operating Systems Supported:

  • CentOS Linux 7.3
  • CoreOS (Ladybug 1298.6.0 and above)
  • Ubuntu (Trusty 14.04 LTS, Xenial 16.04.2 LTS)

Environments Supported :

  • Docker (v1.13 and above)
  • Swarm
  • Mesos 1.8 and above

Other software dependencies:

  • Latest iscsi initiator software for your operating system
  • Latest linux multipath software package for your operating system

Review: To install the plugin –


docker plugin install store/purestorage/docker-plugin:1.0 --alias pure

OR if you are annoyed by having to hit Y for the permissions the plugin requests.


docker plugin install store/purestorage/docker-plugin:1.0 --alias pure --grant-all-permissions

The installation process is the same as a standalone docker host except you must specify your clusterid. This is a unique string you assign to your swarm nodes.


docker plugin disable pure
docker plugin set pure PURE_DOCKER_NAMESPACE=<clusterid>
docker plugin enable pure

When you first install the Pure Volume Plugin the plugin is enabled. Docker will not allow you to modify the namespace while the plugin is in use. So we need to disable the plugin before making changes. This also means it is best to do this before creating and using any volumes.

Remember to put your API token and array management IP in the pure.json file under /etc/pure-docker-plugin/ – for each host.

More information Here

Demo for setting up Swarm and testing container failover

Previous post about installing the Plugin

Setting Docker_gwbridge Subnet

I had an issue with the Docker Swarm subnet automatically generated when I do:

$docker swarm init

Basically it was choosing the subnet my VPN connection was using to assign an IP to my machine on the internal network. Obviously this wreaked havoc on me being able to connect to the docker hosts I was working with in our lab.

I decided it would be worth it to create the docker_gwbridge network and assign the CIDR subnet for the network that would not overlap with the VPN.

$docker network create –subnet 192.168.249.0/24 docker_gwbridge

I did this before I created the swarm cluster. So far everything is working fine in the lab and I am able to SSH to the Docker Host and connect to the services I am testing on those machines. There may be other issues and I will report back as I find them.

MK_SPACEMOUNTAINB_7567608042