/openstack/kolla_archlinux_notes.org

* OVERVIEW
  Project Kolla uses Ansible scripts and Python 2 to build an
  Openstack cloud using Docker containers for each infrastructure
  component. Setting up Kolla on Archlinux requires some tweaks
  because the default 'python' binary is Python 3 instead of Python
  2, the latter  of which is required to run Ansible.

  Before you start, make sure you have 40G+ space left in /var
  because 40G will be taken up by /var/lib/docker to build all
  the necessary containers! If you don't have enough space under
  /var you will need to specify another path for docker to use
  for storing containers and images.
  https://forums.docker.com/t/how-do-i-change-the-docker-image-installation-directory/1169


** Install Project Kolla and tools

   The git repo contains Kolla tools, but to install Kolla cmds
   to system directories /usr/bin, you need to install kolla from
   pip2

   Archlinux includes the latest versions of docker from its default
   repos, so just make sure docker is installed locally (if not,
   simply 'pacman -S docker').

   'pip' for python 2 must also be installed (python2-pip)

   Unlike the instructions from
   http://docs.openstack.org/developer/kolla/quickstart.html
   http://docs.openstack.org/developer/kolla/dev-quickstart.html
   make sure to replace /pip/ with /pip2/ for Archlinux!

   #+BEGIN_SRC shell
   git clone https://git.openstack.org/openstack/kolla
   cd kolla
   sudo pip2 install -r requirements.txt
   #sudo pip2 install kolla
   #sudo cp -r etc/kolla /etc/
   sudo pip2 install -U python-openstackclient
   #+END_SRC

   Don't install the python2 pip package 'kolla'! It installs similar
   files to the ones installed by git clone, but the former installs
   files under /usr/share/kolla/...

   Instead of running commands from pip2 kolla package, I will run fresh
   commands from $HOME/kolla (the kolla git repo which was cloned above)

   Make sure you have chronyd or ntpd setup on localhost
   If not, install the package for one or the other and
   systemctl enable ... and systemctl start ...


** Customizations for Arch

  1. *Specify location of Python 2 binary*
     edit the ansible /inventory/ file (not /etc/ansible/ansible.cfg):
     #+BEGIN_SRC shell
     #sudo vim /usr/share/kolla/ansible/inventory/all-in-one
     vim $HOME/kolla/ansible/inventory/all-in-one
     #+END_SRC

     And add the following somewhere in the file
     #+BEGIN_SRC shell
     localhost       ansible_python_interpreter=/usr/bin/python2
     #+END_SRC

  2. *Install graphviz dependency for python 2 graphviz module from pip2*
     https://pypi.python.org/pypi/graphviz
     From the pip page,

     "This package facilitates the creation and rendering of graph
     descriptions in the DOT language of the Graphviz graph drawing
     software"

     I tried to find a native Arch package for python2 module graphviz
     but only came up with the following:

     #+BEGIN_SRC shell
     $ sudo pacman -Ss graphviz
     extra/graphviz 2.38.0-11
         Graph visualization software
     community/dot2tex 2.9.0-1
         Graphviz to LaTeX converter
     community/perl-graphviz 2.20-1
         GraphViz - Interface to AT&T's GraphViz. Deprecated. See GraphViz2
     community/python2-pydot 1.0.28-3
         Python interface to Graphviz's Dot language
     community/xdot 0.6-4
         Interactive viewer for graphs written in Graphviz's dot language
     #+END_SRC

     'graphviz' (a program written in C) is a dependency of python2 graphviz,
     so make sure to install it

     #+BEGIN_SRC shell
     sudo pacman -Syyu graphviz
     #+END_SRC

     python2 graphviz should have already been installed by pip2 when you
     invoked 'sudo pip2 install -r requirements.txt' from the kolla
     directory

** Build docker containers
   - Disable libvirtd
     Before you begin, make sure that libvirtd daemon is not running!

     #+BEGIN_SRC shell
     systemctl status libvirtd
     #+END_SRC

     Stop and disable if libvirtd is running

   - Disable firewalld
     docker daemon tries to append iptables rules, but these fail in
     firewalld. After stopping firewalld, restart the docker daemon

     #+BEGIN_SRC shell
     systemctl stop firewalld
     systemctl restart docker
     #+END_SRC

   - Build Kolla containers (using commands from cloned repo)

     #+BEGIN_SRC shell
     cd $HOME/kolla/tools
     python2 build.py --base [centos|fedora|ubuntu|oraclelinux]
     #+END_SRC

     You can also specify a single package with something like:
     #+BEGIN_SRC shell
     cd $HOME/kolla/tools
     python2 build.py openstack-base --base ubuntu
     #+END_SRC

     *Avoid using /kolla-build/ from the kolla pip package!*
     The problem is that this package is not as frequently updated
     as the git repo! I installed kolla-build and related prog's
     from the 'kolla' package from pip2; check the system logs to
     be sure...

     The nice thing about the ubuntu containers is that they include iproute2
     and other utilities that make container debugging convenient, unlike
     centos

     + Alternative method (not recommended)
       from pip2 kolla pkg:

       #+BEGIN_SRC shell
       kolla-build --base [centos|fedora|ubuntu|oraclelinux]
       #+END_SRC


** Customize globals.yaml
   /etc/kolla/globals.yaml contains the global config for Kolla

   - Internal Network Interface
     default is 'eth0' but I changed to 'enp1s0' as the only wired
     iface on the Lenovo S310

   - External Network Interface
     There is a wirless iface on wlp2s0 but it cannot be directly
     connected to a bridge. The default in globals.yaml is 'eth1'
     but I changed this value to point to one end of a virtual
     ethernet iface I created

** Start Kolla


   Running 'kolla-ansible' without any arguments will return a command
   reference

** Errors
   + Network connectivity issues due to firewalld running
     INFO:__main__:base:Step 9 : RUN yum -y install http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm
     INFO:__main__:base: ---> Running in 105c3cc123ca
     INFO:__main__:base:Loaded plugins: fastestmirror, ovl
     INFO:__main__:base:Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. Skipping.

     A clone of the centos7 container fails to yum install a
     package. After the container builds failed I checked the existing
     images with 'docker images'

     [archjun@pinkS310 tools]$ docker images
     REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
     <none>              <none>              aec129e7e87d        5 minutes ago       196.6 MB
     busybox             latest              0cb40641836c        4 weeks ago         1.114 MB
     centos              latest              61b442687d68        7 weeks ago         196.6 MB

     I then got a shell into the image with:

     #+BEGIN_SRC shell
     docker run -it aec129e7e87d
     #+END_SRC

     I verified that ping 8.8.8.8 works but that DNS resolution
     is not working at all despite /etc/resolv.conf containing KT's
     name servers

     *NOTE*: the problem turns out to be firewalld; docker attempts
     to insert iptables rules, but these fail in firewalld; until I
     figure out the proper settings for docker0 in firewalld, just
     disable it when running Kolla

     Finally I checked the existing containers

     [archjun@pinkS310 ~]$ docker ps -a
     CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
     b340327be74a        aec129e7e87d        "/bin/bash"         17 minutes ago      Up 17 minutes                           jovial_wing

     I then deleted all images and containers (except the busybox one).
     With some googling, I learned that recently centos7 base
     container image removed common utils like iproute2
     which makes network troubleshooting very difficult.

     https://github.com/CentOS/sig-cloud-instance-images/issues/40

     Using 'kolla-build --base' option you can choose the distro type
     in the containers being built (this is not possible if you simply
     run build.py)

     #+BEGIN_SRC shell
     kolla-build --base fedora
     #+END_SRC

   + Error in 'kolla-build --base ubuntu' openstack-base container
     INFO:kolla.cmd.build:openstack-base:Step 6 : RUN patch -d /usr/lib/python2.7/site-packages -p1 < /var/lib/kolla/oslo-log-bug1499620.patch
     INFO:kolla.cmd.build:openstack-base: ---> Running in 1996635b6ea6
     INFO:kolla.cmd.build:openstack-base:patching file oslo_log/log.py
     INFO:kolla.cmd.build:openstack-base:Reversed (or previously applied) patch detected!  Assume -R? [n]
     INFO:kolla.cmd.build:openstack-base:Apply anyway? [n]
     INFO:kolla.cmd.build:openstack-base:Skipping patch.
     INFO:kolla.cmd.build:openstack-base:1 out of 1 hunk ignored -- saving rejects to file oslo_log/log.py.rej
     INFO:kolla.cmd.build:openstack-base:Removing intermediate container 1996635b6ea6
     ERROR:kolla.cmd.build:openstack-base:Error'd with the following message
     ERROR:kolla.cmd.build:openstack-base:The command '/bin/sh -c patch -d /usr/lib/python2.7/site-packages -p1 < /var/lib/kolla/oslo-log-bug1499620.patch' returned a non-zero code: 1

     When the patch fails, all the subsequent container builds fail as well
     as they all depend on container 'openstack-base'
     I will try editing kolla/docker/openstack-base/Dockerfile.j2 and
     commenting out this line; patch is not contained in this file!

     + After git pull and updating repos, I verified that the erroneous patch is
       no longer showing up in openstack-base/Dockerfile.j2; but when I run
       kolla-build --base ubuntu, the old patch is still trying to be applied!
       I suspected that there was another openstack-base/Dockerfile.j2 somewhere
       on my system, and sure enough, I found it with:

       #+BEGIN_SRC shell
       sudo find / -type d -name openstack-base
       #+END_SRC
       returns the following two paths:
       /home/archjun/kolla/docker/openstack-base
       /usr/share/kolla/docker/openstack-base