====== Ansible ====== This infrastructure section is both a information source of current ansible usage options and a guide on how to use ansible. Ansible is a simple it automation software. It uses ssh to connect to machines and deploy/configure/install software and more. More information can be found on their [[https://www.ansible.com/|website.]]\\ It is basically an easy way to execute a command on many machines at the same time or many commands on one machine. For detailed information about documentation visit [[http://docs.ansible.com/|ansible-doc]] ===== Start Ansible ===== There is one Controller Machine: the machine where Ansible is installed, responsible for running the provisioning on the servers you are managing. Most of the time it is installed on the administrators working station. From there ssh keys need to be set up for all machines that should be managed by ansible.\\ To download the newest version of ansible add the the repository as described: sudo apt-add-repository ppa:ansible/ansible sudo apt-get update ---- Set up already configured ssh hosts in ''..ansible-scripts/hosts''. Already configured = Passwordless ssh login on the hosts with ssh public keys, configured in the ~/.ssh/config file.\\ ==== Introduction ==== You can group hosts with ''[some-name]''. Later you can specify commands only for some groups.\\ Now you can use different commands: ^Command^Effect^ |ansible all -m ping| Ping all nodes| |ansible all -m ping -u paulth | Ping all nodes as user paulth| |ansible new -m ping -u paulth -b | Ping all nodes in group new as user paulth with privilige escalation as sudo| |ansible all -m ping -u paulth -b --become-user batman | Ping all nodes as paulth sudoing to batman| |ansible all -m setup | Show all information of remote system| To run a live command use the a flag: ansible all -a "/usr/bin/whoami" #run with 10 parallel forks ansible new -a "/sbin/reboot" -f 10 ---- The next step is to differentiate between ad-hoc commands and playbooks. Ad-hoc commands are for quick tasks and commands that you don't want to save for later. Playbooks are different steps and commands packed into one yml file. They are useful for basic setup.\\ In the table above there were already examples for ad-hoc commands. For more commands have a look at this [[https://docs.ansible.com/ansible/intro_adhoc.html | site]].\\ For Playbooks have a look at the next section: ===== Playbooks/Roles ===== Executing a playbook is done from the root of the ansible-scripts directory e.g.: ansible-playbook plays/vault_test.yml This is a list with available playbooks that can be executed. the playbook most likely contains roles that are documented below: == Usage/Result:== Checks if the ansible vault key is correct and displays all variables defined for group named 'test'\\ ==Variables:== ^Name^Note^ |test_variable|variable stored in usual text-file and showed in the output| |test_secret_variable|variable stored encrypted in an ansible vault file and also shown in the playbook run output!| == Modules: == ^Name^Parameters/Keys^Funktion |debug|msg|to display statements during execution we used the module ''debug''.The key ''msg'' is used for string outputs| == Notes: == In the playbook we could have used the original name of the secret variable as they were defined in the vault file but for practical reasons we prefer to store all variables in one file and when needed define secret variables separately and redefine them additionally in the non-secret file. ==Usage/Result== Deploy a full [[https://snipeitapp.com/|SnipeIT]] installation. With an apache and Mysql database. The Result should be a fresh SnipeIT Installation available under snipeit_env_url, ready to create admin user and further configurations. \\ ==Important Variables== ^Name^Roles^Note^ |common_hostname_chair|common|was defined inside the playbook and committed to the role 'common'| |apache2_sites|apache|contains fqhn and the webroot| |mysql_root_pass|mysql|MYSQL root password stored under defaults/vault.yml| |mysql_user_pass|mysql|MYSQL user password stored under defaults/vault.yml| ==Basic structure == this part explains the first part of the playbook (all other parts contain almost the same basic elements and are therefore ignored). name: Perform common Tasks # the name of the task that would be displayed if you run your playbook host: snipeit # name of the remote-machine you want to perform you tasks on remote_user: i11 # connect to the remote-machine as the named user become: yes # execute all commands as superuser -> No password is required! gather_facts: false # if false is set, no facts(variables) about the remote-system # are discovered and saved tasks: - include_role: # list of directories of roles which are used name: roles/common vars: # list of different variables (one variable can have more than common_hostname_chair: snipeit-dev # one attribute: -> var-name.attribute) tags: common # if you want to exclude or specify the tasks you want to # perform in your playbook use ad-hoc commands: # --tags ; or # --skip-tags ==Included Roles== Fore more Information about the content and modules of each role go to the next chapter ^Name^main function^ |common|set the hostname to snipeit-dev.cm.in.tum.de| |apache|deploy a apache server| |mysql|secure installation of mysql| |snipeit|installation of snipeit and preparation of all important folders and files| == Usage/Result:== $ ansible-playbook plays/adduser.yml -e "username=test sudo= host=testhost" # example execution Results in a new user ''test'' with passwordless sudo rights on every host from the [testhost] - block from the hosts file. ==Variables:== ^Name^Required^Note^Where to put it?^ |username|yes|name of the user to create|within -e quote| |sudo|no|if set (a simple ''sudo='' is enough), gives the created user sudo rights (adds him to adm group)|within -e quote| |host|no|define another hostblock to execute the playbook on (default is "add_users")|within -e quote| |ssh-key|yes|has to exist as .pub|pubkeys folder| |remote_user|yes|has to have sudo rights and passwordless ssh login on hosts|edit in playbook| == Notes: == This playbook **requires** a ssh-key called .pub in the pubkeys folder, because it automatically adds it to the authorized_keys file of the created user on the hosts. If you ''don't'' want the user to create to have sudo rights, then don't pass a sudo= as an extra variable. Same goes for the host. Only pass those variables if you want to make use of them! == Usage/Result:== $ ansible-playbook plays/ssh_keys.yml -e "username=test replace= host=testhost key_to_replace=replace" # example execution replace Replaces the key replace.key with test.key in authorized_keys file on every host in the [testhost] block from the hosts file. $ ansible-playbook plays/ssh_keys.yml -e "username=test append=" # example execution append Appends the key test.key to authorized_keys file of user test on every host in the [ssh_keys] block from the hosts file. $ ansible-playbook plays/ssh_keys.yml -e "username=test delete= host=otherhost" # example execution delete Deletes the key test.key from authorized_keys file of user test on every host in the [otherhost] block from the hosts file. ==Variables:== ^Name^Required^Note^Where to put it?^ |username|yes|name of the user to edit the ssh-keys|within -e quote| |host|no|define another hostblock to execute the playbook on (default is "ssh_keys")|within -e quote| |ssh-key .pub|yes|has to exist at all times for execution|pubkeys folder| |key_to_replace|no|Optional: name of the key to replace ''without'' .pub extension (default: _delete.key). It (either variant) has to exist in the pubkeys folder!|within -e quote| |append|Option 1|If defined, appends the .key to authorized_keys file|within -e quote| |replace|Option 2|If defined, replaces (if passed as variable) .pub **or** _delete.pub with .pub for in authorized_keys file|within -e quote| |delete|Option 3|If defined, deletes .key from authorized_keys file of |within -e quote| |remote_user|yes|has to have sudo rights and passwordless ssh login on hosts|edit in playbook| == Notes: == This playbook **requires** a ssh-key called .pub in the pubkeys folder, because it is the key, that will be appended/replaced with/deleted to/in/from the authorized_keys file of the user on the hosts. The key_to_replace variable is optional, which means you don't need to pass it for successful execution! It will try to use _delete.pub instead, which has to exist in this case!\\ **You can only pass and use ''one'' option for execution!** === Functionality === Parses the output of ''getent passwd'' as well as ''/etc/login.defs'' to list all local users on an OpenNebula or RBG-VM. The users are then stored in the according database. An entry for the VM is created if not already present. Note that this only works for local users; **LDAP-Users** will **NOT** be listed! === Functionality === Reads the contents of ''/etc/login.user.allowed'' and ''/etc/security/group.conf'' and creates corresponding user entries in the VM-database === Functionality === retrieves the last date every user logged into the virtual machine and stores the result in the correct database (For either ONE or RBG-VMs). The variable ''users'' should be defined in the hostvars. === Functionalit === clears and recreates the files ''/etc/login.user.allowed'' and ''/etc/security/group.conf'', which control which users are allowed to log in and which users are granted sudo priviledges via the ''adm'' group. === Variables === ^Name^Type^Notes^ |users|List of String|defines which users are allowed to log in. Should be defined in hostvars| |sudousers|List of String|These users are added to the ''adm'' group upon login. Should be defined in hostvars| === Functionality === Instantiates a new OpenNebula VM and creates a corresponding database entry. === Variables === ^Name^Type^Notes^ |vm_name|String|will also be the Hostname of the VM| |purpose|String|What the VM will be used for| |network|String|either mwn or chair| |users|List of String|Which users are allowed to log in| |sudousers|List of String|Which users are granted sudo access| |survey_cpu|Float|CPU Power allocated to VM| |survey_vcpu|Integer|Number of CPUs the VM sees| |survey_disk|String|Disk size with suffix (KB, MB or GB)| |survey_memory|String|RAM size with Suffix| |services|List of String|list of services to be installed on the VM (see deploy_host)| |custom_vars|JSON Dictionary|list of extra variables (can be used to set configurations for services, etc.)| === Functionality === Creates necessary entries in ''/etc/login.user.allowed'' and ''/etc/security/group.conf'' to allow new user to log in and/or give them sudo access. Also creates or updates the corresponding database entries. === Variables === ^Name^Type^Notes^ |login_users|List of String|New users to grant ssh login access| |sudo_users|List of String|Users to grant sudo access. They are automatically also allowed to log in via ssh| ===== Roles ===== == Functionality == Changes the hostname in /etc/hosts accordingly to the common_hostname_chair variable and disables SSH root login in all instances (do ''NOT'' change this, unless you ''REALLY'' need to). It also disables password authentication for SSH by default. == Important Variables == ^Variable^Type^Function^ |common_hostname_chair|String|The system identifier under which the machine should be known to DNS and other hosts| |disable_ssh_pw_login|Boolean|Default: yes. Change to no, if password authentication is desired| == Functionality == Rolls out a whole MySQL setup with databases on the remote machine. == Important Variables == ^Name^Type^Function^ |mysql_root_pass|Vault Variable|MySQL root password. ''Edit in the vault.yml file, and ensure re-encryption after editing''| |mysql_user_pass|Vault Variable|Default password for new MySQL-users. ''Edit in the vault.yml file''| |mysql_dbs|Array|Consists of database **name** and **state**. Those are the databases, MySQL will have available after installation| |mysql_users|Array|Consists of user **name**, **state**, **password** and **priv**iliges the user will have| |mysql_secure_installation|Boolean|Default: yes. Deletes anonymous user ''""'' for the database| |mysql_service_enabled|Boolean|Default: yes. Whether MySQL should be enabled, after installation| |mysql_service_state|started/stopped|Default: started. The state MySQL will be, after installation| === Functionality === Enables LDAP-Authentication on a machine. migrates local users to ldap users if possible **Note: ** A former configuration used the file ''/etc/login.user-sudo.allowed'' to manage which users are granted sudo access. Now every user gets an entry in ''/etc/security/group.conf''. The task-file ''migrate-sudo-users.yml'' may be used to update the configuration and create the new entries. === Important Variables === ^Variable Name^Type^Function^ |users|List of String|LDAP users allowed to log in (should be in hostvars)| |sudousers|List of String|LDAP users grantedd sudo access (should be in hostvars)| == Functionality == Rolls out a whole Apache Installation with a custom webpage (content is needed though) and SSL and Redirects, if necessary and configured. For SSL functionality, you need to insert the required files in the ssl_certs directory of the apache role directory and define the correct filenames in the playbook (see SSL variables). == Important Variables == ^Variable^Type^Function^ |apache2_packages|Array|Default: apache2. Packages needed for the Apache Installation| |apache2_sites|Array|**id**: identifier used in sites-available\\ **fqdn**: Domain name used for the webpage\\ **webroot**: Directory, where the webpage content will be stored and accessed from\\ **ip**: IP adress, under which the webpage will explicitly be able to be accessed from\\ **port**: Port number used for the webpage\\ **state**: Whether the webpage will be available after installation| |ssl|Array|**port**: Port number under which SSL will communicate\\ **key_name**: Name of the key file in ssl_certs\\ **cert_name**: Name of the certificate file in ssl_certs\\ **chain_name**: Name of the chain file in ssl_certs (this one is optional for SSL functionality)| |apache2_remove_default|Boolean|Default: no. Whether the Apache Default pages should be disabled during installation| |apache2_service_enabled|Boolean|Default: yes. Whether Apache should be enabled after installation| |apache2_service_state|started/stopped|Default: started. The state the Apache service should be in after installation| |apache2_modules|Array|**id**: Name of the module, which should be ''enabled''| ^Name^Parameter^Function^ TODO === Functionality === Provides tasks to instantiate virtual machines on OpenNebula and manage LDAP-users who have access to them. === Task Files === ^File^Functionality^Required Variables^Notes^ |instantiate.yml|instantiates a new virtual machine|vm_name, network, users, sudousers|includes tasks from addusers.yml| |addusers.yml|grants login/sudo access on a vm to new users by updating database|VM_ID, users, sudousers|sudousers are automatically also granted login permission| |removeusers.yml|revokes permissions on vm for specified users|VM_ID, users, sudousers|users specified in sudousers will still be able to log in, those specified in users will be revoked all permissions| |updateusers.yml|updates ''/etc/login.users.allowed'' and ''/etc/security/group.conf''.|users, sudousers|| === Variables === ^Variable Name^Type^Function^ |vm_name|String|Name of the virtual machine to instantiate| |network|String|Network to attach to VM. Either "chair" or "mwn"| |users|List of String|Users who are allowed to log in| |sudousers|List of String|Users who are allowed sudoaccess| ===== Write Playbooks ===== In general you are going to execute one Playbook. This Playbook consists of tasks, **Roles** or **Modules** which will be performed on the remote-machines you have listed on top of each task. If you wish to work with Variables you can define them either directly inside a playbook, or in a different file (see: directory structure). The concept Ansible Scripts are managed in this directory structure: group_vars/ # variables for specific host groups all.yml # variables which can be used by all roles/playbooks dbservers/ # variables for e.g. dbservers host group vars.yml # non secret variables vault.yml # secret variables protected by ansible vault ... # other group folders as dbserver play/ # folder with playbooks as one yml files roles/ # contains roles used in the playbooks common/ # folder with the role 'common' used for all servers, vms, etc. defaults/ # contains default variables for role named 'common' main.yml # non-secret variables vault.yml # secret variables which must be encryptd by ansible vault! tasks/ # tasks for common role main.yml # task file files/ #contains files which can be deployed via this role handlers/ #contains handlers, which may be used by this role or even anywhere outside this role mysql/ # mysql role folder ansible.cfg # scripts specific ansible variables, e.g. point to vault password hosts # text file with host groups, should contain all servers vpass # ansible vault file containing the vault password, ignored by git! * Add a new group_vars folder for the group your playbook is written for, create one ''vars'' and one ''vault'' yml file * Add group variables to the vars file, copy secret variables to vaul file and prepend with vault_, link values to vars file, e.g. mysql_root_pass: '{{ vault_mysql_root_pass }}' * This can also be done for specific roles under ''..role/defaults/vault.yml'', it must be separately included in the role tasks file ''..role/tasks/main.yml'' - name: Include Vault Variables include_vars: ../defaults/vault.yml * Encrypt vault file with ansible-vault ansible-vault encrypt group_vars/dbservers/vault.yml # use different command for decryption in order to change passwords, etc. ansible-vault decrypt group_vars/dbservers/vault.yml During the execution of an ansible-playbook passwords may be to be set for services. We save those passwords in the vault-file and encrypt those. In order to let ansible properly decrypt the files, ask your supervisor for the vault-password, safe it as ''vpass'' in the root directory of the repository and execute the Makefile.\\ ''It is **very** important to execute the Makefile at least once after cloning the repository and before the first commit. The Makefile sets up git hooks, that check whether every vault file in the repository is encrypted before every commit.''