SaltStack: execute state modules on Master and minions from runner

2020-03-16T13:21:12Z

---

Contents

• Introduction:
• Pre-Requisites

---

Introduction:

We will be creating a SaltStack runner module. This runner module will

• call a state module on salt-master itself
• execute a state file (hence state module) on a client machine

Pre-Requisites

• salt-master installed and configured on a machine. I used fedora-24 VM.
• salt-minion installed and configured to accept command from salt-master.

/etc/salt/master file used:

log_level: debug

fileserver_backend:
  - roots

file_roots:
  base:
    - /srv/salt

runner_dirs:
  - /srv/salt/_runners

/etc/salt/minion file used:

master: salt

Note1: In /etc/salt/master file, log_level is set to debug. Normally this will be info.

Note2: we have to explicitly specify runners directory. In case of executions modules and state modules this is not required. we also need to restart salt-master if any entry is added to /etc/salt/master file.

mkdir /srv/salt/_runners
touch /srv/salt/_runners/custom_runner.py

Contents of /srv/salt/_runners/custom_runner.py are as follows:

import salt.client
import salt.loader
import salt.config
import pprint

client = salt.client.LocalClient()

def call_state_mod_on_saltmaster_from_runner():
    __opts__ = salt.config.minion_config('/etc/salt/master')
    __grains__ = salt.loader.grains(__opts__)
    __opts__['grains'] = __grains__
    __utils__ = salt.loader.utils(__opts__)
    __salt__ = salt.loader.minion_mods(__opts__, utils=__utils__)

    # state_mods is not being populated with all the state modules. Perhaps arguments need to be set.
    state_mods = salt.loader.states(__opts__, None, None, None)
    execution_mods = salt.loader.minion_mods(__opts__)

    # print all state modules available in this runner
    #for mod in state_mods.items():
    #    print(mod)

    # print all execution modules available in this runner
    #for mod in execution_mods.items():
    #    print(mod)

    # Note that salt.wait_for_event is a state module that is available. But pkg.installed state module is not available.
    ret = state_mods['salt.wait_for_event'](name='myevent/test', id_list=['fedora'], timeout=10)
    #pprint.pprint(ret)
    return ret

def execute_state_file_on_minion_from_runner():
    #ret = client.cmd('*', 'state.sls', ['first'])

    # Another Variation
    ret = client.cmd('*', 'state.sls', kwarg={'mods': 'first'})
    #pprint.pprint(ret)
    return ret


if __name__ == '__main__':
    #call_state_mod_from_runner()
    execute_state_mod_on_minion_from_runner()

Now we are ready to execute this runner.

Call a state module on salt-master itself via runner.

root@salt-master# salt-run custom_runner.call_state_mod_on_saltmaster_from_runner
changes:
    ----------
comment:
    Timeout value reached.
name:
    myevent/test
result:
    False
[INFO    ] Runner completed: 20170109175501129807

Note: Please note that wait_for_event is a state module available to salt-Master. There are only a handful of state modules available to salt-master. Most of the state modules are for clients. Read comments in custom_runner.py file for more information.

Execute a state file from runner onto minion (client):

root@salt-master# salt-run custom_runner.execute_state_file_on_minion_from_runner
fedora2.vagrant.box:
    ----------
    pkg_|-install tcpdump_|-tcpdump_|-installed:
        ----------
        __id__:
            install tcpdump
        __run_num__:
            0
        changes:
            ----------
        comment:
            Package tcpdump is already installed
        duration:
            355.026
        name:
            tcpdump
        result:
            True
        start_time:
            17:59:58.929729
[INFO    ] Runner completed: 20170109175953282083

Saltstack Custom Execution Module With Arguments

2020-03-16T13:41:26Z

---

Contents

• Introduction:
• Pre-Requisites

---

Introduction:

We will be creating a simple SaltStack execution module. And will see how can this module be called

• from a state file
• from command line

Pre-Requisites

• salt-master installed and configured on a machine. I used fedora-24 VM.
• salt-minion installed and configured to accept command from salt-master.

/etc/salt/master file used:

log_level: debug

fileserver_backend:
  - roots

file_roots:
  base:
    - /srv/salt

/etc/salt/minion file used:

master: salt

Note: In /etc/salt/master file, log_level is set to debug. Normally this will be info.

my_module.py will be our execution module

mkdir /srv/salt/_modules
touch /srv/salt/_modules/my_module.py

Contents of /srv/salt/_modules/my_module.py are as follows:

#! /usr/bin/env python

# Below virtual name can be any string.
__virtualname__ = 'my_module'

def __virtual__():
    return __virtualname__

def my_module_function(arg1, arg2):
    modified_arg1 = '%s_%s' % (arg1, 'modified')
    modified_arg2 = '%s_%s' % (arg2, 'modified')
    return modified_arg1, modified_arg2

Now we are ready to execute this module to our minion (client machine). Run following command to sync modules first to all minions.

root@salt-master# salt '*' saltutil.sync_all
fedora2.vagrant.box:
    ----------
    beacons:
    engines:
    grains:
    log_handlers:
    modules:
        - modules.my_module
    output:
    proxymodules:
    renderers:
    returners:
    sdb:
    states:
    utils:

Now we are ready to execute this module to our minion (client machine). Run following command to sync modules first to all minions.

root@salt-master# salt 'fedora2.vagrant.box' my_module.my_module_function 'a_string' 'b_string2'
fedora2.vagrant.box:
    - a_string_modified
    - b_string2_modified

Now run above execution module using a state file.

root@salt-master# touch /srv/salt/state_file_call_custom_exec_module.sls

Contents of /srv/salt/state_file_call_custom_exec_module.sls are as follows:

run_custom_module:
  module.run:
    - name: my_module.my_module_function
    - arg1: a_string
    - arg2: b_string

Run the following command to execute above state file on client machine:

root@salt-master# salt 'fedora2.vagrant.box' state.sls state_file_call_custom_exec_module
fedora2.vagrant.box:
----------
          ID: run_custom_module
    Function: module.run
        Name: my_module.my_module_function
      Result: True
     Comment: Module function my_module.my_module_function executed
     Started: 16:38:50.250250
    Duration: 1.085 ms
     Changes:
              ----------
              ret:
                  - a_string_modified
                  - b_string_modified

Summary for fedora2.vagrant.box
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:   1.085 ms

JSON: Full paths from root node to leaf node

Json Full Paths From Root Node to Leaf

2020-03-16T22:09:29Z

---

Contents

• Introduction:

---

Introduction:

Often there is a requirement to create full paths from root node to leaf node in JSON file. Following python program will generate these paths.

Sample JSON file sample_json_file.json is below:

{
   "data": [
      {
         "id": "X999_Y999",
         "from": {
            "name": "Tom Brady", "id": "X12"
         },
         "message": "Looking forward to 2010!",
         "actions": [
            {
               "name": "Comment",
               "link": "http://www.facebook.com/X999/posts/Y999"
            },
            {
               "name": "Like",
               "link": "http://www.facebook.com/X999/posts/Y999"
            }
         ],
         "type": "status",
         "created_time": "2010-08-02T21:27:44+0000",
         "updated_time": "2010-08-02T21:27:44+0000"
      },
      {
         "id": "X998_Y998",
         "from": {
            "name": "Peyton Manning", "id": "X18"
         },
         "message": "Where's my contract?",
         "actions": [
            {
               "name": "Comment",
               "link": "http://www.facebook.com/X998/posts/Y998"
            },
            {
               "name": "Like",
               "link": "http://www.facebook.com/X998/posts/Y998"
            }
         ],
         "type": "status",
         "created_time": "2010-08-02T21:27:44+0000",
         "updated_time": "2010-08-02T21:27:44+0000"
      }
   ],
  "name": "Anonymous",
  "job": { "hobbyjob": [ "Gardening", "InteriorDesign"], "Fulltimejob": "Lawyer" }
}

Python program to create full paths from above JSON file

#! /usr/bin/env python

# This programm will create paths from root nodes to leafnodes along with values from any json file or structure.

import json
import pprint


json_data = open('sample_json_file.json', 'r').read()
json_dict = json.loads(json_data)

stack = []
final_dict = {}

def do_walk(datadict):

    if isinstance(datadict, dict):
        for key, value in datadict.items():
            stack.append(key)
            #print("/".join(stack))
            if isinstance(value, dict) and len(value.keys()) == 0:
                final_dict["/".join(stack)] = "EMPTY_DICT"
            if isinstance(value, list) and len(value) == 0:
                final_dict["/".join(stack)] = 'EMPTY_LIST'
            if isinstance(value, dict):
                do_walk(value)
            if isinstance(value, list):
                do_walk(value)
            if isinstance(value, unicode):
                final_dict["/".join(stack)] = value
            stack.pop()

    if isinstance(datadict, list):
        n = 0
        for key in datadict:
            stack.append(str(n))
            n = n + 1
            if isinstance(key, dict):
                do_walk(key)
            if isinstance(key, list):
                do_walk(key)
            if isinstance(key, unicode):
                final_dict["/".join(stack)] = key
            stack.pop()


do_walk(json_dict)
pprint.pprint(final_dict)

Below is the result:

# python create_json_paths.py
{u'data/0/actions/0/link': u'http://www.facebook.com/X999/posts/Y999',
 u'data/0/actions/0/name': u'Comment',
 u'data/0/actions/1/link': u'http://www.facebook.com/X999/posts/Y999',
 u'data/0/actions/1/name': u'Like',
 u'data/0/created_time': u'2010-08-02T21:27:44+0000',
 u'data/0/from/id': u'X12',
 u'data/0/from/name': u'Tom Brady',
 u'data/0/id': u'X999_Y999',
 u'data/0/message': u'Looking forward to 2010!',
 u'data/0/type': u'status',
 u'data/0/updated_time': u'2010-08-02T21:27:44+0000',
 u'data/1/actions/0/link': u'http://www.facebook.com/X998/posts/Y998',
 u'data/1/actions/0/name': u'Comment',
 u'data/1/actions/1/link': u'http://www.facebook.com/X998/posts/Y998',
 u'data/1/actions/1/name': u'Like',
 u'data/1/created_time': u'2010-08-02T21:27:44+0000',
 u'data/1/from/id': u'X18',
 u'data/1/from/name': u'Peyton Manning',
 u'data/1/id': u'X998_Y998',
 u'data/1/message': u"Where's my contract?",
 u'data/1/type': u'status',
 u'data/1/updated_time': u'2010-08-02T21:27:44+0000',
 u'job/Fulltimejob': u'Lawyer',
 u'job/hobbyjob/0': u'Gardening',
 u'job/hobbyjob/1': u'InteriorDesign',
 u'name': u'Anonymous'}