Docker Logging With Rsyslog

2019-07-27T12:15:28+01:00

---

Contents

• Introduction
• Running Docker with syslog logging
• What we want:
  • Version of Rsyslog used
  • Configure rsyslog:
    • Details of regular expressions used above:
      • NOTE1:
      • NOTE2:
    • Restart rsyslog:
    • Other ways:
      • Using Latest template syntax, String type and Field values.
    • Using old template syntax and Using field values.
    • Using latest template syntax and list type
      • A consolidated file describing all above scenarios is below:

---

Introduction

This document will describe a simple strategy to logging for docker container using Rsyslog. Often we may have to run multiple containers on single machine. We may require logging for different container in different directories or files. This can be achieved using Rsyslog. Approach below is very generic and flexible and can be modified as per requirement easily.

Running Docker with syslog logging

I use following command to run docker container.

docker run --rm --log-driver=syslog  --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" ubuntu echo atestoutput

Above command will append a line similar to following line in /var/log/syslog file.

Jun  7 15:58:27 machine docker/ubuntu/trusting_dubinsky/b7d1373fccaf[20642]: atestoutput

What we want:

We want the logging of container to go in its own directory. E.g. We probably want that each docker container should log into /var/log/company/dockerapps/containerName/docker.log. For a few containers we may hard code into rsyslog.conf file. But for a number of containers, we will have to use more generic configuration of rsyslog.

Take a look at pattern in above output i.e docker/ubuntu/trusting_dubinsky/b7d1373fccaf[20642]. This is called syslog tag and rsyslog will store this pattern in its property variable which can be accessed by referring to syslogtag in rsyslog.conf file. We will exploit this property to generate dynamic filenames and directory structure.

Version of Rsyslog used

$ rsyslogd -v
        rsyslogd 8.16.0, compiled with:
        PLATFORM:                               x86_64-pc-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        Number of Bits in RainerScript integers: 64

Configure rsyslog:

Create a regular file /etc/rsyslog.d/40-dockerapp.conf. We will put our configuration in this file. rsyslog comes with various defaults in /etc/rsyslog.conf and /etc/rsyslog.d/50-default.conf. We named our file as 40-dockerapp.conf, because we want to get it executed before 50-default.conf. Populate /etc/rsyslog.d/40-dockerapp.conf file with following contents.

# To create logging directories/filenames dynamically.
template(name="Dockerlogfiles" type="string" string="/var/log/company/dockerapps/%syslogtag:R,ERE,2,FIELD:docker/(.*)/(.*)/(.*)\\[--end%/%syslogtag:R,ERE,3,FIELD:docker/(.*)/(.*)/(.*)\\[--end%/docker.log")

# To format message that will be forwarded to remote syslog server
template (name="LongTagForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:::sp-if-no-1st-sp%%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")

if $syslogtag startswith "docker" then {

    # Local logging
    action(name="localFiles" type="omfile" DynaFile="Dockerlogfiles")

    # Remote logging to remote syslog server. 
    action(name="forwardToOtherSyslog" type="omfwd" Target="IP ADDRESS of Target Syslog server" Port="514" Protocol="udp" template="LongTagForwardFormat")
    &~
}

Above template will generate filenames dynamically based on syslogtag pattern shown above in output. In this case we are using regular expressions. &~ will stop the processing of further rules for the message. If &~ is missing, then 50-default.conf will pickup the message, process it and will send it to /var/log/syslog file as well. So we will eventually end up having duplicate syslog entries in two different files.

Details of regular expressions used above:

regex details

%syslogtag:R,ERE,1,FIELD:docker/(.*)/(.*)/(.*)\\[--end%  == will generate ==> ImageName
%syslogtag:R,ERE,2,FIELD:docker/(.*)/(.*)/(.*)\\[--end%  == will generate ==> ContainerName
%syslogtag:R,ERE,3,FIELD:docker/(.*)/(.*)/(.*)\\[--end%  == will generate ==> ContainerID

Note: more accurately above statements are actually called property replacers using regex

NOTE1:

Rsyslog limits length of syslogtag to 32 characters when message is sent to remote rsyslog server(32 char limit is NOT there for local logging though).My tags are longer than 32 characters. Therefore I used another template that will not restrict the limit to 32 when sending the message to remote server. This template is taken from http://www.rsyslog.com/sende-messages-with-tags-larger-than-32-characters/. However I have modified this template.

NOTE2:

I have modified the above template "not to restrict syslogtag length limit to 32 chars" a bit as stated above. The reason for doing this is very subtle. My syslogtag is greater than 32 characters and it consists of format like "docker/A_very_long_docker_image_name:A_very_long_docker_tag_name/containerName/ContainerID:[somePID]". Take a careful look at ":" colon in the middle and near the end of above syslogtag(before somePID). While forwarding above syslogtag (i.e message containing this syslogtag) to remote rsyslog server, destination rsyslog server was creating a space after the middle : colon. In order to prevent this I added %syslogtag:::sp-if-no-1st-sp% construct in above mentioned template.

Restart rsyslog:

Restart rsyslog and test by starting container with command shown above. You will see a file docker.log in /var/log/company/dockerapps/containerName/containerID/docker.log

Note:Rsyslog will create directory structure automatically.

Other ways:

Dynamic filename generation can be done in other ways too. Following are the other ways to write above template.

Using Latest template syntax, String type and Field values.

template syntax

# Using Latest template syntax (String type and Field values)
#        /%syslogtag:F,47:1% => represents docker in $syslogtag . 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:2% => represents ImageNAme. 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:3% => represents ContainerName.
#        /%syslogtag:F,47:4% => represents ContainerID.
template(name="Dockerlogfiles" type="string" string="/var/log/company/dockerapps/%syslogtag:F,47:2%/%syslogtag:F,47:3%/docker.log")

Using old template syntax and Using field values.

old template syntax

#        /%syslogtag:F,47:1% => represents "docker" in $syslogtag . 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:2% => represents ImageNAme. 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:3% => represents ContainerName.
#        /%syslogtag:F,47:4% => represents ContainerID.
# $template Dockerlogfiles, "/var/log/company/dockerapps/%syslogtag:F,47:2%/docker.log"
$template Dockerlogfiles, "/var/log/company/dockerapps/%syslogtag:F,47:2%/%syslogtag:F,47:3%/docker.log"

Using latest template syntax and list type

Template syntax list type

#     regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="1") == will generate ==> imagename
#     regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="2") == will generate ==> containername
#     regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="2") == will generate ==> containerid
template(name="Dockerlogfiles" type="list") {
   constant(value="/var/log/company/dockerapps/")
   property(name="syslogtag" securepath="replace" \
            regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="2")\
   constant(value="/docker.log")
}

A consolidated file describing all above scenarios is below:

Full file

# We assume that a syslogtag generated by docker container is of following format
#        docker/ImageNAme/ContainerName/ContainerID
# An exmaple docker command to generate above tag is
#        docker run --rm --log-driver=syslog  --log-opt tag="{{.ImageName}}/{{.Name}}/{{.ID}}" ubuntu echo atestwithouttag
#    Above command will log following similar message
#        Jun  7 15:58:27 machine docker/ubuntu/trusting_dubinsky/b7d1373fccaf[20642]: atestwithouttag



# A very Simple way to filter messages
#:syslogtag, ereregex, "docker/ubuntu" /var/log/docker-syslog/syslog.log


# To create logging directories/filenames dynamically.
# Using latest template syntax (string type and regex). Below will create log file like : /var/log/company/dockerapps//docker.log
#     %syslogtag:R,ERE,1,FIELD:docker/(.*)/(.*)/(.*)\\[--end%  == will generate ==> imageName
#     %syslogtag:R,ERE,2,FIELD:docker/(.*)/(.*)/(.*)\\[--end%  == will generate ==> containerName
#     %syslogtag:R,ERE,3,FIELD:docker/(.*)/(.*)/(.*)\\[--end%  == will generate ==> containerID
# template(name="Dockerlogfiles" type="string" string="/var/log/company/dockerapps/%syslogtag:R,ERE,2,FIELD:docker/(.*)/(.*)/(.*)\\[--end%/docker.log")
template(name="Dockerlogfiles" type="string" string="/var/log/company/dockerapps/%syslogtag:R,ERE,2,FIELD:docker/(.*)/(.*)/(.*)\\[--end%/%syslogtag:R,ERE,3,FIELD:docker/(.*)/(.*)/(.*)\\[--end%/docker.log")

# To format message that will be forwarded to remote syslog server
template (name="LongTagForwardFormat" type="string" string="<%PRI%>%TIMESTAMP:::date-rfc3339% %HOSTNAME% %syslogtag:::sp-if-no-1st-sp%%syslogtag%%msg:::sp-if-no-1st-sp%%msg%")

# Using Latest template syntax (String type and Field values)
#        /%syslogtag:F,47:1% => represents docker in $syslogtag . 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:2% => represents ImageNAme. 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:3% => represents ContainerName.
#        /%syslogtag:F,47:4% => represents ContainerID.
# template(name="Dockerlogfiles" type="string" string="/var/log/company/dockerapps/%syslogtag:F,47:2%/%syslogtag:F,47:3%/docker.log")



# Using old template syntax and Using field values.
#        /%syslogtag:F,47:1% => represents "docker" in $syslogtag . 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:2% => represents ImageNAme. 47 is ASCII decimal value of / character
#        /%syslogtag:F,47:3% => represents ContainerName.
#        /%syslogtag:F,47:4% => represents ContainerID.
# $template Dockerlogfiles, "/var/log/company/dockerapps/%syslogtag:F,47:2%/docker.log"
# $template Dockerlogfiles, "/var/log/company/dockerapps/%syslogtag:F,47:2%/%syslogtag:F,47:3%/docker.log"




# Using latest template method (list type)
#     regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="1") == will generate ==> imageName
#     regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="2") == will generate ==> containerName
#     regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="2") == will generate ==> containerID
#template(name="Dockerlogfiles" type="list") {
#   constant(value="/var/log/company/dockerapps/")
#   property(name="syslogtag" securepath="replace" \
#            regex.expression="docker/\\(.*\\)/\\(.*\\)/\\(.*\\)\\[" regex.submatch="2")\
#   constant(value="/docker.log")
#
#}

if $syslogtag startswith "docker" then {
    # Local logging
    action(name="localFiles" type="omfile" DynaFile="Dockerlogfiles")

    # Remote logging to remote syslog server. 
    action(name="forwardToOtherSyslog" type="omfwd" Target="IP ADDRESS of Target Syslog server" Port="514" Protocol="udp" template="LongTagForwardFormat")
    &~
}

Pretty JSON print of a CouchDB document

Using Python:

$ curl --silent --insecure https://user:'password'@couchdb-server:5984/database/document | python -m json.tool

Using Ruby:

$ gem install json
$ curl --silent --insecure https://user:'password'@couchdb-server:5984/database/document | ruby -rjson -n -e 'str=JSON.parse $_ ;puts JSON.pretty_generate(str)'

$ curl --silent --insecure https://user:'password'@couchdb-server:5984/database/document | ruby -n -e 'require "json" ; str=JSON.parse $_ ;puts JSON.pretty_generate(str)'

Ruby Procs and Blocks relation

#! /usr/bin/env ruby
def amethod(arg=10)
  yield 2
  arg * 3
end

def bmethod(&block)
  block.call
end

def cmethod(block)
  block.call
end

p = Proc.new { puts "I am a proc object" }





puts "Below is the output when amethod is called as ---> str = amethod(4) { |num| puts num }"
str = amethod(4) { |num| puts num}
puts "str = amethod(4) { |num| puts num  } returned ---> #{str}"

puts "====================="
puts "Below is the output when amethod is called as ---> str1 = amethod { |num| puts num } ; i.e no argument to amethod."
str1 = amethod { |num| puts num }
puts "str1 = amethod { |num| puts num  } returned ---> #{str1}"

puts "====================="
puts "Below is the output when amethod is called as ---> amethod { puts \"This output is from code block that does not have block arg.\" } ; i.e no argument to amethod,  no code block arg and return value discarded."
amethod { puts "This output is from code block that does not have block arg." }

puts "====================="
puts "Below is the output when amethod is called as ---> amethod(&p); where p = Proc.new { puts \"I am a proc object\"}"
amethod(&p)

puts "===================="
puts "Below is the output when amethod is called as ---> ret = amethod(20, &p); where p = Proc.new { puts \"I am a proc object\"}"
ret=amethod(20, &p)
puts "ret = amethod(20, &p) returned: #{ret}"

puts "===================="
puts "Below is the output when bmethod is called as ---> bmethod { puts \"This output is from code block that does not have block arg.\" }"
bmethod { puts "This output is from code block that does not have block arg." }

puts "==================="
puts "Below is the output when bmethod is called as ---> bmethod(&p); where p = Proc.new { puts \"I am a proc object\"}"
bmethod(&p)

puts "==================="
puts "below is the output when cmethod is called as ---> cmethod(p); where p = Proc.new { puts \"I am a proc object\"}"
cmethod(p)

Following is the output of above code:

Below is the output when amethod is called as ---> str = amethod(4) { |num| puts num }
2
str = amethod(4) { |num| puts num  } returned ---> 12
=====================
Below is the output when amethod is called as ---> str1 = amethod { |num| puts num } ; i.e no argument to amethod.
2
str1 = amethod { |num| puts num  } returned ---> 30
=====================
Below is the output when amethod is called as ---> amethod { puts "This output is from code block that does not have block arg." } ; i.e no argument to amethod,  no code block arg and return value discarded.
This output is from code block that does not have block arg.
=====================
Below is the output when amethod is called as ---> amethod(&p); where p = Proc.new { puts "I am a proc object"}
I am a proc object
====================
Below is the output when amethod is called as ---> ret = amethod(20, &p); where p = Proc.new { puts "I am a proc object"}
I am a proc object
ret = amethod(20, &p) returned: 60
====================
Below is the output when bmethod is called as ---> bmethod { puts "This output is from code block that does not have block arg." }
This output is from code block that does not have block arg.
===================
Below is the output when bmethod is called as ---> bmethod(&p); where p = Proc.new { puts "I am a proc object"}
I am a proc object
===================
below is the output when cmethod is called as ---> cmethod(p); where p = Proc.new { puts "I am a proc object"}
I am a proc object

Sharing MacBook Pro's Keyboard and TouchPad with other Linux Machine

Introduction

I have a Dell Laptop (Fedora-23) connected to my Big screen TV via HDMI cable. I use Kodi to watch movies on TV. Though Kodi can be controlled via its web interface, but I wanted to control other things as well in Linux remotely via my MacBook pro. I found http://synergy-project.org/ as one of solution to this problem. Although it is an Open Source Project, they do not provide installers free of cost. So I performed following steps to build Synergy for my MacBook pro and Linux both.

Note: I tried VNC but was not happy with it.

Build Synergy for MacBook Pro:
I am running OSX 10.11.3. You may have to install XCODE. Additionally I am using HomeBrew as well.
You have to install "QT" on MacBook pro. I installed it using HomeBrew. I installed "4.8.7_1" QT.

$ brew install qt
$ brew install cmake

$ git clone http://github.com/symless/synergy
$ cd synergy

make changes to  ext/toolchain/commands1.py file as per https://github.com/symless/synergy/wiki/Compiling#Mac_OS_X_1010_and_above
I ran following commands to make relevant changes.
$ cd ext/toolchain/
$ perl -i -wnl -e 'm!(.*)elif sys\.platform == "darwin":(.*)! and print "$1if sys.platform == \"darwin\":$2" or print ' commands1.py
$ perl -i -wnl -e 'm!(.*)frameworkRootDir = ! and print "$1frameworkRootDir = \"/usr/local/Cellar/qt/4.8.7_1/Frameworks\"" or print ' commands1.py

$ cd ../../
$ ./hm.sh conf -g1 --mac-sdk 10.11 --mac-identity `hostname`
$ ./hm.sh build
$ ./hm.sh package mac

Mac package .dmg will be created. Install that .dmg file. First time it will not run and gives error.
To run it , Under system preference -> Security & Privacy -> Accessibility -> check Synergy .
Run Synergy (GUI app) and set it up as server.

Build synergy on Fedora-23

$ git clone http://github.com/symless/synergy
$ cd synergy

Follow instructions in https://github.com/symless/synergy/wiki/Compiling#Unix_Linux_Mac_OS_X (in short these consists of following steps)
$ ./hm.sh conf -g1
$ ./hm.sh build
$ ./hm.sh package rpm

RPM will be generated like following.
bin/synergy-master-beta-7a207b4-Linux-x86_64.rpm

Install above generated RPM. This Linux laptop will be client to Synergy Master running on MacBook Pro.
I use following command on my Linux Laptop to connect it to Synergy Master.

$ synergyc -f 192.168.0.7

192.168.0.7 is the IP ADDRESS of my MacBook Pro.
I usually run it in a screen session as following.

$ screen -S synergy synergyc -f 192.168.0.7

Once it is connected, MacBook Pro's mouse and keyboard can be shared in all three devices i.e (MacBook Pro, Linux Laptop and TV).

Understanding packet traversing in IPTABLES

Introduction

when a packet goes through Linux IPTABLES firewall, it goes through various in-built chains and tables. These chains and tables together determine the fate of packet. This article does not tell what will happen to the packet, but will tell how a packet journey goes when it enters IPTABLES. in-short a packet workflow is described.

There are various article that explain these workflow very well. This article will just describe a pattern that would be easy to apply when dealing with IPTABLES.

At this point a diagram worth mentioning is https://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg

IPTABLES has defined following TABLES.

1) raw
2) mangle
3) nat
4) filter

Each table has following in-built CHAINS defined in them.

1) raw => PREROUTING, OUTPUT
2) mangle => PREROUTING, OUTPUT, INPUT, FORWARD, POSTROUTING
3) nat => PREROUTING, OUTPUT, POSTROUTING
4) filter => INPUT, FORWARD, OUTPUT

Three possible scenarios:

a) If a packet is coming from outside and destined for localhost then it will go through following chains:
1) PREROUTING
2) INPUT

b) if a packet is originating from localhost and destined to go out then it will go through following chains:
1) OUTPUT
2) POSTROUTING

c) if a packet is coming from outside and is destined for some other machine then packet will go through following chains:
1) PREROUTING
2) FORWARD
3) POSTROUTING

Note: CHAINS and TABLES written above are in order.
Now what is the pattern I am talking about:

Take first scenario (packet coming from outside and destined for localhost):

Kernel goes through each TABLE in order (written above) and it applies PREROUTING chain if found in that TABLE onto packet.
Kernel does same above scan of each table while applying INPUT chain to packet.

For each chain i.e [PREROUTING, INPUT]: (in order left to right)
  kernel looks for the chain in each table i.e [raw, mangle, nat, filter]: (in order left to right)
    and if chain is found in the table:
      then it applies the chain to packet
      else does nothing.


That means: For this scenario, kernel
1) first scans raw TABLE and finds PREROUTING chain here and applies it onto packet.
2) then it scans mangle TABLE and finds PREROUTING chain there and applies it to packet.
3) then it scans nat TABLE and finds PREROUTING chain there and applies it to packet.
4) then it scans filter TABLE but does not find PREROUTING chain here and does nothing to packet.
5) same above steps (1 - 4 ) are repeated for INPUT chain.
6) then kernel scans raw TABLE but does not find INPUT chain, so nothing happens here.
7) then kernel scans mangle TABLE and finds INPUT chain here and applies it to packet.
8) then kernel scans nat TABLE and does not find INPUT chain, so nothing happens here.
9) then kernel scans filter TABLE and finds INPUT chain here and applies it to packet.

For second scenario (packet originating from localhost and destined to go out):

For each chain i.e [OUTPUT, POSTROUTING]: (in order left to right)
  kernel looks for the chain in each table i.e [raw, mangle, nat, filter]: (in order left to right)
    and if chain is found in the table:
      then it applies the chain to packet
      else does nothing.

For third scenario (packet coming from outside and destined for some other machine):

For each chain i.e [PREROUTING, FORWARD, POSTROUTING]: (in order left to right)
  kernel looks for the chain in each table i.e [raw, mangle, nat, filter]: (in order left to right)
    and if chain is found in the table:
      then it applies the chain to packet
      else does nothing.

Above pattern can be corroborated with diagram above. You have to look only into network layer in that diagram.

SaltStack: Install RPMs from a list in text file

Introduction
To describe a rpm package installation in SaltStack, normally a salt formula with contain both the name of RPM package and a instruction to install it. If list of RPMS is large and requires changes quite regularly then maintaining states file would be very cumbersome. As state files are ususally yaml files, a human typo/error would be quite frequent then. I had a similar kind of issue.

Therefore I created a Salt formula that would use just a plain list of RPMS files in a text file. So I would just update that text file with the list of RPMS.

Strategy
We will create pillar data from a plain text file having list of RPM names on Salt Master. This pillar data will be exposed to agents with "pillar_rpm_packages" pillar key. Following will be the structure of pillar data that will be created.

'pillar_rpm_packages': [ 'package1==ver1.rel1.x86_64', 
                         'package2==ver2.rel2.x86_64',
                         'package3==ver3.rel2.x86_64', ]

Then we will create a salt module custmod_rpm_packages_from_list (on SaltMaster). This module will have a function called "create_packages_states". This salt module will be pushed to salt agents. Agents will call this module.function as custmod_rpm_packages_from_list.create_packages_states. This function accepts "pillar_rpm_packages" as its argument and then generates "json" formatted states of package installation.
create_packages_states function will create similar to following json structure.

{'pkg': ['installed', {'version': pack_version }]}

Please note that we wanted to have package installation states generated with version number included, so that only a specific version of RPM package can be installed not latest one.

Structure
This setup assumes that Salt is configured in Master/Agent mode.
Pillar is in following directory.

/srv/salt/baseenv/pillar/

States are in following directory.

/srv/salt/baseenv/states/

PILLAR DATA GENERATION
Our rpm list file , a text file, will be in following location.

# mkdir /srv/salt/baseenv/pillar/data
# touch /srv/salt/baseenv/pillar/data/rpm_packages.list

Following are the contents of rpm_packages.list file. Ofcourse you can add your own.

# cat /srv/salt/baseenv/pillar/data/rpm_packages.list
lsof-4.82-4.el6.x86_64
dash-0.5.5.1-4.el6.x86_64
ed-1.1-3.3.el6.x86_64

Create Pillar function.

# mkdir /srv/salt/baseenv/pillar/allrpmpackages
# touch /srv/salt/baseenv/pillar/allrpmpackages/init.sls

Following are the contents of pillar/allrpmpackages/init.sls file.

#!py
#
# This state file will just extract packages name versions and release info and create a salt compatibale data structure 
# created data structure will be : 
# config = { 'pillar_rpm_packages': [ 'package1==ver1.rel1.x86_64', 'package2==ver2.rel2.x86_64' ] }
#

import os, subprocess

def run():
  config = { 'pillar_rpm_packages': []}
  for package in open('/srv/salt/baseenv/pillar/data/rpm_packages.list'):
    package=package.rstrip()
    repo_query_command="repoquery " + package + " --qf '%{NAME}==%{VERSION}-%{RELEASE}'"
    rep_query_process=subprocess.Popen(repo_query_command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    rep_query_process.wait()
    if rep_query_process.returncode == 0:
      formatted_package_name=rep_query_process.stdout.readline().rstrip()
      config['pillar_rpm_packages'].append(formatted_package_name)
  return config

/srv/salt/baseenv/pillar/top.sls will have something similar to following.

baseenv:
  '*':
    - allrpmpackages

STATES and MODULE GENERATION
Create module.

# mkdir -p /srv/salt/baseenv/states/_modules
# touch /srv/salt/baseenv/states/_modules/custmod_rpm_packages_from_list.py

Following are the contents of states/_modules/custmod_rpm_packages_from_list.py

#! /usr/bin/env python

import os, subprocess, re, os.path

packages_states = { }
def create_packages_states(pillar_key):
 for pillar_pack in __pillar__[pillar_key]:
   match_obj=re.search(r'(?P.+)==(?P.+)',pillar_pack)
   pack=match_obj.group('rpm_pack_name')
   pack_version=match_obj.group('rpm_version')
   pack_hash = { pack: pack_version } 
   packages_states[pack] = {'pkg': ['installed', {'version': pack_version }]}
 return packages_states

Now if we want to use above module to Install RPMS, then create a state file with following contents.

# touch /srv/salt/baseenv/states/install_rpm_packages.sls

states/install_rpm_packages.sls file will have following contents.

#!py
config = {  }
def run():
  config = __salt__['custmod_rpm_packages_from_list.create_packages_states']('pillar_rpm_packages')
  return config

baseenv/states/top.sls may have following contents.

baseenv:
  '*':
   - install_rpm_packages

Above code can also be found at https://github.com/spareslant/SaltStackRPMInstallFromListFormula

Non-Interactive, Non-GUI VM build using KVM/libvirtd/kickstart

Introduction:
We will build a VM using KVM/Qemu hypervisor. But we will manage this VM using libvirtd. This VM build will use a kickstart file. This kickstart file will be embedded into VM during install process.

Environment/Tools used:
Host OS is Debian Jessie (Debian-8). We will build CentOS-7.0 VM on this Debian Host. We will use
CentOS-7-x86_64-Minimal-1511.iso image stored in Host machine to build VM. 

libvirtd networking:
Please see following link for details.
http://spareslant.blogspot.co.uk/2016/02/libvirtd-netwroking.html

Activate libvirtd "default" network:

# virsh net-list --all

If above output shows that "default" network is inactive then run following command to activate it.

# virsh net-start default

Activate DHCP in "default" network.

# virsh net-update default add ip-dhcp-range '<range start="192.168.122.100" end="192.168.122.254" />' --live

Create storage file.

# qemu-img create -f qcow2 /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2 5G

Note: I created above storage file because I am using non-standard location. Above step is not required if using libvirtd defaults.

Install Virtual Machine.

# virt-install \
--name centos-vm1 \
--memory 1024 \
--cpu=host \
--vcpus 1 \
--os-type=linux \
--graphics none \
--disk path=/home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2 \
--initrd-inject=/tmp/ks.cfg \
--console pty,target_type=serial \
--extra-args='ks=file:/ks.cfg console=ttyS0,115200n8 serial' \
--network=default \
--location /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso

Note:
1) Please note --graphics none parameter. This is required for non-gui installation.
2) Please note "--initrd-inject=/tmp/ks.cfg" parameter. we are injecting ks.cfg file dynamically. No need of external HTTP/FTP/NFS to host ks.cfg.
3) Above command will start installation of VM non-interactively and non-GUI mode.
4) Above setup uses Virtual Private network.
5) VMs setup in this manner however can access external world. But external world cannot access them.
6) VMs setup in this manner can communicate with each other and with HOST machine as well.
7) In this setup HOST machine will have two IPs. One IP on eth0 (as usual) and other IP on virb0 bridge. Both are in different network. Hence isolating VMs network.
8) If we want HOST machine and VMs to be on same network then BRIDGE networking needs to be used.

Following is the ks.cfg file:

#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
# Use CDROM installation media
cdrom
# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --disable
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='uk'
# System language
lang en_GB.UTF-8

# Network information
#network  --bootproto=static --device=eth0 --gateway=192.168.0.1 --ip=192.168.0.61 --nameserver=8.8.8.8 --netmask=255.255.255.0 --ipv6=auto --activate --hostname=centosvm1
network --bootproto=dhcp --onboot=yes

# Root password
rootpw --iscrypted $6$pDcoINZetTlq2e2S$Tjz7tBv14Mrw41paKN0O57o.7m7HNWOmIguqdLO6YAA1yrxUcl1mypt5bBKqjVuOqnlNOOoeQH9zJud6FfXcz1
# Do not configure the X Window System
skipx
# System timezone
timezone Europe/London --isUtc
user --name=ocean1 --password=$6$wTphTlXg/5nlzaNK$YoezS.sO80koCnVgyC.kOxF.t3jo0dzk9ey6ENiAPpWme9dfKTFX7ziC.oONjtAh1hDnlLLLq1j4N5YWUlcrK0 --iscrypted
# System bootloader configuration
bootloader --append=" crashkernel=auto console=ttyAMA0,115200 console=tty console=ttyS0" --location=mbr
autopart --type=plain
# Partition clearing information
clearpart --all --initlabel

%packages
@core
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%post --interpreter /usr/bin/python --log=/root/post-log
import os
entirefile=open('/etc/default/grub').read()
entirefile=entirefile.replace('rhgb', '')
entirefile=entirefile.replace('quiet', '')
open('/etc/default/grub', 'w').write(entirefile)
os.system("grub2-mkconfig -o /boot/grub2/grub.cfg")

%end

Note: Please note "text" parameter used in ks.cfg file. This is for non-GUI text installation.

Build Virtual Machine Using KVM/Qemu and Kickstart

Introduction:
We will build a VM using KVM/Qemu hypervisor. This VM build will use a kickstart file. This kickstart file will be served from a web server. Instead of building a full blown web server like apache or nginx, we will build a simple HTTP server using Golang, that will serve our kickstart file.

Environment/Tools used:
Host OS is Debian Jessie (Debian-8). We will build CentOS-7.0 VM on this Debian Host. We will use
CentOS-7-x86_64-Minimal-1511.iso image stored in Host machine to build VM. Make sure Go compiler is installed.

Create Web Server:

mkdir GO_HTTP_SERVER

Create following two files inside GO_HTTP_SERVER directory

$ ls
ks.cfg serve_ks.go

serve_ks.go file:

package main
import (
  "net/http"
)

func main() {
  http.ListenAndServe(":8080", http.FileServer(http.Dir("./")))
}

ks.cfg file

#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
# Use CDROM installation media
cdrom
# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --disable
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='uk'
# System language
lang en_GB.UTF-8

# Network information
#network  --bootproto=static --device=eth0 --gateway=192.168.0.1 --ip=192.168.0.61 --nameserver=8.8.8.8 --netmask=255.255.255.0 --ipv6=auto --activate --hostname=centosvm1

# Root password
rootpw --iscrypted $6$pDcoINZetTlq2e2S$Tjz7tBv14Mrw41paKN0O57o.7m7HNWOmIguqdLO6YAA1yrxUcl1mypt5bBKqjVuOqnlNOOoeQH9zJud6FfXcz1
# Do not configure the X Window System
skipx
# System timezone
timezone Europe/London --isUtc
user --name=ocean1 --password=$6$wTphTlXg/5nlzaNK$YoezS.sO80koCnVgyC.kOxF.t3jo0dzk9ey6ENiAPpWme9dfKTFX7ziC.oONjtAh1hDnlLLLq1j4N5YWUlcrK0 --iscrypted
# System bootloader configuration
bootloader --append=" crashkernel=auto console=ttyAMA0,115200 console=tty console=ttyS0" --location=mbr
autopart --type=plain
# Partition clearing information
clearpart --all --initlabel

%packages
@core
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%post --interpreter /usr/bin/python --log=/root/post-log
import os
entirefile=open('/etc/default/grub').read()
entirefile=entirefile.replace('rhgb', '')
entirefile=entirefile.replace('quiet', '')
open('/etc/default/grub', 'w').write(entirefile)
os.system("grub2-mkconfig -o /boot/grub2/grub.cfg")

%end

Note:
1) CentOS-7.0 minimal ISO image does not have perl packages. Therefore I have to use python for post installation task in ks.cfg file.
2) Change your encrypted password.
3) Post installation section in ks.cfg will disable "quiet" boot. That means all boot information will be displayed on console.
4) ks.cfg has kernel boot parameters so that console can be redirected.
5) It uses default network configuration provided by QEMU/KVM. (using default DHCP embedded in qemu).

Run HTTP server

$ cd GO_HTTP_SERVER
$ go run serve_ks.go

verify that webserver is running. Run following command on another terminal.
$ curl http://IPADDRESS_OF_HOSTMACHINE:8080/ks.cfg

You should see output.

Note:
1) Web server is started on port 8080
2) Use machine IP address not localhost or 127.0.0.1 because , VM will communicate with host machine for installation and it cannot communicate with host machine on 127.0.0.1.
3) When you start webserver using "go run serve_ks.go" command, it prints nothing on screen. You can stop web server just by pressing Ctrl-c.

Build VM
Create Storage File

# qemu-img create -f qcow2 centos-7.qcow2 5G

Start VM in user networking mode

# qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso -hda /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2  -boot d

A new window will pop up with boot options. Press TAB. Some options will be visible. Append following to those options.

ks=http://IPADDRESS_OF_HOSTMACHINE:8080/ks.cfg

Once system installation is complete, run VM using following command.
Please note the omission of "-boot d".

# screen -S centos qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso -hda /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2

Update kernel boot parameters so that console can be redirected to terminal where qemu command is fired. then run following command. (note -nographic) (attached ks.cfg has this functionality). So you do not need to manually update kernel boot parameters.

# screen -S centos qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso -hda /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2  -nographic

Note: Please note that We started VM in a screen session so that VM console can be attached or detached easily.
We do not need to do this when VMs are build using libvirtd. libvirtd provides many functionalities .

Java and Ruby Inheritance Comparison

Following is a Ruby Program. This is heavily commented with reference to Java and Ruby both.

class A
  def methoda_public
    puts "Hello from methoda_public self=#{self}"
  end

  def methoda_private
    puts "Hello from methoda_private self=#{self}"
  end

  def methoda_protected
    puts "Hello from methoda_protected self=#{self}"
  end

    # 1) Java => class methods are defined using static keyword. instances can access class methods in Java.
    # 2) Ruby => class methods are actually singleton methods of class itself. (in Ruby a class is an object itself.). class methods are actually defined in
               # singleton class of "class object" and hence called as singleton methods. Only that object can call singleton method on which it is defined and none other.
               # Singleton methods are Exclusive to a single particular object only. Hence even the instances created from class A cannot access class methods.
               # derived class can invoke base class' singleton object. Again instances of Derived class cannot access singleton method defined on base class.
  def self.methoda_classmethod_public
    puts "hello from methoda_classmethod_public self=#{self}"
  end

  private :methoda_private
  protected :methoda_protected
end

class B < A
  def public_methodb_calling_private_methoda

      # 1) Java => private members are not inherited directly in derived class. So this will not work in Java.
                 # Derived class inherits base class' protected and public members. So if Base class' protected/public method access private member
                 # then Derived class can also 'work' on private member via inherited protected/public methods.
      # 2) Ruby => private members are inherited in Ruby. So this will work in ruby.
    methoda_private       

      # 1) Java => you cannot access private memebers directly in derived class. So this will not work in Java.
      # 2) Ruby => Will not work as private members must be accessed WITHOUT explicit receiver in derived class.
    #A.new.methoda_private  # A new object is created and we are trying to access its private member.  will not work.
    #self.methoda_private   # An equivalent in Java would be this.methoda_private(). But ofcourse in Java private members are not inherited so will not work.
      
      # 1) Java => Protected members can be accessed in derived class directly becuase they are inherited. So this will work.
      # 2) Ruby => Protected members are inherited in Ruby as well. So this will work.
    methoda_protected

      # 1) Java => protected memeber cannot be accessed in drived class using reference variable(i.e. by creating object).
      # 2) Ruby => This will work. But this will NOT work in Ruby also if below expression i.e. A.new.methoda_protected is called outside class B. 
    A.new.methoda_protected

      # 1) Java => A Java equivalent would be this.methoda_protected() . This will work in Java. As protected methods are inherited in Java directly. Needed verification.
      # 2) Ruby => This will work. Ruby will be calling inherited method. Java will also do the same.
    self.methoda_protected

      # 1) Java => Instance method can call static method (class method).
      # 2) Ruby => This will not work. Instance method CANNOT call class method.
    #methoda_classmethod_public

      # 1) Java => instance method can call static method (class method). In java it would be this.methoda_classmethod_public()
      # 2) Ruby => this will not work. Because inside instance method self will be referring to the instance of B. And instance of B CANNOT call
                 # singleton method of class A.
    #self.methoda_classmethod_public
  end

    # 1) Java => Both below statements are not allowed in Java. These are called outside any method definition.
    # 2) Ruby => Both below will work in Ruby. Hence class methods are inherited but only to be called by derived class itself.
               # class methods are singleton method. Normally singleton methods are called by only their own onjects.
               # But when in case of singleton methods defined on class (class methods), they can be called by dervied class also. (Not by derived class instances).
  methoda_classmethod_public
  self.methoda_classmethod_public
end

#A.new.methoda_private     # => will not work. (same as in Java)
#A.new.methoda_protected   # => will not work. (same as in Java). Compare and take a note of this expression inside class B as well.
#A.new.methoda_classmethod_public # => Will not work in Ruby but will work in Java. Java syntax would be: A a = new A(); a.methoda_classmethod_public() .

B.new.public_methodb_calling_private_methoda
B.methoda_classmethod_public  # => will work both in Java and Ruby

A Simple Vagrant Setup

Preparation:

mkdir MCOLLECTIVE_VAGRANT
cd MCOLLECTIVE_VAGRANT
vagrant init
vagrant box add puppetlabs/centos-6.6-64-nocm

modify contents of MCOLLECTIVE_VAGRANT/Vagrant file to look like below:

Single Instance Vagrantfile

$ cat Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.hostname = "vagrant.virtual.machine"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
end

Multi Instance Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
     config.vm.define "first" do |first|
          config.vm.hostname = "first.virtual.machine"
     end
     config.vm.define "second" do |second|
          config.vm.hostname = "second.virtual.machine"
     end
end

To start Vagrant instances.

vagrant up

To login into new VM machine.

vagrant ssh

To check status

vagrant global-status

Config with autostart: false and Mix of CentOS and Debian.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
     config.vm.define "first", autostart: false do |first|
          config.vm.hostname = "first.virtual.machine"
     end
     config.vm.define "second", autostart: false do |second|
          config.vm.hostname = "second.virtual.machine"
     end
     config.vm.define "debian-7.4", autostart: false do |second|
          config.vm.box = "chef/debian-7.4"
          config.vm.hostname = "debian.virtual.machine"
     end
end

Bring up only Debian machine

vagrant up debian-7.4
vagrant ssh debian-7.4

Bring up now first machine

vagrant up first
vagrant ssh first

Note: When autostart: false is configured , Vagrant machines have to be brought up manually

Understanding JAVA Nested Classes

class OuterClass {
    private String outerClassPrivateInstanceVar = "outerClassPrivateInstanceVar";
    public static String outerClassPublicStaticVar = "outerClassPublicStaticVar";
    public String outerClassPublicInstanceVar = "outerClassPublicInstanceVar";

    void outerClassInstanceMethod() {
        System.out.println("Inside outerClassInstanceMethod.");
    }

    static void outerClassStaticMethod() {
        System.out.println("Inside outerClassStaticMethod.");
    }

    // NON-STATIC Inner class
    class InnerClass {
        // Non-static innerClass cannot have static members. Following will not work.
        // static String InnerClassStaticVar = "InnerClassStaticVar";
        public void innerClassMethod() {
            System.out.println("Inside innerClassMethod.");
        }
        public void innerClassAccessOuterClassVars() {
            System.out.println(outerClassPrivateInstanceVar);
            System.out.println(outerClassPublicInstanceVar);
            System.out.println(outerClassPublicStaticVar);
        }
    }

    // STATIC Inner class
    static class InnerClassStatic {
        // Following is allowed in Static InnerClass but not in non-static inner class.
        static String StaticInnerClassStaticVar = "StaticInnerClassStaticVar";
        String InnerClassStaticInstanceVar = "InnerClassStaticInstanceVar";
        public void innerClassStaticMethod() {
            System.out.println("Inside InnerClassStatic.");
        }
        public void innerClassStaticAccessOuterClassVars() {
            // Following will not work. Beacuse InnerClassStatic is like static member inside OuterClass
            // and static members/methods can never access instance members.
            //System.out.println(outerClassPrivateInstanceVar);
            //System.out.println(outerClassPublicInstanceVar);

            System.out.println(outerClassPublicStaticVar);
        }
    }

    // Local Inner Class inside Instance method.
    void outerClassInstanceMethodForLocalClass() {
        // Local Inner Classes cannot be private, protected, public and static. Because inside method
        // these keywords have no sense. We don't use these words inside methods.
        class LocalClassInInstanceMethod {
            // Static members are not allowed in Local Inner classes defined inside instance method.
            // Following will not work.
            //static String LocalClassInInstanceMethodStaticVar = "LocalClassInInstanceMethodStaticVar";
            void localClassInInstanceMethod_instanceMethod() {
                System.out.println("Inside localClassInInstanceMethod_instanceMethod.");
            }
        }
        LocalClassInInstanceMethod localClassInInstanceMethodObj = new LocalClassInInstanceMethod();
        localClassInInstanceMethodObj.localClassInInstanceMethod_instanceMethod();
    }

    // Local Inner Class inside Static method.
    static void outerClassStaticMethodForLocalClass() {
        // Local Inner Classes cannot be private, protected, public and static. Because inside method
        // these keywords have no sense. We don't use these words inside methods.
        class LocalClassinStaticMethod {
            // Static members are not allowed in Local Inner classes defined inside static  method.
            // Following will not work.
            //static String LocalClassinStaticMethodStaticVar = "LocalClassinStaticMethodStaticVar";
            void LocalClassinStaticMethod_instanceMethod() {
                System.out.println("Inside LocalClassinStaticMethod_instanceMethod.");
            }
        }
        LocalClassinStaticMethod locallassinStaticMethodObj = new LocalClassinStaticMethod();
        locallassinStaticMethodObj.LocalClassinStaticMethod_instanceMethod();
    }

    // Anonymous Inner classes
    interface InnerInterface {
        void InnerInterfaceMethod();
    }
    // Anonymous inner class object defined at class member level.
    InnerInterface anonymousClassObj = new InnerInterface() {
        public void InnerInterfaceMethod() {
            System.out.println("Inside InnerInterfaceMethod.");
        }
    };
    void callAnonymousObjectMethod() {
        anonymousClassObj.InnerInterfaceMethod();
    }
    // Anonymous inner class inside instance method.
    //void anonymousInnerClassInsideMethod() {
    OuterClass  anonymousInnerClassInsideMethod() {
        OuterClass anon = new OuterClass() {
            void outerClassInstanceMethod() {
                System.out.println("Inside anonymousInnerClassInsideMethod.");
            }
        };
        //anon.outerClassInstanceMethod();
        return anon;
    }
}

public class InnerClassesTest {
    public static void main(String[] args) {
        OuterClass outerClassObj = new OuterClass();

        // Call to instance method using instance variable.
        outerClassObj.outerClassInstanceMethod();

        // Call to static method using instance variable. In Ruby there is something called self instead of static. And
        // self.outerClassStaticMethod  in Ruby is actually a singleton method defined on a "class object (OuterClass)" and in Ruby class
        // itself is an object. And singleton methods can be called by only object on which it is defined, in this case OuterClass object.
        // Hence in Ruby object created by a class cannot access class method. Following is invalid in Ruby.
        outerClassObj.outerClassStaticMethod();

        // NON-STATIC INNER CLASSES
        // Creating InnerClass object.
        System.out.println("====NON-STATIC INNER CLASSES====");
        OuterClass.InnerClass innerClassObj = new OuterClass().new InnerClass();
        innerClassObj.innerClassMethod();

        // Following will not work. outerClassPublicInstanceVar is not defined in InnerClass.
        //System.out.println(innerClassObj.outerClassPublicInstanceVar);

        innerClassObj.innerClassAccessOuterClassVars();

        // STATIC INNER CLASSES.
        // Creating Static Innerclass object. You will be using Class not object.
        System.out.println("====STATIC INNER CLASSES====");
        OuterClass.InnerClassStatic innerClassStaticObj = new OuterClass.InnerClassStatic();
        // You cannot use OuterClass object to use StaticInnerClass object. This is in contrast with creating InnerClass(non-static) object.
        // Also normally an object can access static class fields. But Following will not work.
        //OuterClass.InnerClassStatic innerClassStaticObj2 = outerClassObj.InnerClassStatic();

        // Access static inner class' static member using static inner class' object.
        System.out.println(innerClassStaticObj.StaticInnerClassStaticVar);
        // Access static inner class' static member using Classes directly.
        System.out.println(OuterClass.InnerClassStatic.StaticInnerClassStaticVar);
        // Access static inner class' instance member using static inner class' object.
        System.out.println(innerClassStaticObj.InnerClassStaticInstanceVar);

        // LOCAL INNER CLASSES
        System.out.println("====LOCAL INNER CLASSES====");
        outerClassObj.outerClassInstanceMethodForLocalClass();
        outerClassObj.outerClassStaticMethodForLocalClass();
        OuterClass.outerClassStaticMethodForLocalClass();

        // ANONYMOUS INNER CLASSES
        System.out.println("====ANONYMOUS INNER CLASSES====");
        outerClassObj.callAnonymousObjectMethod();
        // Following works because, interface InnerInterface is visible in this class as well due to default access.
        // If inner interface in made private then following will not work. Only inner interfaces can be made private.
        outerClassObj.anonymousClassObj.InnerInterfaceMethod();

        OuterClass anonymousObj = new OuterClass() {
            void outerClassInstanceMethod() {
            System.out.println("inside anonymousObjOfOuterClassSubClass Method.");
            }
            void anotherMethod() {
                System.out.println("Inside anotherMethod.");
            }
        };
        anonymousObj.outerClassInstanceMethod();
        // Following will not work. Because anonymousObj is refering to actually subclass of OuterClass (in memory). But it is declared as OuterClass.
        // That means, anonymousObj is actually a "Subclass of OuterClass" but its type is declared as OuterClass. Hence polymorphism will come in.
        // anonymousObj being declared as superclass variable will only be able to execute methods defined in super class not in sub class. Hence
        // following will not work.
        //anonymousObj.anotherMethod();

        //Following works provided relevant lines are un-commented inside anonymousInnerClassInsideMethod instance method.
        //outerClassObj.anonymousInnerClassInsideMethod();
        OuterClass out = outerClassObj.anonymousInnerClassInsideMethod();
        out.outerClassInstanceMethod();
    }
}

Compile and Output:

$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
$
$ javac InnerClassesTest.java
$ java InnerClassesTest
Inside outerClassInstanceMethod.
Inside outerClassStaticMethod.
====NON-STATIC INNER CLASSES====
Inside innerClassMethod.
outerClassPrivateInstanceVar
outerClassPublicInstanceVar
outerClassPublicStaticVar
====STATIC INNER CLASSES====
StaticInnerClassStaticVar
StaticInnerClassStaticVar
InnerClassStaticInstanceVar
====LOCAL INNER CLASSES====
Inside localClassInInstanceMethod_instanceMethod.
Inside LocalClassinStaticMethod_instanceMethod.
Inside LocalClassinStaticMethod_instanceMethod.
====ANONYMOUS INNER CLASSES====
Inside InnerInterfaceMethod.
Inside InnerInterfaceMethod.
inside anonymousObjOfOuterClassSubClass Method.
Inside anonymousInnerClassInsideMethod.

libvirtd Netwroking

If libvirtd is installed then it comes with a default network called "default".

# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes

If this network is not active you can make it active using following command. The moment you make it active, it places various IPTABLES rules to do NAT forwarding between host and guest.

# virsh net-start default
Network default started

When "default" network is brought up, a bridge is created called virbr0. This bridge is also assigned an IP Address automatically.

# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 34:17:eb:d5:8d:fe brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether fe:e6:45:38:00:ff brd ff:ff:ff:ff:ff:ff

# ip add show virbr0
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether fe:e6:45:38:00:ff brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever

IPTABLES will now be in place.

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc

You can destroy this "default" network by using following command. As a result of this , iptables will vanish, bridge virbr0 will disappear and "default" network will become inactive.

# virsh net-destroy defaultNetwork default destroyed

# iptables -LChain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# ip ad virbr0
Command "virbr0" is unknown, try "ip addr help".

# brctl show
bridge name bridge id STP enabled interfaces

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 34:17:eb:d5:8d:fe brd ff:ff:ff:ff:ff:ff

# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes 

Activate DHCP in "default" network.

# virsh net-list --all

# virsh net-start default 

# virsh net-update default add ip-dhcp-range '<range start="192.168.122.100" end="192.168.122.254" />' --live

Firefox in Docker

Host OS : Fedora 22
Docker Version: 1.7.0-1

steps:

$ mkdir firefox

$ cd firefox; touch Dockerfile

$ ( cat <<-EOF
FROM fedora
MAINTAINER Spare Slant "spareslant@gmail.com"
ENV REFRESHED_AT 2015-11-21
RUN dnf -y install firefox
RUN dnf -y install dejavu-sans-fonts dejavu-serif-fonts
RUN useradd --shell /bin/bash --uid 1000 -m testuser
USER testuser
ENV HOME /home/testuser
CMD ["/usr/bin/firefox", "--no-remote"]
EOF
) > Dockerfile

$ docker build -t="spareslant/firefox:v2"

To run above docker :

docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -v /etc/machine-id:/etc/machine-id -e DISPLAY=$DISPLAY --name firefox spareslant/firefox:v2

A firefox window will pop up. If you close firefox , docker process will also shutdown. To start it again run following: docker start firefox

Find repeated Group pattern match. Case study of Ruby, Python and Go

Find repeated Group pattern match. Case study of Ruby, Python and Go

Sometimes I have to find the a group of patterns that are occurring in a log file. Normal grep -f <filename> does that job but it does not print the delimiter that can distinguish between repeated found patterns.

I needed something that can tell me that it has found a few patterns from a group in a sequence and last pattern has been found. Draw a delimiter here and start search for group again in remaining file.

I first wrote it in Ruby. I had huge files to parse. Ruby script was talking quite long to finish. As a case study I decided to write same utility using Python and Go.

Needless to say , Go was much faster.

Following are the codes.

Following are the sample data files.

pattern file :

hotmail
yahoo
google
gmail

Subject file:

This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Another one is gmail
Now is google
Another one is gmail
But this is going to be a yahoo
This is hotmail test
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR

Following is the output:

$ go run atest.go pattern subjectfile
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========
This is hotmail test
But this is going to be a yahoo
Another one is gmail
===========
Now is google
Another one is gmail
===========
But this is going to be a yahoo
This is hotmail test
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========

Launch AWS EC2 nstance using SaltStack

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli
pip install M2Crypto
pip install pyzmq

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Become root on MacBook now:

mkdir /etc/salt
touch /etc/salt/cloud.profiles
touch /etc/salt/cloud.providers

cat /etc/salt/cloud.profiles
base_ec2_private:
     provider: amazon_ireland_region
     image: ami-9d23aeea

cat /etc/salt/cloud.providers
amazon_ireland_region:
     id: ABCDEFGHIJKLMNO235M
     key: aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv
     keyname: MyEC2Key
     private_key: /Users/MacUser/EC2/MyEC2Key.pem
     location: eu-west-1
     availability_zone: eu-west-1a
     securitygroup: MySecurityGroupSSHOnly
     size: t2.micro
     del_root_vol_on_destroy: True
     ssh_username: ec2-user
     rename_on_destroy: True
     ssh_interface: public_ips
     provider: ec2

Launch Instance now:

source ~MacUser/PythonVirtENVs/SaltStack/bin/activate
salt-cloud --profile=base_ec2_private First_Instance

Launch AWS EC2 instance using awscli

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Following command will spin a FREE TIER instance in AWS cloud

aws ec2 run-instances --image-id ami-9d23aeea --key-name MyEC2Key --instance-type t2.micro --count 1 --security-groups MySecurityGroupSSHOnly

Playing and Encoding Video files on Linux

DVD tracks and Chapters info

mplayer -identify dvd:// -dvd-device /media/DVD_FOLDER/

MPlayer 29092-4.4.0 (C) 2000-2009 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing dvd://.
libdvdread: Encrypted DVD support unavailable.
libdvdread: Couldn't find device name.
ID_DVD_TITLES=2
ID_DVD_TITLE_1_CHAPTERS=51
ID_DVD_TITLE_1_ANGLES=1
ID_DVD_TITLE_2_CHAPTERS=50
ID_DVD_TITLE_2_ANGLES=1
ID_DVD_TITLE_1_LENGTH=13653.600
ID_DVD_TITLE_2_LENGTH=13150.166

PLAYING OPTIONS:
If the DVD video is downloaded on the harddisk, following are the various options to run videos from that DVD folder.

1) To open the DVD folder as a DVD. It will present all the menus of DVD. (note the dvdnav option)

mplayer -mouse-movements dvdnav:// -dvd-device /media/DVD_FOLDER/

2) To run the only chapter 2 from the DVD

mplayer dvd:// -chapter 2-2 -dvd-device /media/DVD_FOLDER/

3) To run the only chapter 5 to 7 from the DVD

mplayer dvd:// -chapter 5-7 -dvd-device /media/DVD_FOLDER/

4) To run 1st track of DVD

mplayer dvd://1 -dvd-device /media/DVD_FOLDER/

5) To view only the specific portion of a video file (based on time slice)

mplayer -ss 00:00:30 -endpos 30 dvdnav://1 -dvd-device /media/DVD_FOLDER/

CONVERSION OPTIONS:

mencoder dvd:// -chapter 2-2 -vf scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af-add lavcresample=44100 -vf-add harddup -vf-add scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/


mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -vf scale=320:240 -o yahoo.mp4 -mc 0 -oac mp3lame -lameopts vbr=3:br=128:q=9 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25:vpass=1 -of lavf -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -ovc frameno -oac mp3lame -lameopts vbr=0:br=96 -o frame.mp3 -of lavf -lavfopts format=mp3 -dvd-device /media/DVD_FOLDER/

Extracting RAW audio AUDIO from a Video File (in PCM format i.e. wav)

mplayer /media/Video.avi -vo null -vc null -ao pcm:fast:file=/tmp/audio.wav

Conversion using ffmpeg

/opt/ffmpeg/bin/ffmpeg -i /media/video.avi -i /tmp/audio.mp3 -map 0:0 -map 1:0 -vcodec msmpeg4v2 -s qvga -b 512k -r 30 -acodec copy /tmp/last.mp4

Cutting a scene from a video file

In mplayer , when file is being played, press "o" to see the current position in (time format).
use this information to know how long scene needs to be cut.


mencoder -ss 00:07:20 -endpos 190 -o /tmp/blue.avi -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 input_vdo_file.avi

Extracting DVD information using HandBrake

./HandBrakeCLI --scan  -i /Volumes/VDO_FOLDER

Extracting Chapter From a DVD dump on a disk using HandBrake

./HandBrakeCLI -i /Volumes/DVD_FOLDER/  -e x264  -q 20.0 -a 1,1 -E faac,copy:ac3 -B 320,160 -6 dpl2,none -R Auto,Auto -D 0.0,0.0 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback ffac3 -f mp4 -4 --decomb --loose-anamorphic --modulus 2 -m --x264-preset medium --h264-profile high --h264-level 4.1 -c 1 -o $HOME/chap1mod.mp4

Note:  “-c 1” in above command is extracting 1st chapter from DVD.

Python Regex Usage

Python Regex Usage

Perl Regex Usages

Multiline Matchings:

Inside a perl script:

#! /usr/bin/perl
$str = "This is a test
Will it work
lets see
";
if ($str =~ m!^This.+test$!)
{
     print "matched\n"
}

Following are the contents of file "data"

# cat data
This is a test file
Will it work
lets see

# As perl one liners:

$ cat data | perl -w -e '$/=""; while(<>) { m!^This.+file$!m and print }'
This is a test file
Will it work
lets see


# DEFAULT BEHAVIOUR
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH

# s modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ 


# m modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^let.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
lets see

# sm modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^lets.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ 

$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+it).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it


# BACKREFRENCES:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\1! and print "MATCHED\n"}'
MATCHED
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\g{-1}! and print "MATCHED\n"}'
MATCHED

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED

# CAPTURING and NON CAPTURING GROUPING:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "$1\n"}'
vis  

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?:vis).+\g{testmatch}! and print "$1\n"}'
Reference to nonexistent named group in regex; marked by <-- here="" in="" m="" testmatch="">vis).+\g{testmatch <-- -e="" -w="" 1.="" a="" at="" echo="" here="" line="" perl="" vis="" while="">) { m!(?:vis).+! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+! and print "$1\n"}'
vis
$

$ echo "datchet etet it underway" | perl -w -e '$/=""; while(<>) { m!(et){2}\s+! and print "MATCHED\n"}'
MATCHED
 
# NAMED CAPTURING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?test file)! and print "$+{fm}\n" or print "NOMATCH\n" }'
test file
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "$1" or print "NOMATCH\n" }'
Use of uninitialized value $1 in string at -e line 1, <> chunk 1.
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED 


# inside modifiers:
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+test! and print "MATCHED\n" or print "UNMATCHED\n"}'
UNMATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+TEST! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ 

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+((?i)test)! and print "$1\n"}'
TEST
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED

# inside modifiers (multiline modifier trick):
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s)(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!((?s)^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work

# inside modifiers non-capturing match:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED


# LOOK AHEAD ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(.+lets)(.+it.+$)!sm and print "$1 $3\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
NOMATCH


# LOOK BEHIND  ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(it)!sm and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(.+it)!sm and print "$1\n" or print "NOMATCH\n" }'
test file
Will it

# CONDITIONAL MATCHING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(asa)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'        # This is not a conditional matching
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "$1 $2\n" or print "NOMATCH\n" }'
Use of uninitialized value $2 in concatenation (.) or string at -e line 1, <> chunk 1.
test file 

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(anotherjunk))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH
 
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=junk)junk)!sm and print "MATCH\n" or print "NOMATCH\n" }'  # if condition is false then it simply skips
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk))!sm and print "MATCH\n" or print "NOMATCH\n" }'  # why is this working ?  no idea
MATCH

(?(?=condition)(then1|then2|then3)|(else1|else2|else3))

# Some Special cases to study:
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(ab)*! ; print "@arr\n";}'
ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((ab)*)! ; print "@arr\n";}'
ab ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)*)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)! ; print "@arr\n";}'
aba
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(?:a|b)*!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]*!g ; print "@arr\n";}'
aba  a 
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]*)!g ; print "@arr\n";}'
aba  a  
$  echo "abacaddb" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a   b  
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([ab]+)! ; print "$1\n";}'
aba
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([abc]+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((a|b|c)+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((?:a|b|c)+)! ; print "$1\n";}'
abaca


#======================================================

$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(a|b)+! ; print "@arr\n";}'
a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)+)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]+! ; print "@arr\n";}'
1
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]+)! ; print "@arr\n";}'
aba