Non-Interactive, Non-GUI VM build using KVM/libvirtd/kickstart

Introduction:
We will build a VM using KVM/Qemu hypervisor. But we will manage this VM using libvirtd. This VM build will use a kickstart file. This kickstart file will be embedded into VM during install process.

Environment/Tools used:
Host OS is Debian Jessie (Debian-8). We will build CentOS-7.0 VM on this Debian Host. We will use
CentOS-7-x86_64-Minimal-1511.iso image stored in Host machine to build VM. 

libvirtd networking:
Please see following link for details.
http://spareslant.blogspot.co.uk/2016/02/libvirtd-netwroking.html

Activate libvirtd "default" network:

# virsh net-list --all

If above output shows that "default" network is inactive then run following command to activate it.

# virsh net-start default

Activate DHCP in "default" network.

# virsh net-update default add ip-dhcp-range '<range start="192.168.122.100" end="192.168.122.254" />' --live

Create storage file.

# qemu-img create -f qcow2 /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2 5G

Note: I created above storage file because I am using non-standard location. Above step is not required if using libvirtd defaults.

Install Virtual Machine.

# virt-install \
--name centos-vm1 \
--memory 1024 \
--cpu=host \
--vcpus 1 \
--os-type=linux \
--graphics none \
--disk path=/home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2 \
--initrd-inject=/tmp/ks.cfg \
--console pty,target_type=serial \
--extra-args='ks=file:/ks.cfg console=ttyS0,115200n8 serial' \
--network=default \
--location /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso

Note:
1) Please note --graphics none parameter. This is required for non-gui installation.
2) Please note "--initrd-inject=/tmp/ks.cfg" parameter. we are injecting ks.cfg file dynamically. No need of external HTTP/FTP/NFS to host ks.cfg.
3) Above command will start installation of VM non-interactively and non-GUI mode.
4) Above setup uses Virtual Private network.
5) VMs setup in this manner however can access external world. But external world cannot access them.
6) VMs setup in this manner can communicate with each other and with HOST machine as well.
7) In this setup HOST machine will have two IPs. One IP on eth0 (as usual) and other IP on virb0 bridge. Both are in different network. Hence isolating VMs network.
8) If we want HOST machine and VMs to be on same network then BRIDGE networking needs to be used.

Following is the ks.cfg file:

#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
# Use CDROM installation media
cdrom
# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --disable
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='uk'
# System language
lang en_GB.UTF-8

# Network information
#network  --bootproto=static --device=eth0 --gateway=192.168.0.1 --ip=192.168.0.61 --nameserver=8.8.8.8 --netmask=255.255.255.0 --ipv6=auto --activate --hostname=centosvm1
network --bootproto=dhcp --onboot=yes

# Root password
rootpw --iscrypted $6$pDcoINZetTlq2e2S$Tjz7tBv14Mrw41paKN0O57o.7m7HNWOmIguqdLO6YAA1yrxUcl1mypt5bBKqjVuOqnlNOOoeQH9zJud6FfXcz1
# Do not configure the X Window System
skipx
# System timezone
timezone Europe/London --isUtc
user --name=ocean1 --password=$6$wTphTlXg/5nlzaNK$YoezS.sO80koCnVgyC.kOxF.t3jo0dzk9ey6ENiAPpWme9dfKTFX7ziC.oONjtAh1hDnlLLLq1j4N5YWUlcrK0 --iscrypted
# System bootloader configuration
bootloader --append=" crashkernel=auto console=ttyAMA0,115200 console=tty console=ttyS0" --location=mbr
autopart --type=plain
# Partition clearing information
clearpart --all --initlabel

%packages
@core
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%post --interpreter /usr/bin/python --log=/root/post-log
import os
entirefile=open('/etc/default/grub').read()
entirefile=entirefile.replace('rhgb', '')
entirefile=entirefile.replace('quiet', '')
open('/etc/default/grub', 'w').write(entirefile)
os.system("grub2-mkconfig -o /boot/grub2/grub.cfg")

%end

Note: Please note "text" parameter used in ks.cfg file. This is for non-GUI text installation.

Build Virtual Machine Using KVM/Qemu and Kickstart

Introduction:
We will build a VM using KVM/Qemu hypervisor. This VM build will use a kickstart file. This kickstart file will be served from a web server. Instead of building a full blown web server like apache or nginx, we will build a simple HTTP server using Golang, that will serve our kickstart file.

Environment/Tools used:
Host OS is Debian Jessie (Debian-8). We will build CentOS-7.0 VM on this Debian Host. We will use
CentOS-7-x86_64-Minimal-1511.iso image stored in Host machine to build VM. Make sure Go compiler is installed.

Create Web Server:

mkdir GO_HTTP_SERVER

Create following two files inside GO_HTTP_SERVER directory

$ ls
ks.cfg serve_ks.go

serve_ks.go file:

package main
import (
  "net/http"
)

func main() {
  http.ListenAndServe(":8080", http.FileServer(http.Dir("./")))
}

ks.cfg file

#version=DEVEL
# System authorization information
auth --enableshadow --passalgo=sha512
# Use CDROM installation media
cdrom
# Use text mode install
text
# Run the Setup Agent on first boot
firstboot --disable
# Keyboard layouts
keyboard --vckeymap=uk --xlayouts='uk'
# System language
lang en_GB.UTF-8

# Network information
#network  --bootproto=static --device=eth0 --gateway=192.168.0.1 --ip=192.168.0.61 --nameserver=8.8.8.8 --netmask=255.255.255.0 --ipv6=auto --activate --hostname=centosvm1

# Root password
rootpw --iscrypted $6$pDcoINZetTlq2e2S$Tjz7tBv14Mrw41paKN0O57o.7m7HNWOmIguqdLO6YAA1yrxUcl1mypt5bBKqjVuOqnlNOOoeQH9zJud6FfXcz1
# Do not configure the X Window System
skipx
# System timezone
timezone Europe/London --isUtc
user --name=ocean1 --password=$6$wTphTlXg/5nlzaNK$YoezS.sO80koCnVgyC.kOxF.t3jo0dzk9ey6ENiAPpWme9dfKTFX7ziC.oONjtAh1hDnlLLLq1j4N5YWUlcrK0 --iscrypted
# System bootloader configuration
bootloader --append=" crashkernel=auto console=ttyAMA0,115200 console=tty console=ttyS0" --location=mbr
autopart --type=plain
# Partition clearing information
clearpart --all --initlabel

%packages
@core
kexec-tools

%end

%addon com_redhat_kdump --enable --reserve-mb='auto'

%end

%post --interpreter /usr/bin/python --log=/root/post-log
import os
entirefile=open('/etc/default/grub').read()
entirefile=entirefile.replace('rhgb', '')
entirefile=entirefile.replace('quiet', '')
open('/etc/default/grub', 'w').write(entirefile)
os.system("grub2-mkconfig -o /boot/grub2/grub.cfg")

%end

Note:
1) CentOS-7.0 minimal ISO image does not have perl packages. Therefore I have to use python for post installation task in ks.cfg file.
2) Change your encrypted password.
3) Post installation section in ks.cfg will disable "quiet" boot. That means all boot information will be displayed on console.
4) ks.cfg has kernel boot parameters so that console can be redirected.
5) It uses default network configuration provided by QEMU/KVM. (using default DHCP embedded in qemu).

Run HTTP server

$ cd GO_HTTP_SERVER
$ go run serve_ks.go

verify that webserver is running. Run following command on another terminal.
$ curl http://IPADDRESS_OF_HOSTMACHINE:8080/ks.cfg

You should see output.

Note:
1) Web server is started on port 8080
2) Use machine IP address not localhost or 127.0.0.1 because , VM will communicate with host machine for installation and it cannot communicate with host machine on 127.0.0.1.
3) When you start webserver using "go run serve_ks.go" command, it prints nothing on screen. You can stop web server just by pressing Ctrl-c.

Build VM
Create Storage File

# qemu-img create -f qcow2 centos-7.qcow2 5G

Start VM in user networking mode

# qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso -hda /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2  -boot d

A new window will pop up with boot options. Press TAB. Some options will be visible. Append following to those options.

ks=http://IPADDRESS_OF_HOSTMACHINE:8080/ks.cfg

Once system installation is complete, run VM using following command.
Please note the omission of "-boot d".

# screen -S centos qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso -hda /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2

Update kernel boot parameters so that console can be redirected to terminal where qemu command is fired. then run following command. (note -nographic) (attached ks.cfg has this functionality). So you do not need to manually update kernel boot parameters.

# screen -S centos qemu-system-x86_64 -enable-kvm -cpu host -m 1024 -cdrom /home/testuser/Downloads/CentOS-7-x86_64-Minimal-1511.iso -hda /home/testuser/Downloads/KVM_IMAGES/centos-7.qcow2  -nographic

Note: Please note that We started VM in a screen session so that VM console can be attached or detached easily.
We do not need to do this when VMs are build using libvirtd. libvirtd provides many functionalities .

Java and Ruby Inheritance Comparison

Following is a Ruby Program. This is heavily commented with reference to Java and Ruby both.

class A
  def methoda_public
    puts "Hello from methoda_public self=#{self}"
  end

  def methoda_private
    puts "Hello from methoda_private self=#{self}"
  end

  def methoda_protected
    puts "Hello from methoda_protected self=#{self}"
  end

    # 1) Java => class methods are defined using static keyword. instances can access class methods in Java.
    # 2) Ruby => class methods are actually singleton methods of class itself. (in Ruby a class is an object itself.). class methods are actually defined in
               # singleton class of "class object" and hence called as singleton methods. Only that object can call singleton method on which it is defined and none other.
               # Singleton methods are Exclusive to a single particular object only. Hence even the instances created from class A cannot access class methods.
               # derived class can invoke base class' singleton object. Again instances of Derived class cannot access singleton method defined on base class.
  def self.methoda_classmethod_public
    puts "hello from methoda_classmethod_public self=#{self}"
  end

  private :methoda_private
  protected :methoda_protected
end

class B < A
  def public_methodb_calling_private_methoda

      # 1) Java => private members are not inherited directly in derived class. So this will not work in Java.
                 # Derived class inherits base class' protected and public members. So if Base class' protected/public method access private member
                 # then Derived class can also 'work' on private member via inherited protected/public methods.
      # 2) Ruby => private members are inherited in Ruby. So this will work in ruby.
    methoda_private       

      # 1) Java => you cannot access private memebers directly in derived class. So this will not work in Java.
      # 2) Ruby => Will not work as private members must be accessed WITHOUT explicit receiver in derived class.
    #A.new.methoda_private  # A new object is created and we are trying to access its private member.  will not work.
    #self.methoda_private   # An equivalent in Java would be this.methoda_private(). But ofcourse in Java private members are not inherited so will not work.
      
      # 1) Java => Protected members can be accessed in derived class directly becuase they are inherited. So this will work.
      # 2) Ruby => Protected members are inherited in Ruby as well. So this will work.
    methoda_protected

      # 1) Java => protected memeber cannot be accessed in drived class using reference variable(i.e. by creating object).
      # 2) Ruby => This will work. But this will NOT work in Ruby also if below expression i.e. A.new.methoda_protected is called outside class B. 
    A.new.methoda_protected

      # 1) Java => A Java equivalent would be this.methoda_protected() . This will work in Java. As protected methods are inherited in Java directly. Needed verification.
      # 2) Ruby => This will work. Ruby will be calling inherited method. Java will also do the same.
    self.methoda_protected

      # 1) Java => Instance method can call static method (class method).
      # 2) Ruby => This will not work. Instance method CANNOT call class method.
    #methoda_classmethod_public

      # 1) Java => instance method can call static method (class method). In java it would be this.methoda_classmethod_public()
      # 2) Ruby => this will not work. Because inside instance method self will be referring to the instance of B. And instance of B CANNOT call
                 # singleton method of class A.
    #self.methoda_classmethod_public
  end

    # 1) Java => Both below statements are not allowed in Java. These are called outside any method definition.
    # 2) Ruby => Both below will work in Ruby. Hence class methods are inherited but only to be called by derived class itself.
               # class methods are singleton method. Normally singleton methods are called by only their own onjects.
               # But when in case of singleton methods defined on class (class methods), they can be called by dervied class also. (Not by derived class instances).
  methoda_classmethod_public
  self.methoda_classmethod_public
end

#A.new.methoda_private     # => will not work. (same as in Java)
#A.new.methoda_protected   # => will not work. (same as in Java). Compare and take a note of this expression inside class B as well.
#A.new.methoda_classmethod_public # => Will not work in Ruby but will work in Java. Java syntax would be: A a = new A(); a.methoda_classmethod_public() .

B.new.public_methodb_calling_private_methoda
B.methoda_classmethod_public  # => will work both in Java and Ruby

A Simple Vagrant Setup

Preparation:

mkdir MCOLLECTIVE_VAGRANT
cd MCOLLECTIVE_VAGRANT
vagrant init
vagrant box add puppetlabs/centos-6.6-64-nocm

modify contents of MCOLLECTIVE_VAGRANT/Vagrant file to look like below:

Single Instance Vagrantfile

$ cat Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.hostname = "vagrant.virtual.machine"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
end

Multi Instance Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
     config.vm.define "first" do |first|
          config.vm.hostname = "first.virtual.machine"
     end
     config.vm.define "second" do |second|
          config.vm.hostname = "second.virtual.machine"
     end
end

To start Vagrant instances.

vagrant up

To login into new VM machine.

vagrant ssh

To check status

vagrant global-status

Config with autostart: false and Mix of CentOS and Debian.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
     config.vm.define "first", autostart: false do |first|
          config.vm.hostname = "first.virtual.machine"
     end
     config.vm.define "second", autostart: false do |second|
          config.vm.hostname = "second.virtual.machine"
     end
     config.vm.define "debian-7.4", autostart: false do |second|
          config.vm.box = "chef/debian-7.4"
          config.vm.hostname = "debian.virtual.machine"
     end
end

Bring up only Debian machine

vagrant up debian-7.4
vagrant ssh debian-7.4

Bring up now first machine

vagrant up first
vagrant ssh first

Note: When autostart: false is configured , Vagrant machines have to be brought up manually

Understanding JAVA Nested Classes

class OuterClass {
    private String outerClassPrivateInstanceVar = "outerClassPrivateInstanceVar";
    public static String outerClassPublicStaticVar = "outerClassPublicStaticVar";
    public String outerClassPublicInstanceVar = "outerClassPublicInstanceVar";

    void outerClassInstanceMethod() {
        System.out.println("Inside outerClassInstanceMethod.");
    }

    static void outerClassStaticMethod() {
        System.out.println("Inside outerClassStaticMethod.");
    }

    // NON-STATIC Inner class
    class InnerClass {
        // Non-static innerClass cannot have static members. Following will not work.
        // static String InnerClassStaticVar = "InnerClassStaticVar";
        public void innerClassMethod() {
            System.out.println("Inside innerClassMethod.");
        }
        public void innerClassAccessOuterClassVars() {
            System.out.println(outerClassPrivateInstanceVar);
            System.out.println(outerClassPublicInstanceVar);
            System.out.println(outerClassPublicStaticVar);
        }
    }

    // STATIC Inner class
    static class InnerClassStatic {
        // Following is allowed in Static InnerClass but not in non-static inner class.
        static String StaticInnerClassStaticVar = "StaticInnerClassStaticVar";
        String InnerClassStaticInstanceVar = "InnerClassStaticInstanceVar";
        public void innerClassStaticMethod() {
            System.out.println("Inside InnerClassStatic.");
        }
        public void innerClassStaticAccessOuterClassVars() {
            // Following will not work. Beacuse InnerClassStatic is like static member inside OuterClass
            // and static members/methods can never access instance members.
            //System.out.println(outerClassPrivateInstanceVar);
            //System.out.println(outerClassPublicInstanceVar);

            System.out.println(outerClassPublicStaticVar);
        }
    }

    // Local Inner Class inside Instance method.
    void outerClassInstanceMethodForLocalClass() {
        // Local Inner Classes cannot be private, protected, public and static. Because inside method
        // these keywords have no sense. We don't use these words inside methods.
        class LocalClassInInstanceMethod {
            // Static members are not allowed in Local Inner classes defined inside instance method.
            // Following will not work.
            //static String LocalClassInInstanceMethodStaticVar = "LocalClassInInstanceMethodStaticVar";
            void localClassInInstanceMethod_instanceMethod() {
                System.out.println("Inside localClassInInstanceMethod_instanceMethod.");
            }
        }
        LocalClassInInstanceMethod localClassInInstanceMethodObj = new LocalClassInInstanceMethod();
        localClassInInstanceMethodObj.localClassInInstanceMethod_instanceMethod();
    }

    // Local Inner Class inside Static method.
    static void outerClassStaticMethodForLocalClass() {
        // Local Inner Classes cannot be private, protected, public and static. Because inside method
        // these keywords have no sense. We don't use these words inside methods.
        class LocalClassinStaticMethod {
            // Static members are not allowed in Local Inner classes defined inside static  method.
            // Following will not work.
            //static String LocalClassinStaticMethodStaticVar = "LocalClassinStaticMethodStaticVar";
            void LocalClassinStaticMethod_instanceMethod() {
                System.out.println("Inside LocalClassinStaticMethod_instanceMethod.");
            }
        }
        LocalClassinStaticMethod locallassinStaticMethodObj = new LocalClassinStaticMethod();
        locallassinStaticMethodObj.LocalClassinStaticMethod_instanceMethod();
    }

    // Anonymous Inner classes
    interface InnerInterface {
        void InnerInterfaceMethod();
    }
    // Anonymous inner class object defined at class member level.
    InnerInterface anonymousClassObj = new InnerInterface() {
        public void InnerInterfaceMethod() {
            System.out.println("Inside InnerInterfaceMethod.");
        }
    };
    void callAnonymousObjectMethod() {
        anonymousClassObj.InnerInterfaceMethod();
    }
    // Anonymous inner class inside instance method.
    //void anonymousInnerClassInsideMethod() {
    OuterClass  anonymousInnerClassInsideMethod() {
        OuterClass anon = new OuterClass() {
            void outerClassInstanceMethod() {
                System.out.println("Inside anonymousInnerClassInsideMethod.");
            }
        };
        //anon.outerClassInstanceMethod();
        return anon;
    }
}

public class InnerClassesTest {
    public static void main(String[] args) {
        OuterClass outerClassObj = new OuterClass();

        // Call to instance method using instance variable.
        outerClassObj.outerClassInstanceMethod();

        // Call to static method using instance variable. In Ruby there is something called self instead of static. And
        // self.outerClassStaticMethod  in Ruby is actually a singleton method defined on a "class object (OuterClass)" and in Ruby class
        // itself is an object. And singleton methods can be called by only object on which it is defined, in this case OuterClass object.
        // Hence in Ruby object created by a class cannot access class method. Following is invalid in Ruby.
        outerClassObj.outerClassStaticMethod();

        // NON-STATIC INNER CLASSES
        // Creating InnerClass object.
        System.out.println("====NON-STATIC INNER CLASSES====");
        OuterClass.InnerClass innerClassObj = new OuterClass().new InnerClass();
        innerClassObj.innerClassMethod();

        // Following will not work. outerClassPublicInstanceVar is not defined in InnerClass.
        //System.out.println(innerClassObj.outerClassPublicInstanceVar);

        innerClassObj.innerClassAccessOuterClassVars();

        // STATIC INNER CLASSES.
        // Creating Static Innerclass object. You will be using Class not object.
        System.out.println("====STATIC INNER CLASSES====");
        OuterClass.InnerClassStatic innerClassStaticObj = new OuterClass.InnerClassStatic();
        // You cannot use OuterClass object to use StaticInnerClass object. This is in contrast with creating InnerClass(non-static) object.
        // Also normally an object can access static class fields. But Following will not work.
        //OuterClass.InnerClassStatic innerClassStaticObj2 = outerClassObj.InnerClassStatic();

        // Access static inner class' static member using static inner class' object.
        System.out.println(innerClassStaticObj.StaticInnerClassStaticVar);
        // Access static inner class' static member using Classes directly.
        System.out.println(OuterClass.InnerClassStatic.StaticInnerClassStaticVar);
        // Access static inner class' instance member using static inner class' object.
        System.out.println(innerClassStaticObj.InnerClassStaticInstanceVar);

        // LOCAL INNER CLASSES
        System.out.println("====LOCAL INNER CLASSES====");
        outerClassObj.outerClassInstanceMethodForLocalClass();
        outerClassObj.outerClassStaticMethodForLocalClass();
        OuterClass.outerClassStaticMethodForLocalClass();

        // ANONYMOUS INNER CLASSES
        System.out.println("====ANONYMOUS INNER CLASSES====");
        outerClassObj.callAnonymousObjectMethod();
        // Following works because, interface InnerInterface is visible in this class as well due to default access.
        // If inner interface in made private then following will not work. Only inner interfaces can be made private.
        outerClassObj.anonymousClassObj.InnerInterfaceMethod();

        OuterClass anonymousObj = new OuterClass() {
            void outerClassInstanceMethod() {
            System.out.println("inside anonymousObjOfOuterClassSubClass Method.");
            }
            void anotherMethod() {
                System.out.println("Inside anotherMethod.");
            }
        };
        anonymousObj.outerClassInstanceMethod();
        // Following will not work. Because anonymousObj is refering to actually subclass of OuterClass (in memory). But it is declared as OuterClass.
        // That means, anonymousObj is actually a "Subclass of OuterClass" but its type is declared as OuterClass. Hence polymorphism will come in.
        // anonymousObj being declared as superclass variable will only be able to execute methods defined in super class not in sub class. Hence
        // following will not work.
        //anonymousObj.anotherMethod();

        //Following works provided relevant lines are un-commented inside anonymousInnerClassInsideMethod instance method.
        //outerClassObj.anonymousInnerClassInsideMethod();
        OuterClass out = outerClassObj.anonymousInnerClassInsideMethod();
        out.outerClassInstanceMethod();
    }
}

Compile and Output:

$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
$
$ javac InnerClassesTest.java
$ java InnerClassesTest
Inside outerClassInstanceMethod.
Inside outerClassStaticMethod.
====NON-STATIC INNER CLASSES====
Inside innerClassMethod.
outerClassPrivateInstanceVar
outerClassPublicInstanceVar
outerClassPublicStaticVar
====STATIC INNER CLASSES====
StaticInnerClassStaticVar
StaticInnerClassStaticVar
InnerClassStaticInstanceVar
====LOCAL INNER CLASSES====
Inside localClassInInstanceMethod_instanceMethod.
Inside LocalClassinStaticMethod_instanceMethod.
Inside LocalClassinStaticMethod_instanceMethod.
====ANONYMOUS INNER CLASSES====
Inside InnerInterfaceMethod.
Inside InnerInterfaceMethod.
inside anonymousObjOfOuterClassSubClass Method.
Inside anonymousInnerClassInsideMethod.

libvirtd Netwroking

If libvirtd is installed then it comes with a default network called "default".

# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes

If this network is not active you can make it active using following command. The moment you make it active, it places various IPTABLES rules to do NAT forwarding between host and guest.

# virsh net-start default
Network default started

When "default" network is brought up, a bridge is created called virbr0. This bridge is also assigned an IP Address automatically.

# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 34:17:eb:d5:8d:fe brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether fe:e6:45:38:00:ff brd ff:ff:ff:ff:ff:ff

# ip add show virbr0
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether fe:e6:45:38:00:ff brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever

IPTABLES will now be in place.

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc

You can destroy this "default" network by using following command. As a result of this , iptables will vanish, bridge virbr0 will disappear and "default" network will become inactive.

# virsh net-destroy defaultNetwork default destroyed

# iptables -LChain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# ip ad virbr0
Command "virbr0" is unknown, try "ip addr help".

# brctl show
bridge name bridge id STP enabled interfaces

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 34:17:eb:d5:8d:fe brd ff:ff:ff:ff:ff:ff

# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes 

Activate DHCP in "default" network.

# virsh net-list --all

# virsh net-start default 

# virsh net-update default add ip-dhcp-range '<range start="192.168.122.100" end="192.168.122.254" />' --live

Firefox in Docker

Host OS : Fedora 22
Docker Version: 1.7.0-1

steps:

$ mkdir firefox

$ cd firefox; touch Dockerfile

$ ( cat <<-EOF
FROM fedora
MAINTAINER Spare Slant "spareslant@gmail.com"
ENV REFRESHED_AT 2015-11-21
RUN dnf -y install firefox
RUN dnf -y install dejavu-sans-fonts dejavu-serif-fonts
RUN useradd --shell /bin/bash --uid 1000 -m testuser
USER testuser
ENV HOME /home/testuser
CMD ["/usr/bin/firefox", "--no-remote"]
EOF
) > Dockerfile

$ docker build -t="spareslant/firefox:v2"

To run above docker :

docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -v /etc/machine-id:/etc/machine-id -e DISPLAY=$DISPLAY --name firefox spareslant/firefox:v2

A firefox window will pop up. If you close firefox , docker process will also shutdown. To start it again run following: docker start firefox

Find repeated Group pattern match. Case study of Ruby, Python and Go

Find repeated Group pattern match. Case study of Ruby, Python and Go

Sometimes I have to find the a group of patterns that are occurring in a log file. Normal grep -f <filename> does that job but it does not print the delimiter that can distinguish between repeated found patterns.

I needed something that can tell me that it has found a few patterns from a group in a sequence and last pattern has been found. Draw a delimiter here and start search for group again in remaining file.

I first wrote it in Ruby. I had huge files to parse. Ruby script was talking quite long to finish. As a case study I decided to write same utility using Python and Go.

Needless to say , Go was much faster.

Following are the codes.

Following are the sample data files.

pattern file :

hotmail
yahoo
google
gmail

Subject file:

This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Another one is gmail
Now is google
Another one is gmail
But this is going to be a yahoo
This is hotmail test
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR

Following is the output:

$ go run atest.go pattern subjectfile
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========
This is hotmail test
But this is going to be a yahoo
Another one is gmail
===========
Now is google
Another one is gmail
===========
But this is going to be a yahoo
This is hotmail test
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========

Launch AWS EC2 nstance using SaltStack

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli
pip install M2Crypto
pip install pyzmq

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Become root on MacBook now:

mkdir /etc/salt
touch /etc/salt/cloud.profiles
touch /etc/salt/cloud.providers

cat /etc/salt/cloud.profiles
base_ec2_private:
     provider: amazon_ireland_region
     image: ami-9d23aeea

cat /etc/salt/cloud.providers
amazon_ireland_region:
     id: ABCDEFGHIJKLMNO235M
     key: aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv
     keyname: MyEC2Key
     private_key: /Users/MacUser/EC2/MyEC2Key.pem
     location: eu-west-1
     availability_zone: eu-west-1a
     securitygroup: MySecurityGroupSSHOnly
     size: t2.micro
     del_root_vol_on_destroy: True
     ssh_username: ec2-user
     rename_on_destroy: True
     ssh_interface: public_ips
     provider: ec2

Launch Instance now:

source ~MacUser/PythonVirtENVs/SaltStack/bin/activate
salt-cloud --profile=base_ec2_private First_Instance

Launch AWS EC2 instance using awscli

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Following command will spin a FREE TIER instance in AWS cloud

aws ec2 run-instances --image-id ami-9d23aeea --key-name MyEC2Key --instance-type t2.micro --count 1 --security-groups MySecurityGroupSSHOnly

Playing and Encoding Video files on Linux

DVD tracks and Chapters info

mplayer -identify dvd:// -dvd-device /media/DVD_FOLDER/

MPlayer 29092-4.4.0 (C) 2000-2009 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing dvd://.
libdvdread: Encrypted DVD support unavailable.
libdvdread: Couldn't find device name.
ID_DVD_TITLES=2
ID_DVD_TITLE_1_CHAPTERS=51
ID_DVD_TITLE_1_ANGLES=1
ID_DVD_TITLE_2_CHAPTERS=50
ID_DVD_TITLE_2_ANGLES=1
ID_DVD_TITLE_1_LENGTH=13653.600
ID_DVD_TITLE_2_LENGTH=13150.166

PLAYING OPTIONS:
If the DVD video is downloaded on the harddisk, following are the various options to run videos from that DVD folder.

1) To open the DVD folder as a DVD. It will present all the menus of DVD. (note the dvdnav option)

mplayer -mouse-movements dvdnav:// -dvd-device /media/DVD_FOLDER/

2) To run the only chapter 2 from the DVD

mplayer dvd:// -chapter 2-2 -dvd-device /media/DVD_FOLDER/

3) To run the only chapter 5 to 7 from the DVD

mplayer dvd:// -chapter 5-7 -dvd-device /media/DVD_FOLDER/

4) To run 1st track of DVD

mplayer dvd://1 -dvd-device /media/DVD_FOLDER/

5) To view only the specific portion of a video file (based on time slice)

mplayer -ss 00:00:30 -endpos 30 dvdnav://1 -dvd-device /media/DVD_FOLDER/

CONVERSION OPTIONS:

mencoder dvd:// -chapter 2-2 -vf scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af-add lavcresample=44100 -vf-add harddup -vf-add scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/


mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -vf scale=320:240 -o yahoo.mp4 -mc 0 -oac mp3lame -lameopts vbr=3:br=128:q=9 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25:vpass=1 -of lavf -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -ovc frameno -oac mp3lame -lameopts vbr=0:br=96 -o frame.mp3 -of lavf -lavfopts format=mp3 -dvd-device /media/DVD_FOLDER/

Extracting RAW audio AUDIO from a Video File (in PCM format i.e. wav)

mplayer /media/Video.avi -vo null -vc null -ao pcm:fast:file=/tmp/audio.wav

Conversion using ffmpeg

/opt/ffmpeg/bin/ffmpeg -i /media/video.avi -i /tmp/audio.mp3 -map 0:0 -map 1:0 -vcodec msmpeg4v2 -s qvga -b 512k -r 30 -acodec copy /tmp/last.mp4

Cutting a scene from a video file

In mplayer , when file is being played, press "o" to see the current position in (time format).
use this information to know how long scene needs to be cut.


mencoder -ss 00:07:20 -endpos 190 -o /tmp/blue.avi -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 input_vdo_file.avi

Extracting DVD information using HandBrake

./HandBrakeCLI --scan  -i /Volumes/VDO_FOLDER

Extracting Chapter From a DVD dump on a disk using HandBrake

./HandBrakeCLI -i /Volumes/DVD_FOLDER/  -e x264  -q 20.0 -a 1,1 -E faac,copy:ac3 -B 320,160 -6 dpl2,none -R Auto,Auto -D 0.0,0.0 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback ffac3 -f mp4 -4 --decomb --loose-anamorphic --modulus 2 -m --x264-preset medium --h264-profile high --h264-level 4.1 -c 1 -o $HOME/chap1mod.mp4

Note:  “-c 1” in above command is extracting 1st chapter from DVD.

Python Regex Usage

Python Regex Usage

Perl Regex Usages

Multiline Matchings:

Inside a perl script:

#! /usr/bin/perl
$str = "This is a test
Will it work
lets see
";
if ($str =~ m!^This.+test$!)
{
     print "matched\n"
}

Following are the contents of file "data"

# cat data
This is a test file
Will it work
lets see

# As perl one liners:

$ cat data | perl -w -e '$/=""; while(<>) { m!^This.+file$!m and print }'
This is a test file
Will it work
lets see


# DEFAULT BEHAVIOUR
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH

# s modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ 


# m modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^let.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
lets see

# sm modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^lets.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ 

$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+it).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it


# BACKREFRENCES:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\1! and print "MATCHED\n"}'
MATCHED
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\g{-1}! and print "MATCHED\n"}'
MATCHED

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED

# CAPTURING and NON CAPTURING GROUPING:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "$1\n"}'
vis  

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?:vis).+\g{testmatch}! and print "$1\n"}'
Reference to nonexistent named group in regex; marked by <-- here="" in="" m="" testmatch="">vis).+\g{testmatch <-- -e="" -w="" 1.="" a="" at="" echo="" here="" line="" perl="" vis="" while="">) { m!(?:vis).+! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+! and print "$1\n"}'
vis
$

$ echo "datchet etet it underway" | perl -w -e '$/=""; while(<>) { m!(et){2}\s+! and print "MATCHED\n"}'
MATCHED
 
# NAMED CAPTURING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?test file)! and print "$+{fm}\n" or print "NOMATCH\n" }'
test file
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "$1" or print "NOMATCH\n" }'
Use of uninitialized value $1 in string at -e line 1, <> chunk 1.
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED 


# inside modifiers:
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+test! and print "MATCHED\n" or print "UNMATCHED\n"}'
UNMATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+TEST! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ 

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+((?i)test)! and print "$1\n"}'
TEST
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED

# inside modifiers (multiline modifier trick):
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s)(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!((?s)^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work

# inside modifiers non-capturing match:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED


# LOOK AHEAD ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(.+lets)(.+it.+$)!sm and print "$1 $3\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
NOMATCH


# LOOK BEHIND  ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(it)!sm and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(.+it)!sm and print "$1\n" or print "NOMATCH\n" }'
test file
Will it

# CONDITIONAL MATCHING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(asa)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'        # This is not a conditional matching
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "$1 $2\n" or print "NOMATCH\n" }'
Use of uninitialized value $2 in concatenation (.) or string at -e line 1, <> chunk 1.
test file 

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(anotherjunk))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH
 
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=junk)junk)!sm and print "MATCH\n" or print "NOMATCH\n" }'  # if condition is false then it simply skips
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk))!sm and print "MATCH\n" or print "NOMATCH\n" }'  # why is this working ?  no idea
MATCH

(?(?=condition)(then1|then2|then3)|(else1|else2|else3))

# Some Special cases to study:
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(ab)*! ; print "@arr\n";}'
ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((ab)*)! ; print "@arr\n";}'
ab ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)*)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)! ; print "@arr\n";}'
aba
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(?:a|b)*!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]*!g ; print "@arr\n";}'
aba  a 
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]*)!g ; print "@arr\n";}'
aba  a  
$  echo "abacaddb" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a   b  
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([ab]+)! ; print "$1\n";}'
aba
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([abc]+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((a|b|c)+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((?:a|b|c)+)! ; print "$1\n";}'
abaca


#======================================================

$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(a|b)+! ; print "@arr\n";}'
a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)+)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]+! ; print "@arr\n";}'
1
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]+)! ; print "@arr\n";}'
aba

Selective Backup with permissions preserved.

To copy selective files along with complete directory structure with all permissions preserved. (from one directory to another in same box).

find /var/log/allapps/apps1  | egrep '2013-12-02'  | rsync -Ravz --files-from=- /  /tmp/test

Above command will create a directory structure of var/log/allapps/apps1 + files(filtered by egrep above) inside apps1 dir inside /tmp/test along with all directories/files ownership and permissions intact.

Now take the backup of above copied folder.

cd /tmp/test
tar -cvpf test.tar var

Copy only specified files using rsync.
We need to copy only some files from remote system. These files are scattered around various directories. We also need to keep the directory structure on destination same as in source. Following command was used.

rsync --stats --progress -rvpogStDL --files-from=192.168.1.4:/tmp/200901files 192.168.1.4:/home/remote_files_and_documents 

Note1: /tmp/200901files contains list of files.
e.g.

file1
dir2/dir3/file2
dir6/file4

Note2: Source files are in local machine. But the list of files which are to be copied is being fetched from remote machine.

Remote Execution Tool using Screen

Remote Execution Tool

• No need to place root ssh keys
• No need to install any third party tool/library in advance.
• Keep record of every activity
• Simple and Plain

Often Sys Admins have to do similar kind of tasks on a number of machines. Each and everyone has its own way. Some uses configuration tools like puppet/cfengine. Some uses password less ssh parallel execution or someone might be using either expect or some modules like paramiko to achieve that.
This is also a similar kind of tool. It exploits the power of screen. What it requires to run.

• A central host having bash
• Same above central host having "screen" utility installed.

You do not even need your or root ssh keys in place. You will provide your password in passwd_file (see sample file in code).
What it can provide:

• logging on every host + on central host
• real time visual monitoring.
• can take actions on individual hosts as well.

Caveats:

• Only for private network.
• Knowing screen usage is advantageous.

HowtoRun

This tool consists of only one script that can be found at the bottom of this page. Following is the example run.

1.  Run without any argument.

2. Following Directories are created after running create_connection_configs.sh script.

3. Now create a file boxes. This file will contain the list of hostnames or IPAddresses of remote boxes. REMOTE_TASKS directory will contain directories that will have various tasks in form of script to be executed on remote hosts listed in boxes file. LOGS directory will contain logs of all activities that we carry out per host basis. 

4.  Now we will place local user ssh keys to remote machine. Note that this is for normal user only. Not root user. We assume that localuser is testuser  and testuser exists on each remote machine having same password.  Now create a file passwd_file and  type in testuser password in it followed by ctrl-v+ctrl-Enter 

COPY SSH KEYS to all remote hosts

5.  Run ./create_connection_configs.sh  SSHKEYCOPY command. Following output will be shown.

6 . Follow above screen instruction carefully. Open second terminal window and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -c mainscreenrc

7.  Keep Following first terminal window instructions. And run screen -r on second terminal. Following output follows. Pay attention to the bottom of screen (in red) also.

8.  At this point in second terminal run ctrl-a, ctrl+". Following output will follow. It will show two screen sessions. (one for each host listed in boxes file).

9. Keep following instructions from first terminal window, and open third terminal and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -X readbuf ./passwd_file

Above command will read password contained in passwd_file. Please note passwd_file contains password+<Enter> character.

Keep following instructions from first terminal window and now run, following command on third terminal window.

screen -S MasterSession -X at "#" stuff $'ssh-keyscan -t rsa \$TARGETHOST >> $HOME/.ssh/known_hosts ; ssh-copy-id -i $HOME/.ssh/id_rsa.pub \$TARGETHOST\n'

10. At this point switch to 2nd terminal to see what has happened.

11. As you see above, it is asking for testuser password so that it can copy ssh-keys.  Take a note of bottom of window as well (in red). It tells us that screen session is for 192.168.122.101. If we want to see screen session of 192.168.122.102. then press ctrl-a,ctrl-".   You will see following screens.

12. Keep following instructions in first terminal. Now switch to third terminal and run following command.

screen -S MasterSession -X at "#" paste "."

Above command will paste password on to above two screen sessions opened in 2nd terminal. 

You will see following output.

13. As the above output shows ssh keys have been copied. You can cross verify it.

14. Checkout the logs in LOGS directory. Everything is recorded here.

RUN Tasks on to Remote Machines.

15. Assuming testuser ssh-keys are in place. and testuser is allowed to run some sudo commands (Sudo with password.)

16. Create a task directory and a script that needs to be executed on remote hosts. Please note we have already populated boxes file. Following is the screenshot.

17. Now run following command.

./create_connection_configs.sh firsttask

Following is output.

18. As you see above script.sh has been copied to remote hosts.

19. Follow instruction on terminal window and open 2nd terminal now and run following commands.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -c mainscreenrc

screen -r

20. The moment you run screen -r , you will notice that you are already connected with remote host. To make sure you are connected with other remote hosts too, then press ctrl-a, ctrl-"  to list the screen sessions.  

21.  Follow on-screen instruction and open third terminal . and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -X readbuf ./passwd_file

screen -S MasterSession -X at "#" stuff $'sudo su -\n'

22.  Now switch back to terminal-2 to verify that you are being prompted for sudo password.

Use ctrl-a,ctrl-" to switch to another screen session to verify same.

23. Now switch back to terminal 3 and run following command.

screen -S MasterSession -X at "#" paste "."

Now switch back to terminal-2, you will see that you have logged in to both the machines.

24.  Follow instructions from terminal-1 window and run following command on third terminal.

screen -S MasterSession -X at "#" stuff $'cd /tmp/firsttask ; bash ./script.sh\n'

Switch back to 2nd terminal window to verify results.

25. Check LOGS directory for each action we took.

create_connection_configs.sh

LDAP Queries Examples

1) To determine about the Directory Server info and its capability
ldapsearch  -h ldapserver -Z -x -b '' -s base 'objectClass=*'

2) To search directory from base just one level down.
ldapsearch  -h ldapserver -Z -x -b 'dc=example,dc=net' -s onelevel

3) Pick desired dn from above output and traverse further
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net' -s onelevel

4) To search for a User whose only half name is known
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net' -s onelevel '(cn=Tes*)'
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net'  '(cn=Tes*)'

ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net' -s onelevel '(cn=Tes*)'  => will not yield any result
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net'  '(cn=Tes*)'   => this will yield result

5) to list only specific information (like common name)
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net'  '(uid=testu)' cn

6) To list sudo users and commands
ldapsearch  -h ldapserver -ZZ -x -b 'ou=SUDOers,dc=example,dc=net' -s one

7) To list all groups
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=com' 'objectClass=*roup*'

8) To list all groups in which "cn=Test User, ou=Users,dc=example, dc=com"  exists
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=com' '(&(objectClass=*roup*)(uniqueMember=cn=Test User,ou=Users,dc=example, dc=com))'

9) To list entries using Admin or Directory Manager
ldapsearch -L -b 'ou=Users, dc=example, dc=com' -x -D "cn=directory manager" -w 'Password'

10) Anonymous binding and listing everything
ldapsearch -h ldapserver "objectClass=*"
ldapsearch  -h ldapserver  -x -b 'dc=appauth,dc=example,dc=net'

11) Binded LDAP search
ldapsearch  -H ldaps://ldapserver -D 'uid=Manager,ou=App1,dc=appauth,dc=org'  -b 'dc=appauth,dc=org' -w 'Password' -s onelevel

ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -s onelevel

12) To list all top level trees with Binded authentication
ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -b "" -s base "objectclass=*"

ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -b 'dc=appauth,dc=org'

12) TO list against AD (Active Directory).
ldapsearch -H ldap://AD-Server -x -D 'domain\user' -W

Screen Sessions Management

I used this method when I need to login to multiple remote machines using ssh and then become root and then do things interactively simultaneously on them. 

Following steps were performed:

^M => denotes  "Enter" on keyboard. (ctrl-v Enter => ^M)

screen^M  => can also be run as screen -S  <session name>^M

ctrl-a c  => 2 times to create two screen windows in one screen session.

create a file e.g  /tmp/screen_bufferfile  => this file will work as a screen buffer file.

Note : screen default buffer register is called "."  ( dot )

scenario 1 :  Sending Identical commands to everywhere non-interactively.

step 1 = ctrl-a : readbuf /tmp/screen_bufferfile  => this will copy the contents of file in "." register

step 2 = ctrl-a : at "#" paste .   => this command will paste the contents of "." register on every screen window (even to sshed windows also)

scenario 2 : Sending commands interactively.

ctrl-a : bufferfile /tmp/screen_bufferfile  => sets the default buffer file

ctrl-a <        => reads the contents of buffer to "." register

ctrl-a ]         => paste the contents of buffer to current screen window only ( NOT any other )

even the ssh password and sudo password prompts accepts passwords like this.

scenario 3: Sending commands using screen from outside

Create a screen session and create multiple screen in it using ctrl-a c (multiple times)

Get the screen session id (or name) using screen -list

To send unix shell commands(or any input) to ALL windows in a screen session run following 2 commands:

screen -S 3461.ttys000.MacBook-Pro  -X readbuf /tmp/screen_bufferfile

screen -S 3461.ttys000.MacBook-Pro  -X at "#" paste "."

/tmp/screen_bufferfile will contains unix commands or (any input) you want to send to screen windows.

To send unix shell command( or any input) to a particular window run following.

screen -S 3461.ttys000.MacBook-Pro  -X readbuf /tmp/screen_bufferfile

screen -S 3461.ttys000.MacBook-Pro  -p 5 -X paste "." 

readbuf and paste are used in scenes when we have to input some quite a good amount of data. This also relieves from

taking care of quotes and control characters that we have anyways to do when we use "stuff" command. e.g.

Above can also be done by using following:

screen -S 3461.ttys000.MacBook-Pro -p 5 -X stuff "hostname^M"

screen -S 3461.ttys000.MacBook-Pro -X at "#" stuff "hostname^M"

Above we have used actual data on command line itself to send to screen windows. Please note that we have to explicitly used ^M characters to have Enter effect. But this is not the case when we create /tmp/screen_bufferfile.

screen -S javaupdate -X screen   => will spawn screen windows inside screen session (named javaupdate)

=====================================================

To create multiple windows in one screen session non-interactively

screen -d -m -S TEST   => creates a screen session with name TEST and detach it. Does not go into screen session.

screen -S TEST -X screen -t yahoo 03 ssh remotemachine  => creates a screen window inside TEST with name yahoo at window no 3. In window no 3 , there will an ssh session to remotemachine.

Create Fedora Installer ISO using GRUB

Introduction

This is just method to produce the Fedora Installer DVDs using the Grub boot loader instead of ISOLINUX.

Software Used

Base OS used, on which Fedora-13 installer was created was , Ubuntu 8.04.3 LTS (hardy). Fedora-13 will be used for which installer will be created using GRUB as boot loader.

Steps Performed
Copy the required files
Copy whole DVD contents

mkdir grubdvdrom
mount -o loop /mnt/ISO-IMAGES/Fedora/Fedora-13-i386-DVD.iso /tmp/fedora-13
rsync -avz /tmp/fedora-13/ grubdvdrom/
umount /tmp/fedora-13

Copy grub files from Base OS

mkdir grubdvdrom/boot
cp -r /boot/grub grubdvdrom/boot
cp grubdvdrom/isolinux/{vmlinuz,initrd.img} grubdvdrom/boot
cp /usr/lib/grub/x86_64-pc/stage2_eltorito grubdvdrom/boot/grub

Note: Fedora DVDROM contains some hidden files like .treeinfo, which are must in order to recognize local repos for anaconda installer. If these files are not copied then anaconda will go only to internet for repos. There we used rsync to make sure each file has been copied.Also you have to maintain the tree structure of boot/grub as a whole.

Create menu.lst file
create or modify grubdvdrom/boot/grub/menu.lst file with following contents.
title Fedora-13 Grub ISO Installer
kernel /boot/vmlinuz
initrd /boot/initrd.img

Create ISO

cd grubdvdrom
mkisofs -o Fedora-13.iso -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table -J -r -hide-rr-moved -V Fedora