A Simple Vagrant Setup

Preparation:

mkdir MCOLLECTIVE_VAGRANT
cd MCOLLECTIVE_VAGRANT
vagrant init
vagrant box add puppetlabs/centos-6.6-64-nocm

modify contents of MCOLLECTIVE_VAGRANT/Vagrant file to look like below:

Single Instance Vagrantfile

$ cat Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.hostname = "vagrant.virtual.machine"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
end

Multi Instance Vagrantfile

# -*- mode: ruby -*-
# vi: set ft=ruby :
Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
     config.vm.define "first" do |first|
          config.vm.hostname = "first.virtual.machine"
     end
     config.vm.define "second" do |second|
          config.vm.hostname = "second.virtual.machine"
     end
end

To start Vagrant instances.

vagrant up

To login into new VM machine.

vagrant ssh

To check status

vagrant global-status

Config with autostart: false and Mix of CentOS and Debian.

# -*- mode: ruby -*-
# vi: set ft=ruby :

Vagrant.configure(2) do |config|
     config.vm.box = "puppetlabs/centos-6.6-64-nocm"
     config.vm.network "public_network"
     config.vm.provider "virtualbox" do |v|
          v.memory = 1024
          v.cpus = 1
     end
     config.vm.define "first", autostart: false do |first|
          config.vm.hostname = "first.virtual.machine"
     end
     config.vm.define "second", autostart: false do |second|
          config.vm.hostname = "second.virtual.machine"
     end
     config.vm.define "debian-7.4", autostart: false do |second|
          config.vm.box = "chef/debian-7.4"
          config.vm.hostname = "debian.virtual.machine"
     end
end

Bring up only Debian machine

vagrant up debian-7.4
vagrant ssh debian-7.4

Bring up now first machine

vagrant up first
vagrant ssh first

Note: When autostart: false is configured , Vagrant machines have to be brought up manually

Understanding JAVA Nested Classes

class OuterClass {
    private String outerClassPrivateInstanceVar = "outerClassPrivateInstanceVar";
    public static String outerClassPublicStaticVar = "outerClassPublicStaticVar";
    public String outerClassPublicInstanceVar = "outerClassPublicInstanceVar";

    void outerClassInstanceMethod() {
        System.out.println("Inside outerClassInstanceMethod.");
    }

    static void outerClassStaticMethod() {
        System.out.println("Inside outerClassStaticMethod.");
    }

    // NON-STATIC Inner class
    class InnerClass {
        // Non-static innerClass cannot have static members. Following will not work.
        // static String InnerClassStaticVar = "InnerClassStaticVar";
        public void innerClassMethod() {
            System.out.println("Inside innerClassMethod.");
        }
        public void innerClassAccessOuterClassVars() {
            System.out.println(outerClassPrivateInstanceVar);
            System.out.println(outerClassPublicInstanceVar);
            System.out.println(outerClassPublicStaticVar);
        }
    }

    // STATIC Inner class
    static class InnerClassStatic {
        // Following is allowed in Static InnerClass but not in non-static inner class.
        static String StaticInnerClassStaticVar = "StaticInnerClassStaticVar";
        String InnerClassStaticInstanceVar = "InnerClassStaticInstanceVar";
        public void innerClassStaticMethod() {
            System.out.println("Inside InnerClassStatic.");
        }
        public void innerClassStaticAccessOuterClassVars() {
            // Following will not work. Beacuse InnerClassStatic is like static member inside OuterClass
            // and static members/methods can never access instance members.
            //System.out.println(outerClassPrivateInstanceVar);
            //System.out.println(outerClassPublicInstanceVar);

            System.out.println(outerClassPublicStaticVar);
        }
    }

    // Local Inner Class inside Instance method.
    void outerClassInstanceMethodForLocalClass() {
        // Local Inner Classes cannot be private, protected, public and static. Because inside method
        // these keywords have no sense. We don't use these words inside methods.
        class LocalClassInInstanceMethod {
            // Static members are not allowed in Local Inner classes defined inside instance method.
            // Following will not work.
            //static String LocalClassInInstanceMethodStaticVar = "LocalClassInInstanceMethodStaticVar";
            void localClassInInstanceMethod_instanceMethod() {
                System.out.println("Inside localClassInInstanceMethod_instanceMethod.");
            }
        }
        LocalClassInInstanceMethod localClassInInstanceMethodObj = new LocalClassInInstanceMethod();
        localClassInInstanceMethodObj.localClassInInstanceMethod_instanceMethod();
    }

    // Local Inner Class inside Static method.
    static void outerClassStaticMethodForLocalClass() {
        // Local Inner Classes cannot be private, protected, public and static. Because inside method
        // these keywords have no sense. We don't use these words inside methods.
        class LocalClassinStaticMethod {
            // Static members are not allowed in Local Inner classes defined inside static  method.
            // Following will not work.
            //static String LocalClassinStaticMethodStaticVar = "LocalClassinStaticMethodStaticVar";
            void LocalClassinStaticMethod_instanceMethod() {
                System.out.println("Inside LocalClassinStaticMethod_instanceMethod.");
            }
        }
        LocalClassinStaticMethod locallassinStaticMethodObj = new LocalClassinStaticMethod();
        locallassinStaticMethodObj.LocalClassinStaticMethod_instanceMethod();
    }

    // Anonymous Inner classes
    interface InnerInterface {
        void InnerInterfaceMethod();
    }
    // Anonymous inner class object defined at class member level.
    InnerInterface anonymousClassObj = new InnerInterface() {
        public void InnerInterfaceMethod() {
            System.out.println("Inside InnerInterfaceMethod.");
        }
    };
    void callAnonymousObjectMethod() {
        anonymousClassObj.InnerInterfaceMethod();
    }
    // Anonymous inner class inside instance method.
    //void anonymousInnerClassInsideMethod() {
    OuterClass  anonymousInnerClassInsideMethod() {
        OuterClass anon = new OuterClass() {
            void outerClassInstanceMethod() {
                System.out.println("Inside anonymousInnerClassInsideMethod.");
            }
        };
        //anon.outerClassInstanceMethod();
        return anon;
    }
}

public class InnerClassesTest {
    public static void main(String[] args) {
        OuterClass outerClassObj = new OuterClass();

        // Call to instance method using instance variable.
        outerClassObj.outerClassInstanceMethod();

        // Call to static method using instance variable. In Ruby there is something called self instead of static. And
        // self.outerClassStaticMethod  in Ruby is actually a singleton method defined on a "class object (OuterClass)" and in Ruby class
        // itself is an object. And singleton methods can be called by only object on which it is defined, in this case OuterClass object.
        // Hence in Ruby object created by a class cannot access class method. Following is invalid in Ruby.
        outerClassObj.outerClassStaticMethod();

        // NON-STATIC INNER CLASSES
        // Creating InnerClass object.
        System.out.println("====NON-STATIC INNER CLASSES====");
        OuterClass.InnerClass innerClassObj = new OuterClass().new InnerClass();
        innerClassObj.innerClassMethod();

        // Following will not work. outerClassPublicInstanceVar is not defined in InnerClass.
        //System.out.println(innerClassObj.outerClassPublicInstanceVar);

        innerClassObj.innerClassAccessOuterClassVars();

        // STATIC INNER CLASSES.
        // Creating Static Innerclass object. You will be using Class not object.
        System.out.println("====STATIC INNER CLASSES====");
        OuterClass.InnerClassStatic innerClassStaticObj = new OuterClass.InnerClassStatic();
        // You cannot use OuterClass object to use StaticInnerClass object. This is in contrast with creating InnerClass(non-static) object.
        // Also normally an object can access static class fields. But Following will not work.
        //OuterClass.InnerClassStatic innerClassStaticObj2 = outerClassObj.InnerClassStatic();

        // Access static inner class' static member using static inner class' object.
        System.out.println(innerClassStaticObj.StaticInnerClassStaticVar);
        // Access static inner class' static member using Classes directly.
        System.out.println(OuterClass.InnerClassStatic.StaticInnerClassStaticVar);
        // Access static inner class' instance member using static inner class' object.
        System.out.println(innerClassStaticObj.InnerClassStaticInstanceVar);

        // LOCAL INNER CLASSES
        System.out.println("====LOCAL INNER CLASSES====");
        outerClassObj.outerClassInstanceMethodForLocalClass();
        outerClassObj.outerClassStaticMethodForLocalClass();
        OuterClass.outerClassStaticMethodForLocalClass();

        // ANONYMOUS INNER CLASSES
        System.out.println("====ANONYMOUS INNER CLASSES====");
        outerClassObj.callAnonymousObjectMethod();
        // Following works because, interface InnerInterface is visible in this class as well due to default access.
        // If inner interface in made private then following will not work. Only inner interfaces can be made private.
        outerClassObj.anonymousClassObj.InnerInterfaceMethod();

        OuterClass anonymousObj = new OuterClass() {
            void outerClassInstanceMethod() {
            System.out.println("inside anonymousObjOfOuterClassSubClass Method.");
            }
            void anotherMethod() {
                System.out.println("Inside anotherMethod.");
            }
        };
        anonymousObj.outerClassInstanceMethod();
        // Following will not work. Because anonymousObj is refering to actually subclass of OuterClass (in memory). But it is declared as OuterClass.
        // That means, anonymousObj is actually a "Subclass of OuterClass" but its type is declared as OuterClass. Hence polymorphism will come in.
        // anonymousObj being declared as superclass variable will only be able to execute methods defined in super class not in sub class. Hence
        // following will not work.
        //anonymousObj.anotherMethod();

        //Following works provided relevant lines are un-commented inside anonymousInnerClassInsideMethod instance method.
        //outerClassObj.anonymousInnerClassInsideMethod();
        OuterClass out = outerClassObj.anonymousInnerClassInsideMethod();
        out.outerClassInstanceMethod();
    }
}

Compile and Output:

$ java -version
java version "1.8.0_66"
Java(TM) SE Runtime Environment (build 1.8.0_66-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.66-b17, mixed mode)
$
$ javac InnerClassesTest.java
$ java InnerClassesTest
Inside outerClassInstanceMethod.
Inside outerClassStaticMethod.
====NON-STATIC INNER CLASSES====
Inside innerClassMethod.
outerClassPrivateInstanceVar
outerClassPublicInstanceVar
outerClassPublicStaticVar
====STATIC INNER CLASSES====
StaticInnerClassStaticVar
StaticInnerClassStaticVar
InnerClassStaticInstanceVar
====LOCAL INNER CLASSES====
Inside localClassInInstanceMethod_instanceMethod.
Inside LocalClassinStaticMethod_instanceMethod.
Inside LocalClassinStaticMethod_instanceMethod.
====ANONYMOUS INNER CLASSES====
Inside InnerInterfaceMethod.
Inside InnerInterfaceMethod.
inside anonymousObjOfOuterClassSubClass Method.
Inside anonymousInnerClassInsideMethod.

libvirtd Netwroking

If libvirtd is installed then it comes with a default network called "default".

# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes

If this network is not active you can make it active using following command. The moment you make it active, it places various IPTABLES rules to do NAT forwarding between host and guest.

# virsh net-start default
Network default started

When "default" network is brought up, a bridge is created called virbr0. This bridge is also assigned an IP Address automatically.

# brctl show
bridge name bridge id STP enabled interfaces
virbr0 8000.000000000000 yes

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 34:17:eb:d5:8d:fe brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default
link/ether fe:e6:45:38:00:ff brd ff:ff:ff:ff:ff:ff

# ip add show virbr0
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether fe:e6:45:38:00:ff brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever

IPTABLES will now be in place.

# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:domain
ACCEPT tcp -- anywhere anywhere tcp dpt:domain
ACCEPT udp -- anywhere anywhere udp dpt:bootps
ACCEPT tcp -- anywhere anywhere tcp dpt:bootps

Chain FORWARD (policy ACCEPT)
target prot opt source destination
ACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHED
ACCEPT all -- 192.168.122.0/24 anywhere
ACCEPT all -- anywhere anywhere
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable
REJECT all -- anywhere anywhere reject-with icmp-port-unreachable

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
ACCEPT udp -- anywhere anywhere udp dpt:bootpc

You can destroy this "default" network by using following command. As a result of this , iptables will vanish, bridge virbr0 will disappear and "default" network will become inactive.

# virsh net-destroy defaultNetwork default destroyed

# iptables -LChain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination

# ip ad virbr0
Command "virbr0" is unknown, try "ip addr help".

# brctl show
bridge name bridge id STP enabled interfaces

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/ether 34:17:eb:d5:8d:fe brd ff:ff:ff:ff:ff:ff

# virsh net-list --all
Name State Autostart Persistent
----------------------------------------------------------
default inactive no yes 

Activate DHCP in "default" network.

# virsh net-list --all

# virsh net-start default 

# virsh net-update default add ip-dhcp-range '<range start="192.168.122.100" end="192.168.122.254" />' --live

Firefox in Docker

Host OS : Fedora 22
Docker Version: 1.7.0-1

steps:

$ mkdir firefox

$ cd firefox; touch Dockerfile

$ ( cat <<-EOF
FROM fedora
MAINTAINER Spare Slant "spareslant@gmail.com"
ENV REFRESHED_AT 2015-11-21
RUN dnf -y install firefox
RUN dnf -y install dejavu-sans-fonts dejavu-serif-fonts
RUN useradd --shell /bin/bash --uid 1000 -m testuser
USER testuser
ENV HOME /home/testuser
CMD ["/usr/bin/firefox", "--no-remote"]
EOF
) > Dockerfile

$ docker build -t="spareslant/firefox:v2"

To run above docker :

docker run -it -v /tmp/.X11-unix:/tmp/.X11-unix -v /etc/machine-id:/etc/machine-id -e DISPLAY=$DISPLAY --name firefox spareslant/firefox:v2

A firefox window will pop up. If you close firefox , docker process will also shutdown. To start it again run following: docker start firefox

Find repeated Group pattern match. Case study of Ruby, Python and Go

Find repeated Group pattern match. Case study of Ruby, Python and Go

Sometimes I have to find the a group of patterns that are occurring in a log file. Normal grep -f <filename> does that job but it does not print the delimiter that can distinguish between repeated found patterns.

I needed something that can tell me that it has found a few patterns from a group in a sequence and last pattern has been found. Draw a delimiter here and start search for group again in remaining file.

I first wrote it in Ruby. I had huge files to parse. Ruby script was talking quite long to finish. As a case study I decided to write same utility using Python and Go.

Needless to say , Go was much faster.

Following are the codes.

Following are the sample data files.

pattern file :

hotmail
yahoo
google
gmail

Subject file:

This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Another one is gmail
Now is google
Another one is gmail
But this is going to be a yahoo
This is hotmail test
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR

Following is the output:

$ go run atest.go pattern subjectfile
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========
This is hotmail test
But this is going to be a yahoo
Another one is gmail
===========
Now is google
Another one is gmail
===========
But this is going to be a yahoo
This is hotmail test
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========

Launch AWS EC2 nstance using SaltStack

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli
pip install M2Crypto
pip install pyzmq

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Become root on MacBook now:

mkdir /etc/salt
touch /etc/salt/cloud.profiles
touch /etc/salt/cloud.providers

cat /etc/salt/cloud.profiles
base_ec2_private:
     provider: amazon_ireland_region
     image: ami-9d23aeea

cat /etc/salt/cloud.providers
amazon_ireland_region:
     id: ABCDEFGHIJKLMNO235M
     key: aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv
     keyname: MyEC2Key
     private_key: /Users/MacUser/EC2/MyEC2Key.pem
     location: eu-west-1
     availability_zone: eu-west-1a
     securitygroup: MySecurityGroupSSHOnly
     size: t2.micro
     del_root_vol_on_destroy: True
     ssh_username: ec2-user
     rename_on_destroy: True
     ssh_interface: public_ips
     provider: ec2

Launch Instance now:

source ~MacUser/PythonVirtENVs/SaltStack/bin/activate
salt-cloud --profile=base_ec2_private First_Instance

Launch AWS EC2 instance using awscli

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Following command will spin a FREE TIER instance in AWS cloud

aws ec2 run-instances --image-id ami-9d23aeea --key-name MyEC2Key --instance-type t2.micro --count 1 --security-groups MySecurityGroupSSHOnly

Playing and Encoding Video files on Linux

DVD tracks and Chapters info

mplayer -identify dvd:// -dvd-device /media/DVD_FOLDER/

MPlayer 29092-4.4.0 (C) 2000-2009 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing dvd://.
libdvdread: Encrypted DVD support unavailable.
libdvdread: Couldn't find device name.
ID_DVD_TITLES=2
ID_DVD_TITLE_1_CHAPTERS=51
ID_DVD_TITLE_1_ANGLES=1
ID_DVD_TITLE_2_CHAPTERS=50
ID_DVD_TITLE_2_ANGLES=1
ID_DVD_TITLE_1_LENGTH=13653.600
ID_DVD_TITLE_2_LENGTH=13150.166

PLAYING OPTIONS:
If the DVD video is downloaded on the harddisk, following are the various options to run videos from that DVD folder.

1) To open the DVD folder as a DVD. It will present all the menus of DVD. (note the dvdnav option)

mplayer -mouse-movements dvdnav:// -dvd-device /media/DVD_FOLDER/

2) To run the only chapter 2 from the DVD

mplayer dvd:// -chapter 2-2 -dvd-device /media/DVD_FOLDER/

3) To run the only chapter 5 to 7 from the DVD

mplayer dvd:// -chapter 5-7 -dvd-device /media/DVD_FOLDER/

4) To run 1st track of DVD

mplayer dvd://1 -dvd-device /media/DVD_FOLDER/

5) To view only the specific portion of a video file (based on time slice)

mplayer -ss 00:00:30 -endpos 30 dvdnav://1 -dvd-device /media/DVD_FOLDER/

CONVERSION OPTIONS:

mencoder dvd:// -chapter 2-2 -vf scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af-add lavcresample=44100 -vf-add harddup -vf-add scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/


mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -vf scale=320:240 -o yahoo.mp4 -mc 0 -oac mp3lame -lameopts vbr=3:br=128:q=9 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25:vpass=1 -of lavf -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -ovc frameno -oac mp3lame -lameopts vbr=0:br=96 -o frame.mp3 -of lavf -lavfopts format=mp3 -dvd-device /media/DVD_FOLDER/

Extracting RAW audio AUDIO from a Video File (in PCM format i.e. wav)

mplayer /media/Video.avi -vo null -vc null -ao pcm:fast:file=/tmp/audio.wav

Conversion using ffmpeg

/opt/ffmpeg/bin/ffmpeg -i /media/video.avi -i /tmp/audio.mp3 -map 0:0 -map 1:0 -vcodec msmpeg4v2 -s qvga -b 512k -r 30 -acodec copy /tmp/last.mp4

Cutting a scene from a video file

In mplayer , when file is being played, press "o" to see the current position in (time format).
use this information to know how long scene needs to be cut.


mencoder -ss 00:07:20 -endpos 190 -o /tmp/blue.avi -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 input_vdo_file.avi

Extracting DVD information using HandBrake

./HandBrakeCLI --scan  -i /Volumes/VDO_FOLDER

Extracting Chapter From a DVD dump on a disk using HandBrake

./HandBrakeCLI -i /Volumes/DVD_FOLDER/  -e x264  -q 20.0 -a 1,1 -E faac,copy:ac3 -B 320,160 -6 dpl2,none -R Auto,Auto -D 0.0,0.0 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback ffac3 -f mp4 -4 --decomb --loose-anamorphic --modulus 2 -m --x264-preset medium --h264-profile high --h264-level 4.1 -c 1 -o $HOME/chap1mod.mp4

Note:  “-c 1” in above command is extracting 1st chapter from DVD.

Python Regex Usage

Python Regex Usage

Perl Regex Usages

Multiline Matchings:

Inside a perl script:

#! /usr/bin/perl
$str = "This is a test
Will it work
lets see
";
if ($str =~ m!^This.+test$!)
{
     print "matched\n"
}

Following are the contents of file "data"

# cat data
This is a test file
Will it work
lets see

# As perl one liners:

$ cat data | perl -w -e '$/=""; while(<>) { m!^This.+file$!m and print }'
This is a test file
Will it work
lets see


# DEFAULT BEHAVIOUR
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH

# s modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ 


# m modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^let.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
lets see

# sm modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^lets.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ 

$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+it).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it


# BACKREFRENCES:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\1! and print "MATCHED\n"}'
MATCHED
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\g{-1}! and print "MATCHED\n"}'
MATCHED

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED

# CAPTURING and NON CAPTURING GROUPING:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "$1\n"}'
vis  

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?:vis).+\g{testmatch}! and print "$1\n"}'
Reference to nonexistent named group in regex; marked by <-- here="" in="" m="" testmatch="">vis).+\g{testmatch <-- -e="" -w="" 1.="" a="" at="" echo="" here="" line="" perl="" vis="" while="">) { m!(?:vis).+! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+! and print "$1\n"}'
vis
$

$ echo "datchet etet it underway" | perl -w -e '$/=""; while(<>) { m!(et){2}\s+! and print "MATCHED\n"}'
MATCHED
 
# NAMED CAPTURING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?test file)! and print "$+{fm}\n" or print "NOMATCH\n" }'
test file
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "$1" or print "NOMATCH\n" }'
Use of uninitialized value $1 in string at -e line 1, <> chunk 1.
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED 


# inside modifiers:
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+test! and print "MATCHED\n" or print "UNMATCHED\n"}'
UNMATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+TEST! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ 

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+((?i)test)! and print "$1\n"}'
TEST
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED

# inside modifiers (multiline modifier trick):
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s)(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!((?s)^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work

# inside modifiers non-capturing match:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED


# LOOK AHEAD ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(.+lets)(.+it.+$)!sm and print "$1 $3\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
NOMATCH


# LOOK BEHIND  ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(it)!sm and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(.+it)!sm and print "$1\n" or print "NOMATCH\n" }'
test file
Will it

# CONDITIONAL MATCHING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(asa)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'        # This is not a conditional matching
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "$1 $2\n" or print "NOMATCH\n" }'
Use of uninitialized value $2 in concatenation (.) or string at -e line 1, <> chunk 1.
test file 

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(anotherjunk))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH
 
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=junk)junk)!sm and print "MATCH\n" or print "NOMATCH\n" }'  # if condition is false then it simply skips
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk))!sm and print "MATCH\n" or print "NOMATCH\n" }'  # why is this working ?  no idea
MATCH

(?(?=condition)(then1|then2|then3)|(else1|else2|else3))

# Some Special cases to study:
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(ab)*! ; print "@arr\n";}'
ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((ab)*)! ; print "@arr\n";}'
ab ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)*)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)! ; print "@arr\n";}'
aba
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(?:a|b)*!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]*!g ; print "@arr\n";}'
aba  a 
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]*)!g ; print "@arr\n";}'
aba  a  
$  echo "abacaddb" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a   b  
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([ab]+)! ; print "$1\n";}'
aba
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([abc]+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((a|b|c)+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((?:a|b|c)+)! ; print "$1\n";}'
abaca


#======================================================

$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(a|b)+! ; print "@arr\n";}'
a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)+)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]+! ; print "@arr\n";}'
1
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]+)! ; print "@arr\n";}'
aba

Selective Backup with permissions preserved.

To copy selective files along with complete directory structure with all permissions preserved. (from one directory to another in same box).

find /var/log/allapps/apps1  | egrep '2013-12-02'  | rsync -Ravz --files-from=- /  /tmp/test

Above command will create a directory structure of var/log/allapps/apps1 + files(filtered by egrep above) inside apps1 dir inside /tmp/test along with all directories/files ownership and permissions intact.

Now take the backup of above copied folder.

cd /tmp/test
tar -cvpf test.tar var

Copy only specified files using rsync.
We need to copy only some files from remote system. These files are scattered around various directories. We also need to keep the directory structure on destination same as in source. Following command was used.

rsync --stats --progress -rvpogStDL --files-from=192.168.1.4:/tmp/200901files 192.168.1.4:/home/remote_files_and_documents 

Note1: /tmp/200901files contains list of files.
e.g.

file1
dir2/dir3/file2
dir6/file4

Note2: Source files are in local machine. But the list of files which are to be copied is being fetched from remote machine.

Remote Execution Tool using Screen

Remote Execution Tool

• No need to place root ssh keys
• No need to install any third party tool/library in advance.
• Keep record of every activity
• Simple and Plain

Often Sys Admins have to do similar kind of tasks on a number of machines. Each and everyone has its own way. Some uses configuration tools like puppet/cfengine. Some uses password less ssh parallel execution or someone might be using either expect or some modules like paramiko to achieve that.
This is also a similar kind of tool. It exploits the power of screen. What it requires to run.

• A central host having bash
• Same above central host having "screen" utility installed.

You do not even need your or root ssh keys in place. You will provide your password in passwd_file (see sample file in code).
What it can provide:

• logging on every host + on central host
• real time visual monitoring.
• can take actions on individual hosts as well.

Caveats:

• Only for private network.
• Knowing screen usage is advantageous.

HowtoRun

This tool consists of only one script that can be found at the bottom of this page. Following is the example run.

1.  Run without any argument.

2. Following Directories are created after running create_connection_configs.sh script.

3. Now create a file boxes. This file will contain the list of hostnames or IPAddresses of remote boxes. REMOTE_TASKS directory will contain directories that will have various tasks in form of script to be executed on remote hosts listed in boxes file. LOGS directory will contain logs of all activities that we carry out per host basis. 

4.  Now we will place local user ssh keys to remote machine. Note that this is for normal user only. Not root user. We assume that localuser is testuser  and testuser exists on each remote machine having same password.  Now create a file passwd_file and  type in testuser password in it followed by ctrl-v+ctrl-Enter 

COPY SSH KEYS to all remote hosts

5.  Run ./create_connection_configs.sh  SSHKEYCOPY command. Following output will be shown.

6 . Follow above screen instruction carefully. Open second terminal window and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -c mainscreenrc

7.  Keep Following first terminal window instructions. And run screen -r on second terminal. Following output follows. Pay attention to the bottom of screen (in red) also.

8.  At this point in second terminal run ctrl-a, ctrl+". Following output will follow. It will show two screen sessions. (one for each host listed in boxes file).

9. Keep following instructions from first terminal window, and open third terminal and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -X readbuf ./passwd_file

Above command will read password contained in passwd_file. Please note passwd_file contains password+<Enter> character.

Keep following instructions from first terminal window and now run, following command on third terminal window.

screen -S MasterSession -X at "#" stuff $'ssh-keyscan -t rsa \$TARGETHOST >> $HOME/.ssh/known_hosts ; ssh-copy-id -i $HOME/.ssh/id_rsa.pub \$TARGETHOST\n'

10. At this point switch to 2nd terminal to see what has happened.

11. As you see above, it is asking for testuser password so that it can copy ssh-keys.  Take a note of bottom of window as well (in red). It tells us that screen session is for 192.168.122.101. If we want to see screen session of 192.168.122.102. then press ctrl-a,ctrl-".   You will see following screens.

12. Keep following instructions in first terminal. Now switch to third terminal and run following command.

screen -S MasterSession -X at "#" paste "."

Above command will paste password on to above two screen sessions opened in 2nd terminal. 

You will see following output.

13. As the above output shows ssh keys have been copied. You can cross verify it.

14. Checkout the logs in LOGS directory. Everything is recorded here.

RUN Tasks on to Remote Machines.

15. Assuming testuser ssh-keys are in place. and testuser is allowed to run some sudo commands (Sudo with password.)

16. Create a task directory and a script that needs to be executed on remote hosts. Please note we have already populated boxes file. Following is the screenshot.

17. Now run following command.

./create_connection_configs.sh firsttask

Following is output.

18. As you see above script.sh has been copied to remote hosts.

19. Follow instruction on terminal window and open 2nd terminal now and run following commands.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -c mainscreenrc

screen -r

20. The moment you run screen -r , you will notice that you are already connected with remote host. To make sure you are connected with other remote hosts too, then press ctrl-a, ctrl-"  to list the screen sessions.  

21.  Follow on-screen instruction and open third terminal . and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -X readbuf ./passwd_file

screen -S MasterSession -X at "#" stuff $'sudo su -\n'

22.  Now switch back to terminal-2 to verify that you are being prompted for sudo password.

Use ctrl-a,ctrl-" to switch to another screen session to verify same.

23. Now switch back to terminal 3 and run following command.

screen -S MasterSession -X at "#" paste "."

Now switch back to terminal-2, you will see that you have logged in to both the machines.

24.  Follow instructions from terminal-1 window and run following command on third terminal.

screen -S MasterSession -X at "#" stuff $'cd /tmp/firsttask ; bash ./script.sh\n'

Switch back to 2nd terminal window to verify results.

25. Check LOGS directory for each action we took.

create_connection_configs.sh

LDAP Queries Examples

1) To determine about the Directory Server info and its capability
ldapsearch  -h ldapserver -Z -x -b '' -s base 'objectClass=*'

2) To search directory from base just one level down.
ldapsearch  -h ldapserver -Z -x -b 'dc=example,dc=net' -s onelevel

3) Pick desired dn from above output and traverse further
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net' -s onelevel

4) To search for a User whose only half name is known
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net' -s onelevel '(cn=Tes*)'
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net'  '(cn=Tes*)'

ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net' -s onelevel '(cn=Tes*)'  => will not yield any result
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net'  '(cn=Tes*)'   => this will yield result

5) to list only specific information (like common name)
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net'  '(uid=testu)' cn

6) To list sudo users and commands
ldapsearch  -h ldapserver -ZZ -x -b 'ou=SUDOers,dc=example,dc=net' -s one

7) To list all groups
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=com' 'objectClass=*roup*'

8) To list all groups in which "cn=Test User, ou=Users,dc=example, dc=com"  exists
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=com' '(&(objectClass=*roup*)(uniqueMember=cn=Test User,ou=Users,dc=example, dc=com))'

9) To list entries using Admin or Directory Manager
ldapsearch -L -b 'ou=Users, dc=example, dc=com' -x -D "cn=directory manager" -w 'Password'

10) Anonymous binding and listing everything
ldapsearch -h ldapserver "objectClass=*"
ldapsearch  -h ldapserver  -x -b 'dc=appauth,dc=example,dc=net'

11) Binded LDAP search
ldapsearch  -H ldaps://ldapserver -D 'uid=Manager,ou=App1,dc=appauth,dc=org'  -b 'dc=appauth,dc=org' -w 'Password' -s onelevel

ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -s onelevel

12) To list all top level trees with Binded authentication
ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -b "" -s base "objectclass=*"

ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -b 'dc=appauth,dc=org'

12) TO list against AD (Active Directory).
ldapsearch -H ldap://AD-Server -x -D 'domain\user' -W

Screen Sessions Management

I used this method when I need to login to multiple remote machines using ssh and then become root and then do things interactively simultaneously on them. 

Following steps were performed:

^M => denotes  "Enter" on keyboard. (ctrl-v Enter => ^M)

screen^M  => can also be run as screen -S  <session name>^M

ctrl-a c  => 2 times to create two screen windows in one screen session.

create a file e.g  /tmp/screen_bufferfile  => this file will work as a screen buffer file.

Note : screen default buffer register is called "."  ( dot )

scenario 1 :  Sending Identical commands to everywhere non-interactively.

step 1 = ctrl-a : readbuf /tmp/screen_bufferfile  => this will copy the contents of file in "." register

step 2 = ctrl-a : at "#" paste .   => this command will paste the contents of "." register on every screen window (even to sshed windows also)

scenario 2 : Sending commands interactively.

ctrl-a : bufferfile /tmp/screen_bufferfile  => sets the default buffer file

ctrl-a <        => reads the contents of buffer to "." register

ctrl-a ]         => paste the contents of buffer to current screen window only ( NOT any other )

even the ssh password and sudo password prompts accepts passwords like this.

scenario 3: Sending commands using screen from outside

Create a screen session and create multiple screen in it using ctrl-a c (multiple times)

Get the screen session id (or name) using screen -list

To send unix shell commands(or any input) to ALL windows in a screen session run following 2 commands:

screen -S 3461.ttys000.MacBook-Pro  -X readbuf /tmp/screen_bufferfile

screen -S 3461.ttys000.MacBook-Pro  -X at "#" paste "."

/tmp/screen_bufferfile will contains unix commands or (any input) you want to send to screen windows.

To send unix shell command( or any input) to a particular window run following.

screen -S 3461.ttys000.MacBook-Pro  -X readbuf /tmp/screen_bufferfile

screen -S 3461.ttys000.MacBook-Pro  -p 5 -X paste "." 

readbuf and paste are used in scenes when we have to input some quite a good amount of data. This also relieves from

taking care of quotes and control characters that we have anyways to do when we use "stuff" command. e.g.

Above can also be done by using following:

screen -S 3461.ttys000.MacBook-Pro -p 5 -X stuff "hostname^M"

screen -S 3461.ttys000.MacBook-Pro -X at "#" stuff "hostname^M"

Above we have used actual data on command line itself to send to screen windows. Please note that we have to explicitly used ^M characters to have Enter effect. But this is not the case when we create /tmp/screen_bufferfile.

screen -S javaupdate -X screen   => will spawn screen windows inside screen session (named javaupdate)

=====================================================

To create multiple windows in one screen session non-interactively

screen -d -m -S TEST   => creates a screen session with name TEST and detach it. Does not go into screen session.

screen -S TEST -X screen -t yahoo 03 ssh remotemachine  => creates a screen window inside TEST with name yahoo at window no 3. In window no 3 , there will an ssh session to remotemachine.

Create Fedora Installer ISO using GRUB

Introduction

This is just method to produce the Fedora Installer DVDs using the Grub boot loader instead of ISOLINUX.

Software Used

Base OS used, on which Fedora-13 installer was created was , Ubuntu 8.04.3 LTS (hardy). Fedora-13 will be used for which installer will be created using GRUB as boot loader.

Steps Performed
Copy the required files
Copy whole DVD contents

mkdir grubdvdrom
mount -o loop /mnt/ISO-IMAGES/Fedora/Fedora-13-i386-DVD.iso /tmp/fedora-13
rsync -avz /tmp/fedora-13/ grubdvdrom/
umount /tmp/fedora-13

Copy grub files from Base OS

mkdir grubdvdrom/boot
cp -r /boot/grub grubdvdrom/boot
cp grubdvdrom/isolinux/{vmlinuz,initrd.img} grubdvdrom/boot
cp /usr/lib/grub/x86_64-pc/stage2_eltorito grubdvdrom/boot/grub

Note: Fedora DVDROM contains some hidden files like .treeinfo, which are must in order to recognize local repos for anaconda installer. If these files are not copied then anaconda will go only to internet for repos. There we used rsync to make sure each file has been copied.Also you have to maintain the tree structure of boot/grub as a whole.

Create menu.lst file
create or modify grubdvdrom/boot/grub/menu.lst file with following contents.
title Fedora-13 Grub ISO Installer
kernel /boot/vmlinuz
initrd /boot/initrd.img

Create ISO

cd grubdvdrom
mkisofs -o Fedora-13.iso -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table -J -r -hide-rr-moved -V Fedora

Backing up and Restoring Windows using OpenSource Tools

Introduction:

This section show that how to create image of an installed Windows XP and restore it back using Linux utilities. Linux utilities used are 'ntfsclone' and 'dd'. ntfsclone was used as it creates images of ntfs partition efficiently copying only used blocks from disk. dd will copy entire HD image wasting a lot of space. We will create image over the network.

Preparation:

We had used a windows XP system.

A Liveubuntu CD (10.04)and Internet connection.

A destination machine which will recieve the Backup image over the network. SSH server must be installed and running on it.

 Backing Steps:

The windows system had single 75 GB partition. Boot the system from Live CD. if network has DHCP server machine will be on the network and may be able to connect to internet as well.

To work remotely , install openssh . This is optional not required.

sudo aptitude install openssh-server


After this optinal step perform following .
Backup NTFS partition to an Image and save it to remote location:

sudo ntfsclone --save-image --output - /dev/sda1 | gzip -c | ssh testuser@192.168.1.203 'cat - > /home/testuser/IMAGES/diskimage.img.gz'


Backup MBR and Partition Information:

First 446 bytes of MBR (sector 0) conatins boot code. We want to copy the boot code only. Next 64 Bytes contains the Partition table. We do not want to copy the partition table because destination disk may be larger in size and thus will have different C/H/S information. We will copy partition information based on sectors only. And in restoring process we will create partition table based on this sector information only. Partitions created with sectors are more accurate rather than cylinders. Thus making exact partitions on new disk with old information will be easier and much accurate. 

Copy the Boot code.


sudo dd if=/dev/sda bs=446 count=1 | ssh testuser@192.168.1.203 'dd of=/home/testuser/IMAGES/MBR446.img'


Copy the Partition information using secotrs as unit.

sudo sfdisk -d /dev/sda | ssh testuser@192.168.1.203 'cat - > /home/testuser/IMAGES/partitionInfo.txt'


Opitonally backup fdisk information:

sudo fdisk -u -l | ssh testuser@192.168.1.203 'cat - > /home/testuser/IMAGES/fdisk.txt'



In above commad -u gives sectors rather than cylinders in the output.
Restoring Steps:

Boot the system with LIVE CD. and bring it on the network. It will be by default if network is available and DHCP server is running in network.
Restore MBR:

Restore the Boot code:

ssh testuser@192.168.1.203 'dd if=/home/testuser/IMAGES/MBR446.img' | dd of=/dev/sdb


Create partitions exactly as it were present on old disk.

ssh testuser@192.168.1.203 'cat /home/testuser/IMAGES/partitionInfo.txt' | sfdisk /dev/sdb



Note: We had used the 160GB drive as the restore drive (destination drive). Original drive was 80GB. Both drives were connected to same computer. 80GB drive was /dev/sda and 160GB drive was /dev/sdb. 
Restore NTFS partition:

ssh testuser@192.168.1.203 'cat /home/testuser/IMAGES/diskimage.img.gz' | gunzip -c | ntfsclone --restore-image --overwrite /dev/sdb1 -


When the above step is finished, reboot the system with the second drive. (you may disconnect the first drive). If eveything goes fine it will not complain anything not even filesystem check. System may ask for the reboot after first login because it has detected new disk and have installed its driver. Reboot the system when this message appears. Thats it.