Find repeated Group pattern match. Case study of Ruby, Python and Go

Find repeated Group pattern match. Case study of Ruby, Python and Go

Sometimes I have to find the a group of patterns that are occurring in a log file. Normal grep -f <filename> does that job but it does not print the delimiter that can distinguish between repeated found patterns.

I needed something that can tell me that it has found a few patterns from a group in a sequence and last pattern has been found. Draw a delimiter here and start search for group again in remaining file.

I first wrote it in Ruby. I had huge files to parse. Ruby script was talking quite long to finish. As a case study I decided to write same utility using Python and Go.

Needless to say , Go was much faster.

Following are the codes.

Following are the sample data files.

pattern file :

hotmail
yahoo
google
gmail

Subject file:

This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Another one is gmail
Now is google
Another one is gmail
But this is going to be a yahoo
This is hotmail test
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR

Following is the output:

$ go run atest.go pattern subjectfile
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========
This is hotmail test
But this is going to be a yahoo
Another one is gmail
===========
Now is google
Another one is gmail
===========
But this is going to be a yahoo
This is hotmail test
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========

Launch AWS EC2 nstance using SaltStack

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli
pip install M2Crypto
pip install pyzmq

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Become root on MacBook now:

mkdir /etc/salt
touch /etc/salt/cloud.profiles
touch /etc/salt/cloud.providers

cat /etc/salt/cloud.profiles
base_ec2_private:
     provider: amazon_ireland_region
     image: ami-9d23aeea

cat /etc/salt/cloud.providers
amazon_ireland_region:
     id: ABCDEFGHIJKLMNO235M
     key: aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv
     keyname: MyEC2Key
     private_key: /Users/MacUser/EC2/MyEC2Key.pem
     location: eu-west-1
     availability_zone: eu-west-1a
     securitygroup: MySecurityGroupSSHOnly
     size: t2.micro
     del_root_vol_on_destroy: True
     ssh_username: ec2-user
     rename_on_destroy: True
     ssh_interface: public_ips
     provider: ec2

Launch Instance now:

source ~MacUser/PythonVirtENVs/SaltStack/bin/activate
salt-cloud --profile=base_ec2_private First_Instance

Launch AWS EC2 instance using awscli

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Following command will spin a FREE TIER instance in AWS cloud

aws ec2 run-instances --image-id ami-9d23aeea --key-name MyEC2Key --instance-type t2.micro --count 1 --security-groups MySecurityGroupSSHOnly

Playing and Encoding Video files on Linux

DVD tracks and Chapters info

mplayer -identify dvd:// -dvd-device /media/DVD_FOLDER/

MPlayer 29092-4.4.0 (C) 2000-2009 MPlayer Team
mplayer: could not connect to socket
mplayer: No such file or directory
Failed to open LIRC support. You will not be able to use your remote control.

Playing dvd://.
libdvdread: Encrypted DVD support unavailable.
libdvdread: Couldn't find device name.
ID_DVD_TITLES=2
ID_DVD_TITLE_1_CHAPTERS=51
ID_DVD_TITLE_1_ANGLES=1
ID_DVD_TITLE_2_CHAPTERS=50
ID_DVD_TITLE_2_ANGLES=1
ID_DVD_TITLE_1_LENGTH=13653.600
ID_DVD_TITLE_2_LENGTH=13150.166

PLAYING OPTIONS:
If the DVD video is downloaded on the harddisk, following are the various options to run videos from that DVD folder.

1) To open the DVD folder as a DVD. It will present all the menus of DVD. (note the dvdnav option)

mplayer -mouse-movements dvdnav:// -dvd-device /media/DVD_FOLDER/

2) To run the only chapter 2 from the DVD

mplayer dvd:// -chapter 2-2 -dvd-device /media/DVD_FOLDER/

3) To run the only chapter 5 to 7 from the DVD

mplayer dvd:// -chapter 5-7 -dvd-device /media/DVD_FOLDER/

4) To run 1st track of DVD

mplayer dvd://1 -dvd-device /media/DVD_FOLDER/

5) To view only the specific portion of a video file (based on time slice)

mplayer -ss 00:00:30 -endpos 30 dvdnav://1 -dvd-device /media/DVD_FOLDER/

CONVERSION OPTIONS:

mencoder dvd:// -chapter 2-2 -vf scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af-add lavcresample=44100 -vf-add harddup -vf-add scale=320:240 -o /tmp/test.mp4 -oac faac -faacopts object=2 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25 -of lavf -lavfopts format=mp4 -dvd-device /media/DVD_FOLDER/


mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -vf scale=320:240 -o yahoo.mp4 -mc 0 -oac mp3lame -lameopts vbr=3:br=128:q=9 -ovc lavc -lavcopts aglobal=1:vglobal=1:vcodec=mpeg4:vbitrate=512:keyint=25:vpass=1 -of lavf -dvd-device /media/DVD_FOLDER/

mencoder dvd:// -chapter 2-2 -af lavcresample=44100 -srate 44100 -ovc frameno -oac mp3lame -lameopts vbr=0:br=96 -o frame.mp3 -of lavf -lavfopts format=mp3 -dvd-device /media/DVD_FOLDER/

Extracting RAW audio AUDIO from a Video File (in PCM format i.e. wav)

mplayer /media/Video.avi -vo null -vc null -ao pcm:fast:file=/tmp/audio.wav

Conversion using ffmpeg

/opt/ffmpeg/bin/ffmpeg -i /media/video.avi -i /tmp/audio.mp3 -map 0:0 -map 1:0 -vcodec msmpeg4v2 -s qvga -b 512k -r 30 -acodec copy /tmp/last.mp4

Cutting a scene from a video file

In mplayer , when file is being played, press "o" to see the current position in (time format).
use this information to know how long scene needs to be cut.


mencoder -ss 00:07:20 -endpos 190 -o /tmp/blue.avi -oac faac -faacopts object=2 -ovc lavc -lavcopts vcodec=mpeg4 -of lavf -lavfopts format=mp4 input_vdo_file.avi

Extracting DVD information using HandBrake

./HandBrakeCLI --scan  -i /Volumes/VDO_FOLDER

Extracting Chapter From a DVD dump on a disk using HandBrake

./HandBrakeCLI -i /Volumes/DVD_FOLDER/  -e x264  -q 20.0 -a 1,1 -E faac,copy:ac3 -B 320,160 -6 dpl2,none -R Auto,Auto -D 0.0,0.0 --audio-copy-mask aac,ac3,dtshd,dts,mp3 --audio-fallback ffac3 -f mp4 -4 --decomb --loose-anamorphic --modulus 2 -m --x264-preset medium --h264-profile high --h264-level 4.1 -c 1 -o $HOME/chap1mod.mp4

Note:  “-c 1” in above command is extracting 1st chapter from DVD.

Python Regex Usage

Python Regex Usage

Perl Regex Usages

Multiline Matchings:

Inside a perl script:

#! /usr/bin/perl
$str = "This is a test
Will it work
lets see
";
if ($str =~ m!^This.+test$!)
{
     print "matched\n"
}

Following are the contents of file "data"

# cat data
This is a test file
Will it work
lets see

# As perl one liners:

$ cat data | perl -w -e '$/=""; while(<>) { m!^This.+file$!m and print }'
This is a test file
Will it work
lets see


# DEFAULT BEHAVIOUR
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH

# s modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ 


# m modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^let.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
lets see

# sm modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^lets.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ 

$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+it).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it


# BACKREFRENCES:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\1! and print "MATCHED\n"}'
MATCHED
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\g{-1}! and print "MATCHED\n"}'
MATCHED

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED

# CAPTURING and NON CAPTURING GROUPING:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "$1\n"}'
vis  

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?:vis).+\g{testmatch}! and print "$1\n"}'
Reference to nonexistent named group in regex; marked by <-- here="" in="" m="" testmatch="">vis).+\g{testmatch <-- -e="" -w="" 1.="" a="" at="" echo="" here="" line="" perl="" vis="" while="">) { m!(?:vis).+! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+! and print "$1\n"}'
vis
$

$ echo "datchet etet it underway" | perl -w -e '$/=""; while(<>) { m!(et){2}\s+! and print "MATCHED\n"}'
MATCHED
 
# NAMED CAPTURING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?test file)! and print "$+{fm}\n" or print "NOMATCH\n" }'
test file
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "$1" or print "NOMATCH\n" }'
Use of uninitialized value $1 in string at -e line 1, <> chunk 1.
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED 


# inside modifiers:
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+test! and print "MATCHED\n" or print "UNMATCHED\n"}'
UNMATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+TEST! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ 

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+((?i)test)! and print "$1\n"}'
TEST
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED

# inside modifiers (multiline modifier trick):
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s)(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!((?s)^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work

# inside modifiers non-capturing match:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED


# LOOK AHEAD ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(.+lets)(.+it.+$)!sm and print "$1 $3\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
NOMATCH


# LOOK BEHIND  ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(it)!sm and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(.+it)!sm and print "$1\n" or print "NOMATCH\n" }'
test file
Will it

# CONDITIONAL MATCHING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(asa)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'        # This is not a conditional matching
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "$1 $2\n" or print "NOMATCH\n" }'
Use of uninitialized value $2 in concatenation (.) or string at -e line 1, <> chunk 1.
test file 

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(anotherjunk))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH
 
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=junk)junk)!sm and print "MATCH\n" or print "NOMATCH\n" }'  # if condition is false then it simply skips
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk))!sm and print "MATCH\n" or print "NOMATCH\n" }'  # why is this working ?  no idea
MATCH

(?(?=condition)(then1|then2|then3)|(else1|else2|else3))

# Some Special cases to study:
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(ab)*! ; print "@arr\n";}'
ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((ab)*)! ; print "@arr\n";}'
ab ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)*)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)! ; print "@arr\n";}'
aba
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(?:a|b)*!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]*!g ; print "@arr\n";}'
aba  a 
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]*)!g ; print "@arr\n";}'
aba  a  
$  echo "abacaddb" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a   b  
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([ab]+)! ; print "$1\n";}'
aba
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([abc]+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((a|b|c)+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((?:a|b|c)+)! ; print "$1\n";}'
abaca


#======================================================

$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(a|b)+! ; print "@arr\n";}'
a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)+)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]+! ; print "@arr\n";}'
1
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]+)! ; print "@arr\n";}'
aba

Selective Backup with permissions preserved.

To copy selective files along with complete directory structure with all permissions preserved. (from one directory to another in same box).

find /var/log/allapps/apps1  | egrep '2013-12-02'  | rsync -Ravz --files-from=- /  /tmp/test

Above command will create a directory structure of var/log/allapps/apps1 + files(filtered by egrep above) inside apps1 dir inside /tmp/test along with all directories/files ownership and permissions intact.

Now take the backup of above copied folder.

cd /tmp/test
tar -cvpf test.tar var

Copy only specified files using rsync.
We need to copy only some files from remote system. These files are scattered around various directories. We also need to keep the directory structure on destination same as in source. Following command was used.

rsync --stats --progress -rvpogStDL --files-from=192.168.1.4:/tmp/200901files 192.168.1.4:/home/remote_files_and_documents 

Note1: /tmp/200901files contains list of files.
e.g.

file1
dir2/dir3/file2
dir6/file4

Note2: Source files are in local machine. But the list of files which are to be copied is being fetched from remote machine.

Remote Execution Tool using Screen

Remote Execution Tool

• No need to place root ssh keys
• No need to install any third party tool/library in advance.
• Keep record of every activity
• Simple and Plain

Often Sys Admins have to do similar kind of tasks on a number of machines. Each and everyone has its own way. Some uses configuration tools like puppet/cfengine. Some uses password less ssh parallel execution or someone might be using either expect or some modules like paramiko to achieve that.
This is also a similar kind of tool. It exploits the power of screen. What it requires to run.

• A central host having bash
• Same above central host having "screen" utility installed.

You do not even need your or root ssh keys in place. You will provide your password in passwd_file (see sample file in code).
What it can provide:

• logging on every host + on central host
• real time visual monitoring.
• can take actions on individual hosts as well.

Caveats:

• Only for private network.
• Knowing screen usage is advantageous.

HowtoRun

This tool consists of only one script that can be found at the bottom of this page. Following is the example run.

1.  Run without any argument.

2. Following Directories are created after running create_connection_configs.sh script.

3. Now create a file boxes. This file will contain the list of hostnames or IPAddresses of remote boxes. REMOTE_TASKS directory will contain directories that will have various tasks in form of script to be executed on remote hosts listed in boxes file. LOGS directory will contain logs of all activities that we carry out per host basis. 

4.  Now we will place local user ssh keys to remote machine. Note that this is for normal user only. Not root user. We assume that localuser is testuser  and testuser exists on each remote machine having same password.  Now create a file passwd_file and  type in testuser password in it followed by ctrl-v+ctrl-Enter 

COPY SSH KEYS to all remote hosts

5.  Run ./create_connection_configs.sh  SSHKEYCOPY command. Following output will be shown.

6 . Follow above screen instruction carefully. Open second terminal window and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -c mainscreenrc

7.  Keep Following first terminal window instructions. And run screen -r on second terminal. Following output follows. Pay attention to the bottom of screen (in red) also.

8.  At this point in second terminal run ctrl-a, ctrl+". Following output will follow. It will show two screen sessions. (one for each host listed in boxes file).

9. Keep following instructions from first terminal window, and open third terminal and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -X readbuf ./passwd_file

Above command will read password contained in passwd_file. Please note passwd_file contains password+<Enter> character.

Keep following instructions from first terminal window and now run, following command on third terminal window.

screen -S MasterSession -X at "#" stuff $'ssh-keyscan -t rsa \$TARGETHOST >> $HOME/.ssh/known_hosts ; ssh-copy-id -i $HOME/.ssh/id_rsa.pub \$TARGETHOST\n'

10. At this point switch to 2nd terminal to see what has happened.

11. As you see above, it is asking for testuser password so that it can copy ssh-keys.  Take a note of bottom of window as well (in red). It tells us that screen session is for 192.168.122.101. If we want to see screen session of 192.168.122.102. then press ctrl-a,ctrl-".   You will see following screens.

12. Keep following instructions in first terminal. Now switch to third terminal and run following command.

screen -S MasterSession -X at "#" paste "."

Above command will paste password on to above two screen sessions opened in 2nd terminal. 

You will see following output.

13. As the above output shows ssh keys have been copied. You can cross verify it.

14. Checkout the logs in LOGS directory. Everything is recorded here.

RUN Tasks on to Remote Machines.

15. Assuming testuser ssh-keys are in place. and testuser is allowed to run some sudo commands (Sudo with password.)

16. Create a task directory and a script that needs to be executed on remote hosts. Please note we have already populated boxes file. Following is the screenshot.

17. Now run following command.

./create_connection_configs.sh firsttask

Following is output.

18. As you see above script.sh has been copied to remote hosts.

19. Follow instruction on terminal window and open 2nd terminal now and run following commands.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -c mainscreenrc

screen -r

20. The moment you run screen -r , you will notice that you are already connected with remote host. To make sure you are connected with other remote hosts too, then press ctrl-a, ctrl-"  to list the screen sessions.  

21.  Follow on-screen instruction and open third terminal . and run following command.

cd /localhome/testuser/REMOTE_TOOL ; screen -S MasterSession -X readbuf ./passwd_file

screen -S MasterSession -X at "#" stuff $'sudo su -\n'

22.  Now switch back to terminal-2 to verify that you are being prompted for sudo password.

Use ctrl-a,ctrl-" to switch to another screen session to verify same.

23. Now switch back to terminal 3 and run following command.

screen -S MasterSession -X at "#" paste "."

Now switch back to terminal-2, you will see that you have logged in to both the machines.

24.  Follow instructions from terminal-1 window and run following command on third terminal.

screen -S MasterSession -X at "#" stuff $'cd /tmp/firsttask ; bash ./script.sh\n'

Switch back to 2nd terminal window to verify results.

25. Check LOGS directory for each action we took.

create_connection_configs.sh

LDAP Queries Examples

1) To determine about the Directory Server info and its capability
ldapsearch  -h ldapserver -Z -x -b '' -s base 'objectClass=*'

2) To search directory from base just one level down.
ldapsearch  -h ldapserver -Z -x -b 'dc=example,dc=net' -s onelevel

3) Pick desired dn from above output and traverse further
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net' -s onelevel

4) To search for a User whose only half name is known
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net' -s onelevel '(cn=Tes*)'
ldapsearch  -h ldapserver -ZZ -x -b 'ou=Users,dc=example,dc=net'  '(cn=Tes*)'

ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net' -s onelevel '(cn=Tes*)'  => will not yield any result
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net'  '(cn=Tes*)'   => this will yield result

5) to list only specific information (like common name)
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=net'  '(uid=testu)' cn

6) To list sudo users and commands
ldapsearch  -h ldapserver -ZZ -x -b 'ou=SUDOers,dc=example,dc=net' -s one

7) To list all groups
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=com' 'objectClass=*roup*'

8) To list all groups in which "cn=Test User, ou=Users,dc=example, dc=com"  exists
ldapsearch  -h ldapserver -ZZ -x -b 'dc=example,dc=com' '(&(objectClass=*roup*)(uniqueMember=cn=Test User,ou=Users,dc=example, dc=com))'

9) To list entries using Admin or Directory Manager
ldapsearch -L -b 'ou=Users, dc=example, dc=com' -x -D "cn=directory manager" -w 'Password'

10) Anonymous binding and listing everything
ldapsearch -h ldapserver "objectClass=*"
ldapsearch  -h ldapserver  -x -b 'dc=appauth,dc=example,dc=net'

11) Binded LDAP search
ldapsearch  -H ldaps://ldapserver -D 'uid=Manager,ou=App1,dc=appauth,dc=org'  -b 'dc=appauth,dc=org' -w 'Password' -s onelevel

ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -s onelevel

12) To list all top level trees with Binded authentication
ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -b "" -s base "objectclass=*"

ldapsearch  -H ldaps://ldapserver -D 'cn=Test User,ou=Users,dc=example,dc=com'  -w 'Password' -b 'dc=appauth,dc=org'

12) TO list against AD (Active Directory).
ldapsearch -H ldap://AD-Server -x -D 'domain\user' -W

Screen Sessions Management

I used this method when I need to login to multiple remote machines using ssh and then become root and then do things interactively simultaneously on them. 

Following steps were performed:

^M => denotes  "Enter" on keyboard. (ctrl-v Enter => ^M)

screen^M  => can also be run as screen -S  <session name>^M

ctrl-a c  => 2 times to create two screen windows in one screen session.

create a file e.g  /tmp/screen_bufferfile  => this file will work as a screen buffer file.

Note : screen default buffer register is called "."  ( dot )

scenario 1 :  Sending Identical commands to everywhere non-interactively.

step 1 = ctrl-a : readbuf /tmp/screen_bufferfile  => this will copy the contents of file in "." register

step 2 = ctrl-a : at "#" paste .   => this command will paste the contents of "." register on every screen window (even to sshed windows also)

scenario 2 : Sending commands interactively.

ctrl-a : bufferfile /tmp/screen_bufferfile  => sets the default buffer file

ctrl-a <        => reads the contents of buffer to "." register

ctrl-a ]         => paste the contents of buffer to current screen window only ( NOT any other )

even the ssh password and sudo password prompts accepts passwords like this.

scenario 3: Sending commands using screen from outside

Create a screen session and create multiple screen in it using ctrl-a c (multiple times)

Get the screen session id (or name) using screen -list

To send unix shell commands(or any input) to ALL windows in a screen session run following 2 commands:

screen -S 3461.ttys000.MacBook-Pro  -X readbuf /tmp/screen_bufferfile

screen -S 3461.ttys000.MacBook-Pro  -X at "#" paste "."

/tmp/screen_bufferfile will contains unix commands or (any input) you want to send to screen windows.

To send unix shell command( or any input) to a particular window run following.

screen -S 3461.ttys000.MacBook-Pro  -X readbuf /tmp/screen_bufferfile

screen -S 3461.ttys000.MacBook-Pro  -p 5 -X paste "." 

readbuf and paste are used in scenes when we have to input some quite a good amount of data. This also relieves from

taking care of quotes and control characters that we have anyways to do when we use "stuff" command. e.g.

Above can also be done by using following:

screen -S 3461.ttys000.MacBook-Pro -p 5 -X stuff "hostname^M"

screen -S 3461.ttys000.MacBook-Pro -X at "#" stuff "hostname^M"

Above we have used actual data on command line itself to send to screen windows. Please note that we have to explicitly used ^M characters to have Enter effect. But this is not the case when we create /tmp/screen_bufferfile.

screen -S javaupdate -X screen   => will spawn screen windows inside screen session (named javaupdate)

=====================================================

To create multiple windows in one screen session non-interactively

screen -d -m -S TEST   => creates a screen session with name TEST and detach it. Does not go into screen session.

screen -S TEST -X screen -t yahoo 03 ssh remotemachine  => creates a screen window inside TEST with name yahoo at window no 3. In window no 3 , there will an ssh session to remotemachine.

Create Fedora Installer ISO using GRUB

Introduction

This is just method to produce the Fedora Installer DVDs using the Grub boot loader instead of ISOLINUX.

Software Used

Base OS used, on which Fedora-13 installer was created was , Ubuntu 8.04.3 LTS (hardy). Fedora-13 will be used for which installer will be created using GRUB as boot loader.

Steps Performed
Copy the required files
Copy whole DVD contents

mkdir grubdvdrom
mount -o loop /mnt/ISO-IMAGES/Fedora/Fedora-13-i386-DVD.iso /tmp/fedora-13
rsync -avz /tmp/fedora-13/ grubdvdrom/
umount /tmp/fedora-13

Copy grub files from Base OS

mkdir grubdvdrom/boot
cp -r /boot/grub grubdvdrom/boot
cp grubdvdrom/isolinux/{vmlinuz,initrd.img} grubdvdrom/boot
cp /usr/lib/grub/x86_64-pc/stage2_eltorito grubdvdrom/boot/grub

Note: Fedora DVDROM contains some hidden files like .treeinfo, which are must in order to recognize local repos for anaconda installer. If these files are not copied then anaconda will go only to internet for repos. There we used rsync to make sure each file has been copied.Also you have to maintain the tree structure of boot/grub as a whole.

Create menu.lst file
create or modify grubdvdrom/boot/grub/menu.lst file with following contents.
title Fedora-13 Grub ISO Installer
kernel /boot/vmlinuz
initrd /boot/initrd.img

Create ISO

cd grubdvdrom
mkisofs -o Fedora-13.iso -b boot/grub/stage2_eltorito -no-emul-boot -boot-load-size 4 -boot-info-table -J -r -hide-rr-moved -V Fedora

Backing up and Restoring Windows using OpenSource Tools

Introduction:

This section show that how to create image of an installed Windows XP and restore it back using Linux utilities. Linux utilities used are 'ntfsclone' and 'dd'. ntfsclone was used as it creates images of ntfs partition efficiently copying only used blocks from disk. dd will copy entire HD image wasting a lot of space. We will create image over the network.

Preparation:

We had used a windows XP system.

A Liveubuntu CD (10.04)and Internet connection.

A destination machine which will recieve the Backup image over the network. SSH server must be installed and running on it.

 Backing Steps:

The windows system had single 75 GB partition. Boot the system from Live CD. if network has DHCP server machine will be on the network and may be able to connect to internet as well.

To work remotely , install openssh . This is optional not required.

sudo aptitude install openssh-server


After this optinal step perform following .
Backup NTFS partition to an Image and save it to remote location:

sudo ntfsclone --save-image --output - /dev/sda1 | gzip -c | ssh testuser@192.168.1.203 'cat - > /home/testuser/IMAGES/diskimage.img.gz'


Backup MBR and Partition Information:

First 446 bytes of MBR (sector 0) conatins boot code. We want to copy the boot code only. Next 64 Bytes contains the Partition table. We do not want to copy the partition table because destination disk may be larger in size and thus will have different C/H/S information. We will copy partition information based on sectors only. And in restoring process we will create partition table based on this sector information only. Partitions created with sectors are more accurate rather than cylinders. Thus making exact partitions on new disk with old information will be easier and much accurate. 

Copy the Boot code.


sudo dd if=/dev/sda bs=446 count=1 | ssh testuser@192.168.1.203 'dd of=/home/testuser/IMAGES/MBR446.img'


Copy the Partition information using secotrs as unit.

sudo sfdisk -d /dev/sda | ssh testuser@192.168.1.203 'cat - > /home/testuser/IMAGES/partitionInfo.txt'


Opitonally backup fdisk information:

sudo fdisk -u -l | ssh testuser@192.168.1.203 'cat - > /home/testuser/IMAGES/fdisk.txt'



In above commad -u gives sectors rather than cylinders in the output.
Restoring Steps:

Boot the system with LIVE CD. and bring it on the network. It will be by default if network is available and DHCP server is running in network.
Restore MBR:

Restore the Boot code:

ssh testuser@192.168.1.203 'dd if=/home/testuser/IMAGES/MBR446.img' | dd of=/dev/sdb


Create partitions exactly as it were present on old disk.

ssh testuser@192.168.1.203 'cat /home/testuser/IMAGES/partitionInfo.txt' | sfdisk /dev/sdb



Note: We had used the 160GB drive as the restore drive (destination drive). Original drive was 80GB. Both drives were connected to same computer. 80GB drive was /dev/sda and 160GB drive was /dev/sdb. 
Restore NTFS partition:

ssh testuser@192.168.1.203 'cat /home/testuser/IMAGES/diskimage.img.gz' | gunzip -c | ntfsclone --restore-image --overwrite /dev/sdb1 -


When the above step is finished, reboot the system with the second drive. (you may disconnect the first drive). If eveything goes fine it will not complain anything not even filesystem check. System may ask for the reboot after first login because it has detected new disk and have installed its driver. Reboot the system when this message appears. Thats it.

Creating Debian USB Installer

Prerequisites:

An already insalled and working linux distribution. My laptop already had Fedora Core 10.

Target OS

Debian Lenny 5.0.1 was used for this purpose. It can be downloaded fromhttp://cdimage.debian.org/debian-cd/5.0.1/i386/bt-dvd/

We also need a different installer kernel and initrd image as we will be installing from USB. In this case hd-image installers will be used. vmlinuz and initrd for hd-image installes can be downloaded from http://http.us.debian.org/debian/dists/lenny/main/installer-i386/current/images/hd-media/. It is very important to have correct installer image for a distribution.

Preparing USB disk:

We will create an ext2 partition with fdisk command. USB disk device name can be found from dmesg command output. Create a single partition on USB disk and make it bootable. All these activities are performed using fdisk commands. So if USB disk is called as /dev/sdb , it will have one partition as /dev/sdb1

Copy the required grub files to USB stick:

mount /dev/sdb1 /mnt
mkdir -p /mnt/boot/grub
cp /boot/grub/* /mnt/boot/grub

Copy the downloaded hd-media installer image and iso files to USB stick:

cp /home/test/Download/{vmlinuz,initrd.gz} /mnt
cp /home/test/debian-501-i386-DVD-1.iso /mnt

Configure and Install Grub:
Edit /mnt/boot/grub/menu.lst to have following contents only.

title USB Installer
root (hd0,0)
kernel /vmlinuz rootdelay=10
initrd /initrd.gz

It is important to have rootdelay option other wise installer will not kick off. This is required because USB flash drives take a bit longer to be prepared.

Now Install the grub bootloader to USB disk.

# grub
grub> device (hd0) /dev/sdb
grub> root (hd0,0)
grub> setup (hd0)

Thats it. Boot the system from USB disk.

Creating Fedora11 USB Installer With Grub

Introduction:

As the new OSes are coming in the market, to test and install these OSes requires DVDs to be created. An efficient method was required, so that DVD burning can be stopped. Since my Laptop can be booted from USB device , so It makes much sense to use USB devices (i.e. USB stick or USB drive) to do the installation. For this purpose and 8GB USB disk was used.

Prerequisites:

An already insalled and working linux distribution. My laptop already had Fedora Core 10.

Target OS.

Fedora-11 was used for this purpose. It can be downloaded from http://download.fedoraproject.org/pub/fedora/linux/releases/11/Fedora/i386/iso/Fedora-11-i386-DVD.iso

In case of Debian we had used different installer kernel and initrd image which had to be downloaded. But in case of Fedora-11 we will use the same Installer and initrd image which is available in its ISO file. For more information on Debian USB installer kernel see Preparing USB disk:

We wil create an ext2 partition with fdisk command. USB disk device name can be found from dmesg command output. Create a single partition on USB disk and make it bootable. All these activities are performed using fdisk commands. So if USB disk is called as /dev/sdb , it will have one partition as /dev/sdb1

Copy the required grub files to USB stick:

mount /dev/sdb1 /mnt
mkdir -p /mnt/boot/grub
cp /boot/grub/* /mnt/boot/grub

Copy the installer image and iso files to USB stick:

mount -o loop Fedora-11-i386-DVD.iso /tmp/image
cp /tmp/image/isolinux/vmlinuz /mnt/vmlinuz-fedora-11
cp /tmp/image/isolinux/initrd.img /mnt/initrd.img-fedora-11
cp -pr /tmp/image/images /mnt
cp Fedora-11-i386-DVD.iso /mnt

Note: vmlinuz and initrd.img files are available at two locations in ISO image. First location is isolinux/ and second is images/pxeboot/ . But kernel and initrd.img are exactly same. So it doesn't matter what files are copied. Also there is additional step of copy in Fedora-11 than in Debian.

Configure and Install Grub:
Edit /mnt/boot/grub/menu.lst to have following contents (or grub.conf).

title USB Fedora-11 Installer
root (hd0,0)
kernel /vmlinuz-fedora-11 rootdelay=10
initrd /initrd.img-fedora-11

It is important to have rootdelay option other wise installer will not kick off. This is required because USB flash drives take a bit longer to be prepared.
Now Install the grub bootloader to USB disk.

# grub
grub> device (hd0) /dev/sdb
grub> root (hd0,0)
grub> setup (hd0)

Thats it. Boot the system from USB disk.

OS Remastering CentOS 5.5 using Kickstart

Introduction

This document will describe How to Create a Linux LiveCD using kickstart method. This is the very fast and convenient method of producing LiveCDs. This method can be used only on RedHat based distribution only e.g. Fedora, CentOS. The Method of producing LiveCDs based on Debian distribution is different and is not described here. A manual procedure of modifying a LiveCD is described in the following link.
OS Remastering CentOS 5.5 using Manual Method

Objective

We will be creating a highly customized CentOS 5.5 Based LiveCD which will have some external third party rpm packages as well.

Software Requirement

1) CentOS 5.5 DVD
This can be downloaded from below link
http://mirror.as29550.net/mirror.centos.org/5.5/isos/i386/CentOS-5.5-i386-bin-DVD.iso
2) Third Party RPM package which is to be installed in Live CD. In this case we are using NTFSProgs package which is not available in CentOS repositories.
http://packages.sw.be/ntfsprogs/ntfsprogs-1.13.1-1.el5.rf.i386.rpm

Procedure Overview

First we will install a regular CentOS 5.5 on a system. This can be done on a VM as well. In this installed machine we will build our LiveCD.

Install the Base System

We installed the CentOS 5.5 using CentOS 5.5 DVD in a VM. We used KVM for this purpose.

First create the Virtual Harddisk using following command. and then run VM and finish OS installation

qemu-img create -f qcow2 /VIRTUAL/QEmu/CentOS/CentOS-5.5-i386.img 10G

kvm -smp 4 -m 2048 -hda /VIRTUAL/QEmu/CentOS/CentOS-5.5-i386.img -cdrom /DATA/Install/ISO/CentOS/CentOS-5.5-i386-bin-DVD.iso -usb -usbdevice tablet -vnc :40 -net nic,macaddr=00:00:00:00:00:01 -net tap,script=

I installed the system with minimum utilities. Only Base and Administration Utilities.

Configure the Base system.
Add a repository for LiveCD tools.
Create a file /etc/yum.repos.d/CentOS-LiveCD.repo with following contents.

# Name: CentOS LiveCD repository
[livecd]
name = CentOS $releasever - LiveCD
baseurl = http://www.nanotechnologies.qc.ca/propos/linux/centos-live/$basearch/live
enabled=1
protect=0
gpgkey = http://www.nanotechnologies.qc.ca/propos/linux/RPM-GPG-KEY-PGuay

Install LiveCD tools.
Run the following command to install the LiveCD tools.

yum install livecd-tools syslinux

This is our full blown running OS on a harddisk. We will be creating the LiveCD in this system. All the following commands henceforth will be carried out in this machine only. Mount cdrom
Run the following command to mount the cdrom.

mkdir /mnt/cdrom
mount /dev/cdrom /mnt/cdrom

Prepare the directory containing third party tools.

mkdir /root/livecdkickstart_files

Place the files in this above directory which you want to include in your customized LiveCD. In this case we had included following files.

BackupRestore.sh 
command.txt 
isolinux.cfg 
kick.ks 
ntfsprogs-1.13.1-1.el5.rf.i386.rpm 
pcbackup_keys 
README 
root_ssh_config 
ssh 
WindowsPCBackup.sh

In the above listed files kick.ks is the kickstart file which will contain all the instructions for the customization of LIveCD. kick.ks is given at the end of this article.

Following are the contents of isolinux.cfg file

default menu.c32
timeout 100
menu title Welcome to BackupRestore CD ver 1.0 !
menu color border 0 #ffffffff #00000000
menu color sel 7 #ffffffff #ff000000
menu color title 0 #ffffffff #00000000
menu color tabmsg 0 #ffffffff #00000000
menu color unsel 0 #ffffffff #00000000
menu color hotsel 0 #ff000000 #ffffffff
menu color hotkey 7 #ffffffff #ff000000
menu color timeout_msg 0 #ffffffff #00000000
menu color timeout 0 #ffffffff #00000000
menu color cmdline 0 #ffffffff #00000000
menu hidden
menu hiddenrow 5
label linux0
menu label Backup This System
kernel vmlinuz0
append initrd=initrd0.img root=CDLABEL=LiveBackupRestoreCD rootfstype=iso9660 ro liveimg mode=Backup vga=0x314
label linux1
menu label Restore This System
kernel vmlinuz0
append initrd=initrd0.img root=CDLABEL=LiveBackupRestoreCD rootfstype=iso9660 ro liveimg mode=Restore vga=0x314

Create the LiveCD
Once you have required files and kickstart file run the following command to create the LiveCD.

cd /root/livecdkickstart_files
livecd-creator --config=kick.ks --fslabel=LiveBackupRestoreCD

kick.ks file

#platform=x86, AMD64, or Intel EM64T
#version=DEVEL
# Firewall configuration
firewall --disabled
# Install OS instead of upgrade
install
# Use hard drive installation media
#harddrive --dir=/tmp/fedora --partition=sda8
repo --name=cdrom --baseurl=file:///tmp/fedora
# To compose against the current release tree, use the following "repo" (enabled by default)
repo --name=released --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-13&arch=$basearch
# To include updates, use the following "repo" (enabled by default)
repo --name=updates --mirrorlist=http://mirrors.fedoraproject.org/mirrorlist?repo=updates-released-f13&arch=$basearch

# Root password
rootpw --iscrypted $1$x9PSyNru$RR5iKsKjlRpaMBgGrg3a81
# System authorization information
auth  --useshadow  --passalgo=md5
# Use text mode install
text
# System keyboard
keyboard uk
# System language
lang en_GB
# SELinux configuration
selinux --disabled
# Do not configure the X Window System
skipx
# Installation logging level
#logging --level=info
# Reboot after installation
reboot
# System timezone
timezone  Europe/London
# System bootloader configuration
bootloader --location=mbr
# Partition clearing information
clearpart --all  

%post
%end

%packages --nobase
@admin-tools
openssh-server
-PackageKit-yum-plugin
-coolkey
-fedora-release-notes
-finger
-fprintd-pam
-ftp
-gnupg2
-hunspell
-iptstate
-irda-utils
-jwhois
-krb5-workstation
-lftp
-mlocate
-mtr
-nc
-nss_db
-nss_ldap
-pam_krb5
-pam_pkcs11
-pam_smb
-perf
-rdate
-rdist
-rsh
-sos
-sssd
-talk
-tcp_wrappers
-telnet
-time
-tree
-vconfig
-words
-ypbind

%end

OS Remastering CentOS 5.5

Introduction

As part of Backup and Restore , We will be customizing an OS CD for our use. We will be using LiveCD for this purpose. We will be customizing it according to our requirement.

Objective

We will customize the LiveCD so that it will represent two customize menu when this system is booted using this CD. We will be inserting our own script which will execute automatically when system is booted without any login. It will execute the script and reboots the system

Software Requirement

1) CentOS 5.5 DVD
This can be downloaded from below link
http://mirror.as29550.net/mirror.centos.org/5.5/isos/i386/CentOS-5.5-i386-bin-DVD.iso
2) LiveCD
This can downloaded from below link
http://mirror.as29550.net/mirror.centos.org/5.5/isos/i386/CentOS-5.5-i386-LiveCD-Release2.iso

Procedure Overview

First we will install a regular CentOS 5.5 on a system. This can be done on a VM as well. We have used. In this installed machine we will modify our LiveCD. This is necessary to have base system same as that of LiveCD. This is due to the squashfs version incompatibility.

We have to use squashfs-tools in order to modify the LIveCD. But If the version of squashFS used to modify the LiveCD is different from the version inside the LiveCD, then resulting LiveCD will not work. Moreover Some OSes used their own patch to patch the utilities. So it is better to use the same Base OS and Same OS based LiveCD.

Install the Base System
We installed the CentOS 5.5 using CentOS 5.5 DVD in a VM. We used KVM for this purpose.
First create the Virtual Harddisk using following command.

qemu-img create -f qcow2 /VIRTUAL/QEmu/CentOS/CentOS-5.5-i386.img 10G

Run VM and start Install Process.

kvm -smp 4 -m 2048 -hda /VIRTUAL/QEmu/CentOS/CentOS-5.5-i386.img -cdrom /DATA/Install/ISO/CentOS/CentOS-5.5-i386-bin-DVD.iso -usb -usbdevice tablet -vnc :40 -net nic,macaddr=00:00:00:00:00:01 -net tap,script=

I installed the system with minimum utilities. Only Base and Administration Utilities.
Configure the Base system(Newly created VM).
Add a repository for LiveCD tools(inside newly created VM).
Create a file /etc/yum.repos.d/CentOS-LiveCD.repo with following contents.

# Name: CentOS LiveCD repository
[livecd]
name = CentOS $releasever - LiveCD
baseurl = http://www.nanotechnologies.qc.ca/propos/linux/centos-live/$basearch/live
enabled=1
protect=0
gpgkey = http://www.nanotechnologies.qc.ca/propos/linux/RPM-GPG-KEY-PGuay

Install LiveCD tools (inside newly created VM).
Run the following command to install the LiveCD tools.

yum install livecd-tools syslinux

This is our full blown running OS on a harddisk. We will be making the changes to LIveCD in this system. All the following commands henceforth will be carried out in this machine only.
CentOS-5.5-i386.img was a 10GB Image. It is necessary to have sufficient space in order to modify the LiveCD. Installation of Base OS on 10GB disk space is enough. All the following commands are now run on this Base OS i.e. CentOS 5.5 (On this VM machine)

Modify LiveCD
Copy the LiveCD ISO to /root directory(inside newly created VM).

# cd /root
#
# ls
anaconda-ks.cfg  centos5.5-livecd.iso  install.log  install.log.syslog
# mkdir -p live/{isofiles,isomount,osmount}
# mount -o loop centos5.5-livecd.iso live/isomount
# cp -R live/isomount/* live/isofiles
# unsquashfs -dest live/squashfiles live/isofiles/LiveOS/squashfs.img
created 1 files
created 2 directories
created 0 symlinks
created 0 devices
created 0 fifos
#
# mount -o loop,rw live/squashfiles/LiveOS/ext3fs.img live/osmount
# cd live/osmount/
# cd etc/rc3.d
#
# mv S08ip6tables K08ip6tables
# mv S08iptables K08iptables
# mv S11auditd K11auditd
# mv S12syslog K12syslog
# mv S23setroubleshoot K23setroubleshoot
# mv S25bluetooth K25bluetooth
# mv S26acpid K26acpid
# mv S26apmd K26apmd
# mv S56cups K56cups
# mv S80sendmail K80sendmail
# mv S85gpm K85gpm
# mv S90crond K90crond
# mv S90xfs K90xfs
# mv S95anacron K95anacron
# mv S95atd K95atd
# mv S97yum-updatesd K97yum-updatesd
# mv S98avahi-daemon K98avahi-daemon
# mv S99firstboot K99firstboot
# mv S99smartd K99smartd
# mv S26hidd K26hidd
# mv S25pcscd K25pcscd
# mv S08mcstrans K08mcstrans

Once changes have been done. Pack the contents back to the CD.

# cd /root
# umount /root/live/osmount
# rm live/isofiles/LiveOS/squashfs.img
# mksquashfs live/squashfiles/* live/isofiles/LiveOS/squashfs.img -keep-as-directory

Modify isolinux.cfg file. (I used following contents.)

# cd live/isofiles/isolinux/
# cat isolinux.cfg
default menu.c32
timeout 100
menu title Welcome to livecd-kick-201008021533!
menu color border 0 #ffffffff #00000000
menu color sel 7 #ffffffff #ff000000
menu color title 0 #ffffffff #00000000
menu color tabmsg 0 #ffffffff #00000000
menu color unsel 0 #ffffffff #00000000
menu color hotsel 0 #ff000000 #ffffffff
menu color hotkey 7 #ffffffff #ff000000
menu color timeout_msg 0 #ffffffff #00000000
menu color timeout 0 #ffffffff #00000000
menu color cmdline 0 #ffffffff #00000000
menu hidden
menu hiddenrow 5
label linux0
  menu label Backup
  kernel vmlinuz0
  append initrd=initrd0.img root=CDLABEL=BackupRestore rootfstype=iso9660 ro quiet liveimg 3
label linux1
  menu label Restore
  kernel vmlinuz0
  append initrd=initrd0.img root=CDLABEL=BackupRestore rootfstype=iso9660 ro quiet liveimg 4

Make ISO

# cd /root/live/isofiles/
# mkisofs -o /root/BackupRestore.iso -b isolinux/isolinux.bin -c isolinux/boot.cat \
-no-emul-boot -boot-load-size 4 -boot-info-table -J -r -hide-rr-moved -V BackupRestore /root/live/isofiles

/root/BackupRestore.iso is the resulting customized LiveCD
Note: Please note -V option in above mkisofs command. The option to -V must be same as that of CDLABEL in isolinux.cfg file.

Creating Solaris OS VM

Solaris10 U6 virtual machine build up process Host OS used is Ubuntu 8.04 Hardy ( IP: 192.168.1.140)
KVM/Qemu is used as Virtualization software. KVM uses hardware assisted virtualization.
Solaris-10 ISO is needed for the purpose.
Following steps were followed Create the virtual disk on which OS will be installed.

# qemu-img create -f qcow2 Solaris10U6-x86.img

Start the OS installation.

kvm -m 4028 -cdrom /home/testuser/sol-10-u6-ga1-x86-dvd.iso -hda Solaris10U6-x86.img -vnc :14

Note: Above command will start a virtual machine with 4G of RAM (-m 4028) along with a VNC session started at VNC port 14
From another machine or same machine capture the above started VNC session and complete the installation.

kvm -m 2048  -hda Solaris10U6-x86.img -vnc :30 -net nic,macaddr=00:00:00:00:00:11 -net tap,script=

Above command will start machine with the networking provided bridging is in place. Also you may have to configure the network interface on guest machine manually.
if you have previously stopped the installation of Virtual guest, starting it again may give you some error. This is due to the fact that OS image (the harddisk of virtual OS) will now have corrupted OS and loader and virtual guest will attempt to boot from this corrupted image rather than cdrom image. To avoid this , run the following command ( it has an added "boot -d" parameter).

kvm -m 4028 -cdrom /home/testuser/sol-10-u6-ga1-x86-dvd.iso -hda Solaris10U6-x86.img -vnc :14 -boot d

If Solaris-10U6 was used to install as virtual guest. Its grub boot loader will not update the grub.conf correctly. ( This is only in Solaris10-U6)
Boot the virtual machine in single user mode Solaris Failsafe -- the other grub boot option. and run following commands.

# cd /a/boot/grub
# TERM=xterm
# export TERM
# EDITOR=vi
# export EDITOR

Change the following line under the section title Solaris 10 10/08 s10x_u6wos_07b x86

kernel /platform/i86pc/multiboot
to
kernel /platform/i86pc/multiboot kernel/unix

save and reboot Note: You may have to use traditional vi editor commands like h,j,k,l to navigate.