Find repeated Group pattern match. Case study of Ruby, Python and Go

Find repeated Group pattern match. Case study of Ruby, Python and Go

Sometimes I have to find the a group of patterns that are occurring in a log file. Normal grep -f <filename> does that job but it does not print the delimiter that can distinguish between repeated found patterns.

I needed something that can tell me that it has found a few patterns from a group in a sequence and last pattern has been found. Draw a delimiter here and start search for group again in remaining file.

I first wrote it in Ruby. I had huge files to parse. Ruby script was talking quite long to finish. As a case study I decided to write same utility using Python and Go.

Needless to say , Go was much faster.

Following are the codes.

Following are the sample data files.

pattern file :

hotmail
yahoo
google
gmail

Subject file:

This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Another one is gmail
Now is google
Another one is gmail
But this is going to be a yahoo
This is hotmail test
Junk lines
more junk lines
MORRRRRRRRRR
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
Junk lines
more junk lines
MORRRRRRRRRR

Following is the output:

$ go run atest.go pattern subjectfile
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========
This is hotmail test
But this is going to be a yahoo
Another one is gmail
===========
Now is google
Another one is gmail
===========
But this is going to be a yahoo
This is hotmail test
This is hotmail test
But this is going to be a yahoo
Now is google
Another one is gmail
===========

Launch AWS EC2 nstance using SaltStack

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli
pip install M2Crypto
pip install pyzmq

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Become root on MacBook now:

mkdir /etc/salt
touch /etc/salt/cloud.profiles
touch /etc/salt/cloud.providers

cat /etc/salt/cloud.profiles
base_ec2_private:
     provider: amazon_ireland_region
     image: ami-9d23aeea

cat /etc/salt/cloud.providers
amazon_ireland_region:
     id: ABCDEFGHIJKLMNO235M
     key: aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv
     keyname: MyEC2Key
     private_key: /Users/MacUser/EC2/MyEC2Key.pem
     location: eu-west-1
     availability_zone: eu-west-1a
     securitygroup: MySecurityGroupSSHOnly
     size: t2.micro
     del_root_vol_on_destroy: True
     ssh_username: ec2-user
     rename_on_destroy: True
     ssh_interface: public_ips
     provider: ec2

Launch Instance now:

source ~MacUser/PythonVirtENVs/SaltStack/bin/activate
salt-cloud --profile=base_ec2_private First_Instance

Launch AWS EC2 instance using awscli

Prepare environment: Install Required libraries.

virtualenv SaltStack
source SaltStack/bin/activate
pip install salt
pip install apache-libcloud
pip install awscli

created a new user in AWS console using Identity Management console

testuser
Access Key ID:
ABCDEFGHIJKLMNO235M
Secret Access Key:
aVeBUeixIlt1ScfseCV344NMnrx4fecNnex9mNNmjyjWv

Note: Above Key ID and Access Key are replaced with junk vales and will not work.
Above user “testuser” was added to AdministratorAccess policy in IAM (identity access management) in AWS console (Web interface).

In order to spin new instance and to be able to connect to them afterwards , we need a key pair. Either we can generate a new pair and upload it to AWS or generate it in AWS console (web interface) itself. In my case I had already generated the KeyPair. This key pair is called as “MyEC2Key”. This can be viewed under “compute” -> “EC2” -> “Key Pair"
Now create a “Security Group” that will allow ssh.

aws ec2 create-security-group --group-name MySecurityGroupSSHOnly --description "Inbound SSH"
aws ec2 authorize-security-group-ingress --group-name MySecurityGroupSSHOnly --cidr 0.0.0.0/0 --protocol tcp --port 22

Following command will spin a FREE TIER instance in AWS cloud

aws ec2 run-instances --image-id ami-9d23aeea --key-name MyEC2Key --instance-type t2.micro --count 1 --security-groups MySecurityGroupSSHOnly