Very simple template substitutions using Python and Perl

Very Simple Template Substitutions Using Python Perl

2020-03-15T20:50:05Z

---

Contents

• Introduction
• Using Python
• Using Shell and Perl

---

Introduction

It is quite common to treat text files as templates which contains place holders that will be replaced by some real values. Python jinja or Ruby erb template engines are there to resolve for these kind of issues. They are quite feature rich. Following is a very simple Template substitution program in Python and Perl that does similar kind of job. Its very basic but works for simple scenarios.

Using Python

Consider following text file test_file.txt with place holders in it.

This is a %TEST1% . Another %TEST1%  %TEST1%
Now this is a %TEST2%.

Following is the file var.py containing values of above place holder.

TEST1="test_string1"
TEST2="test_string2"

Following is the python program that will replace above values in test_file.txt from var.py file.

#! /usr/bin/env python

import re
import fileinput
import sys
from var import *

def print_data_vars(matchObj):
    return globals()[matchObj.group(1)]

for line in fileinput.input('test_file', inplace=True):
    line = re.sub(r'%(.+?)%', print_data_vars, line)
    sys.stdout.write(line)

when above program is run , it will replace placeholders with values in existing test_file.txt file.

Using Shell and Perl

Assuming var.py and test_file.txt contents are same as above, same effect can also be achieved using shell and perl combination as follows.

#! /usr/bin/env bash
set -a
source ./var.py
set +a
perl -i -wpl -e 's!%(.+?)%!$ENV{$1}!g' test_file

Perl Regex Usages

Multiline Matchings:

Inside a perl script:

#! /usr/bin/perl
$str = "This is a test
Will it work
lets see
";
if ($str =~ m!^This.+test$!)
{
     print "matched\n"
}

Following are the contents of file "data"

# cat data
This is a test file
Will it work
lets see

# As perl one liners:

$ cat data | perl -w -e '$/=""; while(<>) { m!^This.+file$!m and print }'
This is a test file
Will it work
lets see


# DEFAULT BEHAVIOUR
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH

# s modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!s and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ 


# m modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!m and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!m and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see)!m and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^let.+see$)!m and print "$1\n" or print "NOMATCH\n" }'
lets see

# sm modifier behaviour:
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+file$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+work$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^Will.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^lets.+see$)!sm and print "$1\n" or print "NOMATCH\n" }'
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ 

$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!s and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work$).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+it).+^let.+see$!sm and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it


# BACKREFRENCES:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\1! and print "MATCHED\n"}'
MATCHED
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+\g{-1}! and print "MATCHED\n"}'
MATCHED

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED

# CAPTURING and NON CAPTURING GROUPING:
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "$1\n"}'
vis  

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?:vis).+\g{testmatch}! and print "$1\n"}'
Reference to nonexistent named group in regex; marked by <-- here="" in="" m="" testmatch="">vis).+\g{testmatch <-- -e="" -w="" 1.="" a="" at="" echo="" here="" line="" perl="" vis="" while="">) { m!(?:vis).+! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(vis).+! and print "$1\n"}'
vis
$

$ echo "datchet etet it underway" | perl -w -e '$/=""; while(<>) { m!(et){2}\s+! and print "MATCHED\n"}'
MATCHED
 
# NAMED CAPTURING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?test file)! and print "$+{fm}\n" or print "NOMATCH\n" }'
test file
$ echo "vis a vis" | perl -w -e '$/=""; while(<>) { m!(?vis).+\g{testmatch}! and print "MATCHED\n"}'
MATCHED
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "$1" or print "NOMATCH\n" }'
Use of uninitialized value $1 in string at -e line 1, <> chunk 1.
$ cat data | perl -w -e '$/=""; while(<>) { m!(?:test file)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED 


# inside modifiers:
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+test! and print "MATCHED\n" or print "UNMATCHED\n"}'
UNMATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+TEST! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED
$ 

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i)test! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+((?i)test)! and print "$1\n"}'
TEST
$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "$1\n"}'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ echo "This is a TEST" | perl -w -e '$/=""; while(<>) { m!a\s+(?i:test)! and print "MATCHED\n" or print "UNMATCHED\n"}'
MATCHED

# inside modifiers (multiline modifier trick):
$ cat data | perl -w -e '$/=""; while(<>) { m!(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s)(^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work
$ cat data | perl -w -e '$/=""; while(<>) { m!((?s)^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
This is a test file
Will it work

# inside modifiers non-capturing match:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "$1\n" or print "NOMATCH\n" }'
Use of uninitialized value $1 in concatenation (.) or string at -e line 1, <> chunk 1.

$ cat data | perl -w -e '$/=""; while(<>) { m!(?s:^This.+work)! and print "MATCHED\n" or print "NOMATCH\n" }'
MATCHED


# LOOK AHEAD ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(.+lets)(.+it.+$)!sm and print "$1 $3\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it work
lets see
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=.+lets)(.+it)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
test file
Will it
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'
NOMATCH


# LOOK BEHIND  ASSERTIONS:
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(it)!sm and print "$1\n" or print "NOMATCH\n" }'
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(?<=This is a )(.+it)!sm and print "$1\n" or print "NOMATCH\n" }'
test file
Will it

# CONDITIONAL MATCHING:
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(asa)(?(1)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?=lets)(.+it.+$)!sm and print "$1 $2\n" or print "NOMATCH\n" }'        # This is not a conditional matching
NOMATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(.+it.+$))!sm and print "$1 $2\n" or print "NOMATCH\n" }'
Use of uninitialized value $2 in concatenation (.) or string at -e line 1, <> chunk 1.
test file 

$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(.+it.+$))!sm and print "MATCH\n" or print "NOMATCH\n" }'
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk)|(anotherjunk))!sm and print "MATCH\n" or print "NOMATCH\n" }'
NOMATCH
 
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=junk)junk)!sm and print "MATCH\n" or print "NOMATCH\n" }'  # if condition is false then it simply skips
MATCH
$ cat data | perl -w -e '$/=""; while(<>) { m!(test file)(?(?=lets)(junk))!sm and print "MATCH\n" or print "NOMATCH\n" }'  # why is this working ?  no idea
MATCH

(?(?=condition)(then1|then2|then3)|(else1|else2|else3))

# Some Special cases to study:
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(ab)*! ; print "@arr\n";}'
ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((ab)*)! ; print "@arr\n";}'
ab ab
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)*)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)! ; print "@arr\n";}'
aba
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(?:a|b)*!g ; print "@arr\n";}'
aba  a  
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]*!g ; print "@arr\n";}'
aba  a 
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]*)!g ; print "@arr\n";}'
aba  a  
$  echo "abacaddb" | perl -w -e 'while(<>) { @arr = $_ =~ m!((?:a|b)*)!g ; print "@arr\n";}'
aba  a   b  
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([ab]+)! ; print "$1\n";}'
aba
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!([abc]+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((a|b|c)+)! ; print "$1\n";}'
abaca
$  echo "abacaddb" | perl -w -e 'while(<>) {  $_ =~ m!((?:a|b|c)+)! ; print "$1\n";}'
abaca


#======================================================

$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!(a|b)+! ; print "@arr\n";}'
a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!((a|b)+)! ; print "@arr\n";}'
aba a
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m![ab]+! ; print "@arr\n";}'
1
$ echo "abaca" | perl -w -e 'while(<>) { @arr = $_ =~ m!([ab]+)! ; print "@arr\n";}'
aba