Learn to Love Regular Expressions

A presentation at Future of Web Apps London in October 2015 in London, UK by Drew McLellan

Slide 1

Slide 1

Regular Expressions

  • DREW MCLELLAN -
  • FOWA LONDON 2015 -
  • LEARN TO LOVE -

Slide 2

Slide 2

For the fearful.

Slide 3

Slide 3

Hello!

flickr.com/photos/85520404@N03/9535499657

Slide 4

Slide 4

Created by b mijnlie ff from the Noun Project

Slide 5

Slide 5

Created by Christy Presler from the Noun Project

Slide 6

Slide 6

Created by Yi Chen from the Noun Project

Slide 7

Slide 7

Humans are great at matching patterns.

Slide 8

Slide 8

RegExp are great at matching patterns.

Slide 9

Slide 9

RegExp Humans

Slide 10

Slide 10

Donec in euismod mi. Ut a ullamcorper eros, id ultricies odio. In ullamcorper lobortis fi nibus. Nunc molestie, ex id ultrices lobortis, ante elit Finding mauris consequat lacus, at scelerisque leo nisl vitae leo. cursus lacus eu erat euismod tincidunt. Etiam ultrices elementum nulla, eu ornare elit eleifend a. Mauris lacinia velit non maximus ultrices. Praesent in condimentum metus. Curabitur hendrerit eget text id egestas. Nam et sodales dui. Suspendisse potenti. Mauris sed suscipit dui. Suspendisse ultricies felis non lacus maximus rutrum. Duis vel ante et neque ornare sagittis eu a nisi. Curabitur ultrices aliquet magna ut venenatis. Duis nec rhoncus that , sed pulvinar dui. Nunc pellentesque tortor sem, convallis eleifend nibh pharetra eu. Nulla congue, nisi vitae consectetur sollicitudin, felis nisl malesuada tortor, ut semper sem tellus ut dui. Donec eget augue quis justo vestibulum sodales sit amet eget tortor. Donec viverra risus turpis, sit amet congue dolor vel matches . Pellentesque sollicitudin purus a ligula tristique, et posuere justo faucibus. Pellentesque vehicula id nisl sit amet mollis. Integer tempor eros id varius aliquam. Phasellus vel est ullamcorper, dignissim nulla et, iaculis ex. Maecenas a dictum orci, eu sagittis felis. Vestibulum scelerisque diam elit, vitae placerat ipsum congue nec. Nulla blandit magna vel velit feugiat, eget maximus tortor feugiat. In vel metus ex. Ut molestie enim vel dolor elementum, at patterns turpis volutpat. Sed pulvinar dignissim eros et interdum. Quisque scelerisque diam et facilisis consequat. Etiam gravida sodales ornare. Donec tristique sem vitae ipsum gravida, in fi nibus sem vulputate. Sed in ex at dolor euismod commodo sed nec augue. Maecenas sed dictum turpis, nec bibendum neque. Pellentesque dapibus mi vitae elit porttitor elementum. Vestibulum porttitor porta nunc, et laoreet eros fi nibus ac. Suspendisse potenti. Nunc a gravida nisi. Morbi et massa magna. Cras ligula erat, congue sit amet dignissim a, porttitor vel felis.

Slide 11

Slide 11

Regular Expressions Server rewrite rules. Form validation. Text editor search & replace. Application code.

Slide 12

Slide 12

Flavours POSIX basic & extended. Perl and Perl-compatible (PCRE). Most common implementations are Perl-like (PHP, JavaScript and HTML5, mod_rewrite, nginx)

Slide 13

Slide 13

In this exciting episode Basic syntax. Matching. Repeating. Grouping. Replacing.

Slide 14

Slide 14

But fi rst… A regular expression tester is a great way to try things out. There’s an excellent online tester at: regex101.com

Slide 15

Slide 15

Slide 16

Slide 16

Slide 17

Slide 17

RegExp Basics

Slide 18

Slide 18

Basics / regex goes here /

/ regex goes here / modifiers / [A-Z]\w[A-Z] / i Delimiters are usually slashes by default. Some engines allow you to use other delimiters. Modi fi ers include things like case sensitivity.

Slide 19

Slide 19

Basics / this/that / Delimiters and other special characters need to be escaped with backslashes.

Slide 20

Slide 20

Basics / \w\s\d /

  • . * ? ^ | / () {} [] / ferret / Anything proceeded by a backslash has a special meaning. There are also a number of meta-characters with special meaning. Most other things are literal.

Slide 21

Slide 21

Matching

Slide 22

Slide 22

Words \w (lowercase W) / \w / 
 H ello, world, 1234. Matches an alphanumeric character, including

underscore.

Slide 23

Slide 23

Global modi fi er The ‘g’ global modi fi er returns all matches. Doesn’t stop at the fi rst match.

Slide 24

Slide 24

Words \w (lowercase W) / \w / g 
 Hello , world , 1234 . Matches an alphanumeric character, including

underscore.

Slide 25

Slide 25

Digits \d / \d / 
 Hello, world, 1 234.

/ \d / g 
 Hello, world, 1234 . Matches single digits 0-9.

Slide 26

Slide 26

Spaces \s / \s / 
 Hello,

world, 1234.

/ \s / g 
 Hello,

world,

1234 . Matches single whitespace character. Includes spaces, tabs, new lines.

Slide 27

Slide 27

Character classes These are all shorthand character classes . Character classes match one character, but o ff er a set of acceptable possibilities for the match. The tokens we’ve looked at a shorthand for more complex character classes.

Slide 28

Slide 28

Words \w [ A-Za-z0-9_ ] Character classes match one character only. They can use ranges like A-Z. They are denoted by [square brackets].

Slide 29

Slide 29

Digits \d [ 0-9 ] Character classes match one character only. They can use ranges like A-Z. They are denoted by [square brackets].

Slide 30

Slide 30

Spaces \s [

\f 

] Character classes match one character only. They can use ranges like A-Z. They are denoted by [square brackets]. !!!

Carriage return

New line

Ta b \f Form feed

Slide 31

Slide 31

Custom classes [ ol3 ] / [ ol3 ] /g 
 He llo , w o r l d, 12 3 4 . [ a-z0-9- ] / [ a-z0-9- ] /g 
 / 2009 / nice-title

Slide 32

Slide 32

Negative classes [^ ol3 ] / [^ ol3 ] /g 
 He llo , w o r l d, 12 3 4. Use a caret to indicate the class should match none of the given characters. [^ a-z0-9- ] / [^ a-z0-9- ] /g 
 / 2009 / nice-title

Slide 33

Slide 33

Dot A dot (period) matches any character other than a line break. It’s often over-used. Try to use something more speci fi c if possible.

Slide 34

Slide 34

Dot / . /g 
 Hello, world, 1234. Matches any character other than a line break.

Slide 35

Slide 35

!false Developer joke time.

Slide 36

Slide 36

So where does this get us?

Slide 37

Slide 37

Matching Hello world (1980-02-21). / \d\d\d\d-\d\d-\d\d / 
 
 Hello world ( 1980-02-21 ). So that’s something, right?

Slide 38

Slide 38

Repetition

Slide 39

Slide 39

Repetition Matching single characters gets old fast. There are four main operators or ‘quanti fi ers’ for specifying repetition.

Slide 40

Slide 40

Repetition ? Match zero or once. + Match once or more. * Match zero or more. {x} Match x times. {x,y} Match between x and y times.

Slide 41

Slide 41

Repetition / \d\d\d\d-\d\d-\d\d / / \d {4} -\d {2} -\d {2} / / [ a-z0-9- ]+ /g 
 
 / 2009 / nice-title

Slide 42

Slide 42

Greediness Repetition quanti fi ers are ‘greedy’ by default. They’ll try to match as many times as possible, within their scope. Sometimes that’s not quite what we want, and we can change this behaviour to make them ‘lazy’.

Slide 43

Slide 43

Greediness / < .+

/ 
 
 This <em>is</em> some HTML. EXPECTED: 
 This <em> is</em> some HTML. ACTUAL: 
 This <em>is</em> some HTML. Repetition quanti fi ers try to match as many times as they’re allowed to.

Slide 44

Slide 44

Greediness / < .+?

/ 
 
 This <em> is</em> some HTML. Quanti fi ers can be made ‘lazy’ with a question mark.

Slide 45

Slide 45

Anchors

Slide 46

Slide 46

Anchors Anchors don’t match characters, but the position within the string. There are three main anchors in common use.

Slide 47

Slide 47

Anchors ^ The beginning of the string. $ The end of the string. \b A word boundary.

Slide 48

Slide 48

Anchors / ^ Hello /g 
 
 Hello , Hello / Hello $ /g 
 
 Hello, Hello Anchors fi nd matches based on position.

Slide 49

Slide 49

Anchors / cat /g 
 
 cat con cat enation / \b cat \b /g 
 
 cat concat enation Word boundaries are useful for avoiding accidental sub- matches.

Slide 50

Slide 50

[‘hip’, ‘hip’] Developer joke time.

Slide 51

Slide 51

Grouping

Slide 52

Slide 52

Grouping Parts of a pattern can be grouped together with (parenthesis). This enables repetition to be applied on the group, and enables us to control how the result is ‘captured’.

Slide 53

Slide 53

Grouping abc123-def456-ghi789 / [ a-z ]{ 3 }[ 0-9 ]{ 3 }

? /

/ ( [ a-z ]{ 3 }[ 0-9 ]{ 3 }

? )+ / [ 
 ‘ abc123- ’, 
 ‘ def456- ’, 
 ‘ ghi789 ’ 
 ] Round brackets enable us to create groups that can then be repeated.

Slide 54

Slide 54

Grouping / ( [ a-z ]{ 3 }[ 0-9 ]{ 3 }

? )+ / / (?: [ a-z ]{ 3 }[ 0-9 ]{ 3 }

? )+ / Groups are captured by default. If you don’t need the group to be captured, make it non- capturing.

Slide 55

Slide 55

Grouping / \w+ @ \w+ . \w+ / drew@allinthehead.com / ( \w+ ) @ ( \w+ . \w+ ) / [ 
 ‘ drew ’, 
 ‘ allinthehead.com ’ 
 ] Capturing groups is very useful! !!!

Slide 56

Slide 56

Grouping / (?<user> \w+ ) @ (?<domain> \w+ . \w+ ) / [ 
 user: ‘ drew ’, 
 domain: ‘ allinthehead.com ’ 
 ] Some engines o ff er named groups.

Slide 57

Slide 57

Replacing

Slide 58

Slide 58

Replacing If you’ve used capturing groups in your pattern, you can re-insert any of those matched values back into your replacement. This is done with ‘back references’. Back references use the index number of the captured group.

Slide 59

Slide 59

Replacing with back references

<?php $str = 'drew@allinthehead.com' ; $pattern = '/(\w+)@(\w+\.\w+)/' ; $replacement = ' $1 is now fred@ $2 ' ; $result = preg_replace ( $pattern , $replacement , $str ); echo $result ; > drew is now fred@allinthehead.com PHP uses the preg (Perl Regular Expression) functions to perform matches and replacements.

Slide 60

Slide 60

Replacing with back references var

str

'drew@allinthehead.com' ; var

pattern

/(\w+)@(\w+.\w+)/ ; var

replacement

' $1 is now fred@ $2 ’ ;

var

result

str . replace ( pattern , replacement ); console.log ( result );

drew is now fred@allinthehead.com JavaScript uses the replace()

method of a string object.

Slide 61

Slide 61

Putting it to use

Slide 62

Slide 62

HTML5 input validation <input name="sku" type="text" pattern="[A-Z]{3}[0-9]{8-10}"

HTML5 adds the pattern attribute on form fi elds. They’re parsed using the browser’s JavaScript engine.

Slide 63

Slide 63

Apache 
 mod rewrite RewriteEngine On RewriteRule


 ^news/([1-2]{1}[0-9]{3})/([a-z0-9-]+)/?


 /news.php?year=$1&slug=$2 URL rewriting in Apache uses PCRE.

Slide 64

Slide 64

Your application code

<?php $str = 'Look at this https:// www.youtube.com/watch?v=loab4A_SqoQ and this https://www.youtube.com/watch? v=I-19GRsBW-Y' ; $pattern = '/(\w+:\/\/[^\s"]+)/' ; $replacement = '<a href="$1">$1</a>' ; echo preg_replace ( $pattern , $replacement , $str ); > Look at this <a href="https:// www.youtube.com/watch? v=loab4A_SqoQ">https://www.youtube.com/ watch?v=loab4A_SqoQ</a> and this <a href="https://www.youtube.com/watch? v=I-19GRsBW-Y">https://www.youtube.com/ watch?v=I-19GRsBW-Y </a> Don’t copy this example - it’s simpli fi ed and insecure.

Slide 65

Slide 65

Further reading

Slide 66

Slide 66

Further reading Teach Yourself Regular Expressions in 10 minutes, by Ben Forta. (Not actually in 10 minutes.) Mastering Regular Expressions, by Je ff rey E. F. Friedl.

Slide 67

Slide 67

Further learning regex101.com

Slide 68

Slide 68

Thanks! @drewm speakerdeck.com/drewm/getting-to-grips-with-regular-expressions