Juggling Chainsaws: Perl and MongoDB

Juggling Chainsaws
David Golden
Senior Engineer, MongoDB
 
YAPC::NA 2015

I lead the
MongoDB Perl
driver project

Do I need to explain
what MongoDB is?

Documents
Auto-sharding
Native replication
Pluggable storage engines
Aggregation framework queries
Comprehensive secondary indexes
Multiple geospatial indexes and text search
ACID guarantees available at the document level
Fault tolerance and disaster recovery through automated failover
Enterprise-version has extensive capabilities for authentication, authorization, auditing and encryption
Ops Manager and MMS provide continuous incremental backup, point-in-time recovery of replica sets and consistent snapshots of sharded clusters.

"Troll" © S.L.M Roma https://ﬂic.kr/p/4uRvNc (CC BY-NC-SA)

"chainsaws are fun" © Ryan Kathleen https://ﬂic.kr/p/5zY75A (CC BY)

"London Juggler" © Leon Benjamin https://ﬂic.kr/p/cmTzL (CC BY)

Perl and MongoDB are
surprisingly similar

ster·e·o·type
noun
a widely held but ﬁxed and
oversimpliﬁed image or idea of a
particular person or thing

"Dead camel" © Nick Brooks https://ﬂic.kr/p/ivZdP (CC BY-NC-SA)

"MongoDB is web Scale" © gar1t http://www.mongodb-is-web-scale.com/

$_='ev
al("seek040D
ATA,0, 0;");foreach(1..3)
{<DATA>;}my @camel1hump;my$camel;
my$Camel ;while( <DATA>){$_=sprintf("%-6
9s",$_);my@dromedary 1=split(//);if(defined($
_=<DATA>)){@camel1hum p=split(//);}while(@dromeda
ry1){my$camel1hump=0 ;my$CAMEL=3;if(defined($_=shif
t(@dromedary1 ))&&/S/){$camel1hump+=1<<$CAMEL;}
$CAMEL--;if(d efined($_=shift(@dromedary1))&&/S/){
$camel1hump+=1 <<$CAMEL;}$CAMEL--;if(defined($_=shift(
@camel1hump))&&/S/){$camel1hump+=1<<$CAMEL;}$CAMEL--;if(
defined($_=shift(@camel1hump))&&/S/){$camel1hump+=1<<$CAME
L;;}$camel.=(split(//,"040..m`{/J047134}L^7FX"))[$camel1h
ump];}$camel.="n";}@camel1hump=split(/n/,$camel);foreach(@
camel1hump){chomp;$Camel=$_;y/LJF7173175`047/061062063
064065066067070/;y/12345678/JL7F175173047`/;$_=reverse;
print"$_040$Cameln";}foreach(@camel1hump){chomp;$Camel=$_;y
/LJF7173175`047/12345678/;y/12345678/JL7F1751730 47`/;
$_=reverse;print"040$_$Cameln";}';;s/s*//g;;eval; eval
("seek040DATA,0,0;");undef$/;$_=<DATA>;s/s*//g;( );;s
;^.*_;;;map{eval"print"$_"";}/.{4}/g; __DATA__ 124
1 501450401651631450401571 460401 410
40143141 1551451 540401 51155 141
1471450 40151156 040141 16316 3
157143 15114116 41511 57156
040167 1511641 50040 1201
45162 15404015 1163 04014
10401 641621 41144 145
15514 1162 15304 0157
146 04011 7047 1221
4515 11541 54171 040
046 01210116 316
315 714315 114
116 4145163 054
040 11115614 3056
040 12516314514 4040
1671 511641 500 40160
145162 155151
"Camel JAPH" © Erudil http://www.cpan.org/misc/japh

"fresh ramen noodle" © Kropsoq http://goo.gl/mu2fIQ (CC BY-SA)

"A hash brown patty" © Mike.lifeguard http://goo.gl/2we5ye (CC BY-SA)

"buy it in bottles" © Pete https://ﬂic.kr/p/qbvaRb (CC BY)

Perl programmers use
dynamic data structures 
all the time!!!

MongoDB 
♥ 
Data Structures

Licensed under Fair use via Wikipedia - https://en.wikipedia.org/wiki/File:TICK_COMIC_CON_EXTRAVAGANZA_1.jpg

{
name => 'The Tick',
battlecry => 'Spoon!',
birthday => '1973-07-16',
relations => {
friends => [
'Arthur',
'Die Fledermaus',
'American Maid',
],
foes => [
'The Terror',
'Thrakkorzog',
],
}

Sort heroes by 
ﬁrst friend's name?

# Perl using Schwartzian Transform
my @sorted =
map { $_->[0] }
sort { $a->[1] cmp $b->[1] }
map { [$_, $_->{relations}{friends}[0] };

# MongoDB using query with sort clause
my @sorted = $heroes->find(
{}, { sort => [ 'relations.friends.0' => 1 ] }
)->all;

# Perl using Schwartzian Transform
my @sorted =
map { $_->[0] }
sort { $a->[1] cmp $b->[1] }
map { [$_, $_->{relations}{friends}[0] };
# MongoDB using query with sort clause
my @sorted = $heroes->find(
{}, { sort => [ 'relations.friends.0' => 1 ] }
)->all;

✓ Missing things others 
take for granted

✓ Experienced users
work around ﬂaws

✓ Vibrant, enthusiatic 
communities

Manipulexity 
 
Manipulation of complex things
— Larry Wall

Whipuptitude 
 
Aptitude for whipping things up
— Larry Wall

Tools for getting 
the job done

Henry Spencer described the Perl scripting
language as a “Swiss-Army chainsaw”, intending
to convey his evaluation of the language as
exceedingly powerful but ugly and noisy and
prone to belch noxious fumes.
— The Jargon File

This had two results: (1) Perl fans adopted the
epithet as a badge of pride, and (2) it entered
more general usage to describe software that is
highly versatile but distressingly inelegant.
— The Jargon File

Highly versatile but
distressingly inelegant?

Good enough 
for many things

And if one is good, 
wouldn't two be better?

"Camel warning sign UAE" © Katzenmeier http://commons.wikimedia.org/wiki/File:Camel_warning_sign_UAE.jpg (PD)

{
"abstract" : "File path utility",
"author" : [
"David Golden <dagolden@cpan.org>"
],
"name" : "Path-Tiny",
"prereqs" : {
"runtime" : {
"requires" : {
"Carp" : "0",
"Cwd" : "0",
"Digest" : "1.03",
"Digest::SHA" : "5.45",
...
},
},
...
},
"provides" : {
"Path::Tiny" : {
"file" : "lib/Path/Tiny.pm",
"version" : "0.068"
},
...
},
"release_status" : "stable",
...
"x_contributors" : [
"Alex Efros <powerman@powerman.name>",
...
],
...
}

Thought experiment:
Rebuild CPAN?

Rebuilding CPAN?
• Tarball storage
• Tarball processing
• Worker logs
• Index lookup
• Text search

# store a file
open $fh, "<", $tarball_path;
$gridfs = $db->get_gridfs("tarballs");
$id = $gridfs->put($fh, $metadata);
# get a file
$bytes = $gridfs->get($id)->slurp;
Storage: GridFS

# in supervisor
my $queue = MongoDBx::Queue->new(...);
$queue->add_task({ tarball_id => $id });
# in worker
while ( my $task = $queue->reserve_task )
{
# ... process tarball ...
$queue->remove_task( $task );
}
Processing: MongoDBx::Queue

$db->run_command([
create => "pause_log",
capped => true,
max => 1000,
]);
$log->find(
{}, { cursorType => 'tailable' }
);
Logging: capped collection

$packages->find(
{
mod => "Foo",
ver => {
'$gte' => $v_min,
'$lte' => $v_max,
},
}
);
Index lookup: query language

$packages->find(
{
mod => "Foo::Bar",
ver => {
'$gte' => $v_min,
'$lte' => $v_max,
},
}
);
Index lookup: query language

$meta->find(
{
'_authorized.Foo' => true
'release_status' => 'stable',
}
);
# sort through versions client side
# with version.pm
Index lookup (take 2)

$meta->indexes->create_one([
abstract: "text",
keywords: "text",
]);
$meta->find( {
'$text' => {
'$search' => "dist zilla plugin"
}
});
Text search: text indexes

Some other handy tools
• Geographic indexing and search
• TTL indexes (auto-expiration)

Coming soon:
MongoDB v1.0.0 Beta 1

Coming soon:
Right after 
this talk!

Coming soon:
Right after 
this talk!
Before 
Lunch

Coming soon:
Right after 
this talk!
Before 
LunchDuring 
YAPC

Coming soon:
Right after 
this talk!
Before 
LunchDuring 
YAPC
Right
now?

• Better server discovery, monitoring, failover
• Immutable conﬁg (and no more globals!)
• New CRUD and Index APIs
• Consistent error handling
• Progress towards a pure-Perl option
• BSON round-trippability

• Download MongoDB
• Try the beta driver
• Find a MUG near you
• Follow my blog (dagolden.com)
• Follow me on twitter (@xdg)
• ++ the driver on MetaCPAN
• Ask questions

Juggling Chainsaws: Perl and MongoDB

More Related Content

Juggling Chainsaws: Perl and MongoDB