SlideShare a Scribd company logo

Interop 2015: Hardly Enough Theory, Barley Enough Code

•
•
Jeremy Schulman
Jeremy Schulman

Interop 2015 presentation covering Python programming theory and code examples: YAML, JSON, CSV, iterators, generators, JSON-schema

Related slideshows

Agile experiments in Machine Learning with F#
Agile experiments in Machine Learning with F#Agile experiments in Machine Learning with F#
Agile experiments in Machine Learning with F#
 
4th Lecture: JSP and such
4th Lecture: JSP and such4th Lecture: JSP and such
4th Lecture: JSP and such
 
Donald Ferguson - Old Programmers Can Learn New Tricks
Donald Ferguson - Old Programmers Can Learn New TricksDonald Ferguson - Old Programmers Can Learn New Tricks
Donald Ferguson - Old Programmers Can Learn New Tricks
 
1 of 41
Download to read offline
intermediate Python techniques for the automation journey April 2015 Jeremy Schulman @nwkautomaniac Hardly Enough Theory Barely Enough Code
Agenda ● Automation - Why Me? ● Learn key concepts ● Learn cool Pythonic techniques
Why Automate? Don’t be the bottleneck Empower your team Increase agility and productivity Keep the business running Automate the mundane Free up your time Improve uptime and reliability Your time and your expertise
Theory ● Composers, constructors and consumers ● Decomposition ● What matters most ● Zero, one, or infinity What You will Learn Code ● Iterators and generators ● Files - JSON, YAML, CSV ● Validation - JSON schema ● Templating - Jinja2 ○ Basics ○ Programming logic ○ Custom extensions
Theory Just a few slides ...
Role Relationships Design things providing instructions to others Combination of art and science Creative processes and best-practices Composers Build finished goods based on the instructions Practiced skills - years to master the craft Specialized domain knowledge and tools Constructors Use finished goods Generally does not require much skill Sometimes requires use of tools / products Consumers
Decomposition Everyone has favorites: ● Modeling techniques ... ● Diagramming techniques ... ● Process techniques ... Breaking a complex problem or system into parts Easier to conceive, understand, program and maintain Also known as factoring Entity Relationship DiagramFlowchart Diagram
Apply Decomposition Consider a switch configuration file ... ● Create shared and unique data-sets ● Define variables into explicit and derived / calculated items ● Understand cohesion of data and code ● Reuse the data for other purposes, like validation and operations ● Data can be used for multi-vendor Step 1: separate "data" from "code" Step 2: decompose "data" Step 3: decompose "code" Step 4: goto step 2 until manageable Why do this?
What Matters Most? Does it do what it is supposed to do? Are there even any requirements? Is high availability / fault tolerant a consideration? Functionality How fast does it run? Is speed really even an issue?Performance Does it need to handle the growth of your environment? Do you need to deal with concurrent processing?Scale When it breaks, can you fix it (quickly)? Can other people work on it? What happens if you leave? Maintainability
Zero, One, or Infinity No need to do something so just don't do itZero I must think about everyone else that might use this or have to deal with it one day. Empathy. Infinity Doing it only for myself and I don't need to worry about anyone else using it or what happens when I'm gone One
Code Iterators and Generators
Iterators Loop over things ● list, dictionary, set, tuple ● files - each line in text file ● strings - each character in string ● class objects supporting the iter protocol ● generators - see next slide List iteration code example Example output
Why use Generators? Lazy Evaluation Evaluation strategy which delays the evaluation of an expression until its value is needed Benefits ● Reduce memory ● Generate data on-demand ● Generate “infinite” data ● Performance increases ● Use as abstract objects ○ as function args ○ treat like any other iterablehttp://en.wikipedia.org/wiki/Lazy_evaluation
Generators Produce sequence of things ● behave like iterators, but lazy ● generator functions use yield ● generator expressions are comprehensions enclosed by parentheses Generator expression example Generator function example
Code Loading Data: JSON, YAML and CSV
Loading Data from Files - JSON Example file Example code Example output
Loading Data from Files - YAML Example file Example code Example output
Loading Data from Files - CSV Example file Example code Example outputDo you see the differences? 1. id is a string, not integer 2. name is not the key How to fix this?
Loading Data from Files - CSV (fixed!) Example code Example output
Demo: Putting it all together Task Load all of the files, of different types, into a single dictionary variable Pythonic techniques ● iterators and generators ● lambda functions ● reduce function ● iterables.chain ● functools.partial
Example Code
Demo: Filtering while loading Task While reading the data files, filter out records that match a given criteria id >= 20 Pythonic techniques ● generator ● compression ● … extract first/only dict item ● … with if statement
Example Code Example output
Code Templating with Jinja2
Templating with Jinja2 - Intro Loaded from text files Created dynamically by program Commonly used for configuration {% if age > 12 %} ... {% endif %} Hello {{ name | capitalize }} Hello {{ name }} Template content is text Programmatic controls Filters and extensions Variable string interpolation
Very Simple Example - Jinja2
Programmatic Controls - Jinja2 {% if age > 20 %} {% else %} {% endif %} if / then / else {% for this in some_iterable %} {% endfor %}loop over iterators {% set myvar = some_func() %}create variables {% include "this file.j2" %} {% include file_name %}include other files
Load template searching directories Jinja2 "Loader" determines how to find the template files Jinja2 "Environment" allows you to control aspects template loading and rendering
Example: Program Control
Custom Filters - Jinja2 Create a sorted key list by using the inner data dictionary Bind the function to the Jinja2 environment "filters" dictionary
Example Output
Including other files - Jinja2
Example Output The state_list variable using in the template was created in the python code using the groupby function, see next slide ...
Example Code: Report https://docs.python.org/2/library/itertools.html#itertools.groupby
Code Data Validation
Data Validation - JSON Schema Used to validate data Verify the syntactic contents Both inputs and outputs Schema is written in JSON Other schema mechanism: ● ASN.1 - SNMP ● YANG - NETCONF ● XSD - XML Example Data Example Schema
More interesting example Data Schema ● One "record" per file ● Record key is max 16 alpha-chars ● Only the properties: ○ state ○ id ○ job
JSON Schema - Docs http://json-schema.org/latest/json-schema-validation.html#toc
Automation Summary Programming is a tool not a career change It's not about you it's about everyone else Try being right now rather than being right
More? ● JSON - https://docs.python.org/2/library/json.html ● JSON Schema - http://json-schema.org ● YAML - http://pyyaml.org/wiki/PyYAMLDocumentation ● CSV - https://docs.python.org/2/library/csv.html ● Jinja2 - http://jinja.pocoo.org/ ● itertools - https://docs.python.org/2/library/itertools.html ● jq - https://github.com/stedolan/jq ● nwkautomaniac - https://github.com/jeremyschulman/nwkautomaniac
Jeremy Schulman @nwkautomaniac Thank you!

More Related Content

Interop 2015: Hardly Enough Theory, Barley Enough Code

  • 1. intermediate Python techniques for the automation journey April 2015 Jeremy Schulman @nwkautomaniac Hardly Enough Theory Barely Enough Code
  • 2. Agenda ● Automation - Why Me? ● Learn key concepts ● Learn cool Pythonic techniques
  • 3. Why Automate? Don’t be the bottleneck Empower your team Increase agility and productivity Keep the business running Automate the mundane Free up your time Improve uptime and reliability Your time and your expertise
  • 4. Theory ● Composers, constructors and consumers ● Decomposition ● What matters most ● Zero, one, or infinity What You will Learn Code ● Iterators and generators ● Files - JSON, YAML, CSV ● Validation - JSON schema ● Templating - Jinja2 ○ Basics ○ Programming logic ○ Custom extensions
  • 5. Theory Just a few slides ...
  • 6. Role Relationships Design things providing instructions to others Combination of art and science Creative processes and best-practices Composers Build finished goods based on the instructions Practiced skills - years to master the craft Specialized domain knowledge and tools Constructors Use finished goods Generally does not require much skill Sometimes requires use of tools / products Consumers
  • 7. Decomposition Everyone has favorites: ● Modeling techniques ... ● Diagramming techniques ... ● Process techniques ... Breaking a complex problem or system into parts Easier to conceive, understand, program and maintain Also known as factoring Entity Relationship DiagramFlowchart Diagram
  • 8. Apply Decomposition Consider a switch configuration file ... ● Create shared and unique data-sets ● Define variables into explicit and derived / calculated items ● Understand cohesion of data and code ● Reuse the data for other purposes, like validation and operations ● Data can be used for multi-vendor Step 1: separate "data" from "code" Step 2: decompose "data" Step 3: decompose "code" Step 4: goto step 2 until manageable Why do this?
  • 9. What Matters Most? Does it do what it is supposed to do? Are there even any requirements? Is high availability / fault tolerant a consideration? Functionality How fast does it run? Is speed really even an issue?Performance Does it need to handle the growth of your environment? Do you need to deal with concurrent processing?Scale When it breaks, can you fix it (quickly)? Can other people work on it? What happens if you leave? Maintainability
  • 10. Zero, One, or Infinity No need to do something so just don't do itZero I must think about everyone else that might use this or have to deal with it one day. Empathy. Infinity Doing it only for myself and I don't need to worry about anyone else using it or what happens when I'm gone One
  • 12. Iterators Loop over things ● list, dictionary, set, tuple ● files - each line in text file ● strings - each character in string ● class objects supporting the iter protocol ● generators - see next slide List iteration code example Example output
  • 13. Why use Generators? Lazy Evaluation Evaluation strategy which delays the evaluation of an expression until its value is needed Benefits ● Reduce memory ● Generate data on-demand ● Generate “infinite” data ● Performance increases ● Use as abstract objects ○ as function args ○ treat like any other iterablehttp://en.wikipedia.org/wiki/Lazy_evaluation
  • 14. Generators Produce sequence of things ● behave like iterators, but lazy ● generator functions use yield ● generator expressions are comprehensions enclosed by parentheses Generator expression example Generator function example
  • 15. Code Loading Data: JSON, YAML and CSV
  • 16. Loading Data from Files - JSON Example file Example code Example output
  • 17. Loading Data from Files - YAML Example file Example code Example output
  • 18. Loading Data from Files - CSV Example file Example code Example outputDo you see the differences? 1. id is a string, not integer 2. name is not the key How to fix this?
  • 19. Loading Data from Files - CSV (fixed!) Example code Example output
  • 20. Demo: Putting it all together Task Load all of the files, of different types, into a single dictionary variable Pythonic techniques ● iterators and generators ● lambda functions ● reduce function ● iterables.chain ● functools.partial
  • 22. Demo: Filtering while loading Task While reading the data files, filter out records that match a given criteria id >= 20 Pythonic techniques ● generator ● compression ● … extract first/only dict item ● … with if statement
  • 25. Templating with Jinja2 - Intro Loaded from text files Created dynamically by program Commonly used for configuration {% if age > 12 %} ... {% endif %} Hello {{ name | capitalize }} Hello {{ name }} Template content is text Programmatic controls Filters and extensions Variable string interpolation
  • 27. Programmatic Controls - Jinja2 {% if age > 20 %} {% else %} {% endif %} if / then / else {% for this in some_iterable %} {% endfor %}loop over iterators {% set myvar = some_func() %}create variables {% include "this file.j2" %} {% include file_name %}include other files
  • 28. Load template searching directories Jinja2 "Loader" determines how to find the template files Jinja2 "Environment" allows you to control aspects template loading and rendering
  • 30. Custom Filters - Jinja2 Create a sorted key list by using the inner data dictionary Bind the function to the Jinja2 environment "filters" dictionary
  • 33. Example Output The state_list variable using in the template was created in the python code using the groupby function, see next slide ...
  • 36. Data Validation - JSON Schema Used to validate data Verify the syntactic contents Both inputs and outputs Schema is written in JSON Other schema mechanism: ● ASN.1 - SNMP ● YANG - NETCONF ● XSD - XML Example Data Example Schema
  • 37. More interesting example Data Schema ● One "record" per file ● Record key is max 16 alpha-chars ● Only the properties: ○ state ○ id ○ job
  • 38. JSON Schema - Docs http://json-schema.org/latest/json-schema-validation.html#toc
  • 39. Automation Summary Programming is a tool not a career change It's not about you it's about everyone else Try being right now rather than being right
  • 40. More? ● JSON - https://docs.python.org/2/library/json.html ● JSON Schema - http://json-schema.org ● YAML - http://pyyaml.org/wiki/PyYAMLDocumentation ● CSV - https://docs.python.org/2/library/csv.html ● Jinja2 - http://jinja.pocoo.org/ ● itertools - https://docs.python.org/2/library/itertools.html ● jq - https://github.com/stedolan/jq ● nwkautomaniac - https://github.com/jeremyschulman/nwkautomaniac