Staying on Topic - Invoke OpenFaaS functions with Kafka
•
0 likes•385 views
This talk introduced both OpenFaaS & OpenFaaS Cloud before demonstrating the Kafka Connector. Based on the Connector SDK, the Kafka Connector enables Kubernetes based functions to be triggered using messages consumed from Kafka topics. The connector easily integrates with your existing Kafka installations enabling you to flexibly complement your existing systems' functionality with serverless functions.
Report
Share
More Related Content
What's hot
What's hot (20)
Similar to Staying on Topic - Invoke OpenFaaS functions with Kafka
Similar to Staying on Topic - Invoke OpenFaaS functions with Kafka (20)
Recently uploaded
Recently uploaded (20)
Staying on Topic - Invoke OpenFaaS functions with Kafka
- 1. Staying on Topic: OpenFaaS & Kafka Richard Gee Core Contributor OpenFaaS Project Docker Birmingham - May 2019
- 2. The Session • OpenFaaS • OpenFaaS Cloud • Kafka Connector • Get involved: !" Tweet @rgee0 @openfaas
- 4. What’s in a name?
- 6. Application Evolution Monolith Decreasing concern for infrastructure Size&Complexity
- 7. Application Evolution Monolith Decreasing concern for infrastructure MicroserviceMicroserviceMicroserviceMicroservice Size&Complexity
- 8. Application Evolution Monolith Decreasing concern for infrastructure Function MicroserviceMicroserviceMicroserviceMicroservice Size&Complexity
- 9. Application Evolution Monolith Decreasing concern for infrastructure Function MicroserviceMicroserviceMicroserviceMicroservice Size&Complexity Short-lived Single-purpose No state Auto-scaling
- 10. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm
- 11. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm
- 12. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm • CNCF landscape
- 13. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm • CNCF landscape
- 14. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm • CNCF landscape • Multiple Production users
- 15. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm • CNCF landscape • Multiple Production users • 2 InfoWorld Bossies##
- 16. OpenFaaS: Functions as a Service • Started in Nov 2016 • Containers, Alexa, Swarm • CNCF landscape • Multiple Production users • 2 InfoWorld Bossies## • Ecosystem Partners
- 17. Our community & contributors #TeamServerless
- 18. Growth of OpenFaaS 17k Developer- First Operator- Friendly Community- Centric DRIVEN BY CORE VALUES
- 19. OpenFaaS: Serverless on your terms Anywhere Avoid lock-in through the use of Docker. Run on any public or private cloud.
- 20. OpenFaaS: Serverless on your terms Anywhere Avoid lock-in through the use of Docker. Run on any public or private cloud. Any scale Auto-scale with demand and to zero when idle.
- 21. OpenFaaS: Serverless on your terms You decide the limits Anywhere Avoid lock-in through the use of Docker. Run on any public or private cloud. Any code Build microservices & functions in any language. Legacy code and binaries. Any scale Auto-scale with demand and to zero when idle.
- 22. Serverless 2.0
- 23. Serverless 2.0 contract • HTTP/8080 • Docker / OCI image • Stateless • Immutable *adopted by OpenFaaS & Knative /
- 24. High Level Architecture API Gateway Provider Func 1 NATS Prometheus CLI/Web/API Func 2 AlertManager Queue Worker Orchestrator
- 25. Templates
- 26. Create a function $ faas new --lang php7 black-cat provider: name: openfaas gateway: http://127.0.0.1:8080 functions: black-cat: lang: php7 handler: ./black-cat image: rgee0/black-cat:latest stack.yml 1. php7 Template Docker Image
- 27. Create a function public function handle($data) { $res=UnirestRequest::get($data); return $res->code . PHP_EOL; } Handler.php composer.json "require": { "php": "^7.2", "mashape/unirest-php": "3.*" } $ faas up 2. Build Push Deploy } UP $ faas new --lang php7 black-cat 1.
- 29. Demo - What we saw • Created a new PHP7 function • faas up to Build, Push & Deploy locally • Added a second function • Portability - Deploying remotely • Using the Function Store • Function Auto-scaling
- 30. End Users & Case Studies
- 31. Case Study 1: Bullet-proof Bulletproof are using OpenFaaS to build an on demand and scalable Vulnerability Scanning (VA) engine. Using OpenFaaS allows us to use compute resource efficiently yet maintain the ability to grow to meet customer scanning demands. We also like the ability to use pure docker containers to compose multiple scanning tools with different technologies into a single, coherent interface. This has reduced the time need to add new tools to the platform. Case-study 1: BT Research, UK • Production usage • Heavily reliant • Research platform & internal tools • Shrink-wrap into products • KPIs Case Study 1:Case Study 1: BT Research, UK
- 32. Case Study 2: Vision Banco, Paraguay - Digital Transformation - OpenFaaS in production with HomeServe - Initial deployment with Swarm - KubeCon talk @ Seattle - Completely on-prem with GitLab
- 33. Case Study 3: Parking Application Swarm OpenFaaS Scheduler Routing / TLS FrontEnd External State Fully Immutable Flexible / Extensible Portable CI/CD with GitLab Fn Fn Fn Fn Fn Fn Fn Fn Fn Fn Fn Fn
- 35. OpenFaaS Cloud Focus on shipping functions without worrying about the CI/CD pipeline or underlying infrastructure
- 36. OpenFaaS Cloud is OpenFaaS, with • Multi-user • Personal dashboard & HTTPS • CI/CD built-in (GitOps) • Feedback via GitHub • Free Community Cluster • Self Hosted
- 37. High Level Flow Func 1 git repoDevelop Func 2 BuildKit Github* OpenFaaS Cloud Registry OpenFaaS Gateway
- 39. Encrypted secrets secrets.yml access_key=Ydx== secret_key=E2Dvv== access_key=1234 secret_key=abcd Seal() access_key=1234 Unseal() secret_key=abcd Public Key Private Key
- 41. GitHub Feedback
- 43. Dashboard
- 44. Kafka Connector Trigger functions through Topics
- 45. 13 ● Communication asynchronous ● Applications fully decoupled ● No restriction on schema, no shared datastore ● Microservices independent, increased resiliency ● Microservices state still important, persistent ORDERING INVOICING WAREHOUSE SHIPPING LOGGING MESSAGE QUEUE Order Placed Order Placed Invoice Generated Order Placed Item Found Item Found Item Shipped Order Placed Item Shipped All Events Invoice Generated Mike Bywater - https://bit.ly/2K9XBTP
- 46. Kafka Connector • Connector SDK • Helm Chart • Integrate with existing instances • Function Annotations • Consumer of multiple topics
- 47. Kafka Connector Incoming message Topic: signup msg: {"userid":1013} Lookup Functions Use /system/functions to BuildFunctionMap() Invoke: /function/email-welcome with: {"userid":1013} Kafka Connector Topic Function payment billing-audit signup email-welcome ... ... Builds API Gateway Function: billing-audit Annotations: topic: payment Function: email-welcome Annotations: topic: signup Function Map
- 48. Deploying image: openfaas/kafka-connector:0.3.3 replicas: 1 gateway_url: http://gateway.openfaas:8080 topics: faas-request print_response: true print_response_body: false broker_host: kafka basic_auth: true $ helm upgrade kafka-connector openfaas/kafka-connector —-install —-namespace openfaas $ helm upgrade kafka-connector openfaas/kafka-connector —install --namespace openfaas --set image=rgee0/kafka-connector:0.4.0 --set topics=“cats,not-cats" --set print_response_body=true
- 49. Cat Facts Kafka Topic: not-cats Topic: cats not-catsTopic: cats cat-sifter inception notify-slack post-tweet db-stasher { "url": "https://an.image.com/rainbow.jpg", "inferences": [ { "score": 0.99434, "name": "rod" }, { "score": 0.00024, "name": "jane" }, { "score": 0.00006, "name": "freddy" } ], "category": "not-cats" }
- 51. Cat Facts Kafka Topic: not-cats Topic: cats not-catsTopic: cats cat-sifter inception notify-slack post-tweet db-stasher
- 52. What about OpenFaaS Cloud?
- 53. Cat Facts pic-count OpenFaaS Cloud Kafka Topic: not-cats Topic: cats not-catsTopic: cats cat-sifter inception notify-slack post-tweet db-stasher
- 54. Demo - What we saw • OpenFaaS on Digital Ocean DOKS • Kafka Connector & Annotated Functions • Multiple Topics -> Function -> DB • Voice Service • Alexa Skill • OpenFaaS Cloud Function
- 55. Wrapping Up OpenFaaS: Serverless Functions Made Simple OpenFaaS Cloud: Further reduce friction Kafka Connector: Connect Kafka topics to OpenFaaS Functions Integrate your existing tools & applications
- 56. Get Involved Join Slack: https://docs.openfaas.com/community Try the Workshop: https://github.com/openfaas/workshop *Instructor led option available OpenFaaS Cloud Community Cluster: https://forms.gle/8e6ZXJKMcDHpV6Xu6
- 57. Resources Docs: https://docs.openfaas.com Blog: https://www.openfaas.com/blog Follow on Twitter: @OpenFaaS Kafka Connector: https://github.com/openfaas/faas-netes Try Digital Ocean Marketplace: https://m.do.co/c/2962aa9e56a1
- 58. Support OpenFaaS Sponsor the project: https://www.openfaas.com/donate/ Donate using Open Collective: https://opencollective.com/openfaas Donate using LF Community Bridge: https://bit.ly/2V8jasf