SlideShare a Scribd company logo
Scaling Web Applications with Background Jobs:
Takeaways from Generating a Huge PDF
Lydia Cupery
All Things Open, 2023
Hi!
Nice to meet you 󰗝
Lydia Cupery
lydia-cupery.com
Lydia Cupery
The Problem
Scaling Web Applications with Background

Recommended for you

Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details

The practical implementation of Continuous Delivery at Etsy, and how it enables the engineering team to build features quickly, refactor and change architecture, and respond to problems in production. Presented at GOTO Aarhus 2012. Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com. http://www.etsy.com/careers

etsycontinuous deploymentsoftware engineering
Service workers and their role in PWAs
Service workers and their role in PWAsService workers and their role in PWAs
Service workers and their role in PWAs

Progressive Web Applications have gained unparalleled momentum in the tech world and are currently one of the hottest trends in Web Development. Find out how PWA attempts to combine features offered by most modern browsers with the benefits of mobile experience and how service workers make them fast, reliable & engaging.

service workersweb push notificationprogressive web apps
On the importance of done
On the importance of doneOn the importance of done
On the importance of done

The document discusses how teams at The Economist online work incrementally and iteratively on the same codebase to improve the site, with releases every Tuesday. It focuses on the importance of having a definition of "Done" and explains that for their teams, a story is Done when it has been implemented, verified through manual testing, validated by the product owner, and ready to be released to production. Defining and achieving Done increases productivity and allows for early delivery of business value and feedback. The teams use tools like Bazaar, Launchpad, Hudson and EC2 to manage their workflow, do peer reviews, and set up test instances for user stories.

drupalaugust2010drupalcon
Motion
design
Physical
Computing
Download
Invoices
Fetch the invoice data from the database
Transform data to shape of PDF data
Put completed PDF somewhere client can
access
Write the Invoice PDF for each customer
Combine all Invoice PDFs
SERVER
write customer
invoices PDF
reference to
created PDF
CLIENT
Scaling Web Applications with Background
Fetch the invoice data from the database
Transform data
Put completed PDF somewhere client can access
Write the Invoice PDF for each customer
Combine all Invoice PDFs
Write the Invoice PDF for each customer
Combine all Invoice PDFs

Recommended for you

Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...

With no SAP upgrade or additional SAP software, learn how Welch Allyn used their existing SAP environment to cut their SAP transaction times in half through iOS bar code scanning and tailoring the SAP Production Planning and Control module to the specific needs of the business. Welch Allyn improvements include: Change Container Status With Bar Code (PKBC) used to empty KanBan bins on the production floor went from 7 key strokes and 3 mouse clicks to one bar code scan. With 350k transactions, every key stroke matters! Error-proofing the transaction is also done with barcodes and bins no longer have to be moved, saving handling time and money. MFBF (Backflushing In Repetitive Mfg) transaction was streamlined and extended to iOS devices, going from 22 keystrokes to one barcode scan. With 150K of these transactions, the time savings and reduction in errors paid for itself in months.

sap user interfacescreen personasfiori
Office Add-ins community call-March 2019
Office Add-ins community call-March 2019Office Add-ins community call-March 2019
Office Add-ins community call-March 2019

Agenda summary: -Doug Mahugh: High-level overview of Office (web) Add-ins -Sohail Zafar: What’s new with Outlook JavaScript APIs -Nitesh Golchha: Office Add-ins Single Sign-On (SSO) -David Chestnut: Office Add-ins Patterns & Practices -Sudheer Maremanda: Edge WebView control for the Office Add-ins platform -Mark Roden of PSC Group: Office Add-in demo (Word & SharePoint integration)

office add-insoffice365devmsdev
Virtualization Commputing
Virtualization CommputingVirtualization Commputing
Virtualization Commputing

Sorry Guys I am Little Late Publishing It.Virtulization is Too Fast Than My Expectation's.Its Already Shows its Impact Everywhere Corporates,SMB's etc.

parenttoolboxvmwarefuture
First attempt - create one large PDF.
Write each customer statement to the main pdf.
It works but it takes way too long.
Okay, what if we create all the PDFs at the same time? And then
merge them?
We’ll use a Promise.all and store all the individual PDFs in memory.
What if we processed two or three invoices at a time, and write
each of those as pdfs to the file system?
Then, we could merge those PDFs together with pdf-lib.
What if we used an external service? Is there any external
service that could help us out with our bottlenecks?
Write the Invoice PDF for each customer
Combine all Invoice PDFs
…perhaps for combining PDFs!

Recommended for you

Serverless is more findev than devops
Serverless is more findev than devopsServerless is more findev than devops
Serverless is more findev than devops

A lot of the discussions around serverless has been about the benefits it brings to the table with regards to DevOps - more infrastructure automation, scalability and resilience out-of-the-box. Developers love it because they can offload even more undifferentiated heavy-lifting to their cloud vendors, and they can focus their energy on building the things their users want. Businesses benefit hugely too because they have happier developers who can deliver value faster! But the true power of the serverless paradigm, for the business, is the pay-per-invocation model. It allows them to finally understand the cost of user transactions, and calculate the return on investment of features. And if you embrace this superpower then it can even open the door to an entirely new business model built around pay-per-transaction and give your business the competitive advantage over your rivals.

awsaws lambdaserverless
web, spa vs traditional - 2016
web, spa vs traditional - 2016web, spa vs traditional - 2016
web, spa vs traditional - 2016

This document compares single page applications (SPAs) to traditional websites. SPAs load all necessary resources on the initial page load, then dynamically load any additional content. This allows the URL to be frequently updated to simulate page transitions while avoiding full page reloads. SPAs require less server-side processing than traditional sites since more logic runs in the browser. Well-written SPAs are convenient for users because they are responsive like native apps, can notify users of state changes, and work across different devices and platforms similarly to regular websites.

websingle page applicationtraditiaolan web
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...

SpringOne Platform 2019 Session Title: From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can Reduce Operational Toil Speaker: Shaun Norris, Field CIO, APJ, Pivota Youtube: https://youtu.be/iqYg9EfUvoI

developerpivotalpivotal cloud foundry
Cuts down on the time to combine PDFs, but generating the individual PDFs
and writing the PDFs to the file system still takes too long.
Takes Too Long
Generating & writing PDFs to the
file systems takes too long.
Generate too many pdfs in parallel,
use a lot of memory, the dyno runs
out of memory.
Uses Too Much Memory
VS
What if we keep increasing memory, and upgrade our dynos to ones with more
memory?
invoice report please!
report.pdf
The system could still run out of memory and start failing as soon as more
than one user tries to generate a PDF at once.
invoice report please!
invoice report please!
ERROR
please load X page!
ERROR
report.pdf

Recommended for you

PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...

Building a complex deployment that is also reliable is difficult. Not anymore! Join us to learn a new set of technologies created by Puppet and Google that is easy to build and maintain, reduces failures and cuts debugging time. They are a must have set of tools if you are a developer or DevOps professional. You will learn how to build a complete deployment pipeline using Puppet, from scratch to a fully operational, public facing, multi-tier application in just 10 minutes or less. In 30 minutes we'll make it scalable as well.

puppetconf 2017
Max Voloshin - "Organization of frontend development for products with micros...
Max Voloshin - "Organization of frontend development for products with micros...Max Voloshin - "Organization of frontend development for products with micros...
Max Voloshin - "Organization of frontend development for products with micros...

While our product was growing our team came to need to implement microservices. Later it became obvious that our approaches on organization of frontend development should be rethought and significantly improved. The report contains our team's solutions for simple and comfortable frontend product development with microservices. Also, this talk is about how we along with the way updated frontend framework, separated frontend and backend, solved internalization problem and started using Docker for front end tasks.

javascriptminskconference
Redesigning a large B2B website - The FusionCharts revamping story
Redesigning a large B2B website - The FusionCharts revamping storyRedesigning a large B2B website - The FusionCharts revamping story
Redesigning a large B2B website - The FusionCharts revamping story

A detailed look at everything that went behind the redesign of the FusionCharts website - objectives, tech stack and server hardware, information architecture, front-end decisions to make it responsive, design tradeoffs, SEO, and analytics. The decisions we made, the process we followed, the learnings we had and the final results.

websiteb2bseo
Make it a Background Job!
“Background jobs can dramatically
improve the scalability of a web
app by enabling it to offload
slow or CPU-intensive tasks from
its front-end.”
Browser Web Server Background Service
request invoices PDF schedule generate
invoice PDF
in-progress
generate
invoice PDF
is it done yet?
nope!
is it done yet?
nope!
is it done yet?
yes! here it is
Architecture Overview

Recommended for you

Water billing management system project report.pdf
Water billing management system project report.pdfWater billing management system project report.pdf
Water billing management system project report.pdf

Our project entitled “Water Billing Management System” aims is to generate Water bill with all the charges and penalty. Manual system that is employed is extremely laborious and quite inadequate. It only makes the process more difficult and hard. The aim of our project is to develop a system that is meant to partially computerize the work performed in the Water Board like generating monthly Water bill, record of consuming unit of water, store record of the customer and previous unpaid record. We used HTML/PHP as front end and MYSQL as back end for developing our project. HTML is primarily a visual design environment. We can create a android application by designing the form and that make up the user interface. Adding android application code to the form and the objects such as buttons and text boxes on them and adding any required support code in additional modular. MySQL is free open source database that facilitates the effective management of the databases by connecting them to the software. It is a stable ,reliable and the powerful solution with the advanced features and advantages which are as follows: Data Security.MySQL is free open source database that facilitates the effective management of the databases by connecting them to the software.

kamal acharyaproject managementsoftware development
LITE 2018 – A Deep Dive Into the API [Iain Brown]
LITE 2018 – A Deep Dive Into the API [Iain Brown]LITE 2018 – A Deep Dive Into the API [Iain Brown]
LITE 2018 – A Deep Dive Into the API [Iain Brown]

The document provides an overview of Administrate's API platform. It describes the key components of the Administrate API, including the GraphQL API, developer portal, and webhooks. It then discusses how webhooks allow for real-time notifications when events occur, using an example of integrating with Acclaim for digital badges. Finally, it introduces the Piston design system for building consistent user interfaces and how customers can use the platform to build custom solutions that integrate with Administrate.

administratelite2018
APIfying an ERP - ongoing saga
APIfying an ERP - ongoing sagaAPIfying an ERP - ongoing saga
APIfying an ERP - ongoing saga

What kind of problems we solved and what kind of architecture decisions we made when building our ERP platform and new user interfaces

crmerpintegration
web process
(web dyno)
data store
(redis)
library to
implement queue
system on top of
redis (BullMQ)
worker processes
(worker dynos)
With a Background Job You Can…
With background
jobs you can…
Speed Things Up
Browser Web Server Background Service
request invoices PDF schedule generate
invoice PDF
in-progress
generate
invoice PDF
is it done yet?
nope!
is it done yet?
nope!
is it done yet?
yes! here it is

Recommended for you

Application Performance Lecture
Application Performance LectureApplication Performance Lecture
Application Performance Lecture

The document discusses performance analysis of a large intranet web application called Global Projects Delivery (GPD) which was experiencing erratic response times for users. Key findings from the analysis were: 1) Most traffic occurred during morning, post-lunch, and evening peaks and was concentrated on a few frequently accessed pages like daily task updates. 2) While the application server response time was good, 98% of the total response time was spent on data transfer, indicating the network was the bottleneck. 3) Pages with multiple frames and requests slowed down data transfer rates significantly compared to simpler pages. 4) Actions recommended included redesigning pages to reduce frames and requests, relocating servers closer to users, and

Web Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Web Performance, Scalability, and Testing Techniques - Boston PHP MeetupWeb Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Web Performance, Scalability, and Testing Techniques - Boston PHP Meetup

I gave this talk on 4/27/11 at the Boston PHP Meetup Group. It covers both server side and client side optimizations, as well as monitoring tools and techniques.

phpweb developmentweb performance
Building Reliability - The Realities of Observability
Building Reliability - The Realities of ObservabilityBuilding Reliability - The Realities of Observability
Building Reliability - The Realities of Observability

Presented at the ATO RTP Meetup Presented by Jeremy Proffit, Director of DevSecOps & SRE for Customer Care and Communications, Ally Title: Building Reliability - The Realities of Observability Abstract: Join me as we discuss true observability, learn what works and what doesn't. We'll not only discuss dashboards, monitoring and alerting, but how these can be built by automation or included in your IAC modules. We'll talk about how to properly alert staff based on priority to keep your staff and yourself sane. And even discuss architecture and how it impacts reliably and why serverless isn't always the best at being reliable.

all things openopen sourceato2024
Sorted
Customers with
Invoices
fetch invoice data
generate invoices
customerIds: A-E
fetch invoice data
generate invoices
customerIds: F-O
fetch invoice data
generate invoices
customerIds: P-Z
combine generated
invoices
JOB QUEUE
customers A-E
customers F-O
customers P-Z
customer
list/
WORKER_COUNT
generate invoices
customerIds: A-E
generate invoices
customerIds: F-O
generate invoices
customerIds: P-Z
combine generated
invoices
JOB QUEUE
fetch data
(A-E)
write
invoices
upload “batch
invoices xxx - 1”
fetch data
(F-O)
write
invoices
upload “batch
invoices xxx - 2”
fetch data
(P-Z)
write
invoices
upload “batch
invoices xxx - 3”
combine s3 files to
“batch invoices xxx”
upload “batch
invoices xxx”
Job
Partial Job Partial Job Partial Job
Combine Partial Job Outputs
output output
output
With background
jobs you can…
Show Progress

Recommended for you

Modern Database Best Practices
Modern Database Best PracticesModern Database Best Practices
Modern Database Best Practices

Presented at the ATO RTP Meetup Presented by Peter Zaitsev, Founder of Percona Title: Modern Database Best Practices Abstract: There are now more Database choices available for developers than ever before - there are general purpose databases and specialized databases, single node and distributed databases, Open Source, Proprietary databases and databases available exclusively in the cloud. In this presentation we will cover the best practices of choosing database(s) for your applications, best practices as it comes to application development as well as managing those databases to achieve best possible performance, security, availability at the lowest cost.

all things openato2024open source
Open Source and Public Policy
Open Source and Public PolicyOpen Source and Public Policy
Open Source and Public Policy

All Things Open 2023 Presented at All Things Open 2023 Presented by Deb Bryant - Open Source Initiative, Patrick Masson - Apereo Foundation, Stephen Jacobs - Rochester Institute of Technology, Ruth Suehle - SAS, & Greg Wallace - FreeBSD Foundation Title: Open Source and Public Policy Abstract: New regulations in the software industry and adjacent areas such as AI, open science, open data, and open education are on the rise around the world. Cyber Security, societal impact of AI, data and privacy are paramount issues for legislators globally. At the same time, the COVID-19 pandemic drove collaborative development to unprecedented levels and took Open Source software, open research, open content and data from mainstream to main stage, creating tension between public benefit and citizen safety and security as legislators struggle to find a balance between open collaboration and protecting citizens. Historically, the open source software community and foundations supporting its work have not engaged in policy discussions. Moving forward, thoughtful development of these important public policies whilst not harming our complex ecosystems requires an understanding of how our ecosystem operates. Ensuring stakeholders without historic benefit of representation in those discussions becomes paramount to that end. Please join our open discussion with open policy stakeholders working constructively on current open policy topics. Our panelists will provide a view into how oss foundations and other open domain allies are now rising to this new challenge as well as seizing the opportunity to influence positive changes to the public’s benefit. Topics: Public Policy, Open Science, Open Education, current legislation in the US and EU, US interest in OSS sustainability, intro to the Open Policy Alliance Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openopen sourceopen source initiative
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...

This document summarizes a presentation about graph-quilt, an open source GraphQL orchestrator library. It discusses the challenges of building a GraphQL orchestrator to unify data from multiple services. Graph-quilt addresses this by allowing services to register their GraphQL schemas and composing them into a unified schema. It also supports features like remote schema extensions, authorization, and adapting existing REST APIs. The presenters believe graph-quilt provides a flexible way to build GraphQL gateways and help more clients adopt GraphQL.

all things openopen sourceato2023
Progress Indication
With no background job (assuming the server doesn’t time out
or run out of memory) :
Browser Web Server
request invoice PDF
here it is!
generate
pdf…
Indicating Progress
Browser Web Server Background Service
request invoices PDF schedule generate
invoice PDF
in-progress
generate
invoice PDF
jobs
is it done yet?
nope! 30% there
is it done yet?
nope! 70% there
is it done yet?
yes! here it is
Generate Partial Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
JOB PROGRESS
1 .1

Recommended for you

The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil NashThe State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil Nash

Presented at All Things Open 2023 Presented by Phil Nash - Sonar Title: The State of Passwordless Auth on the Web Abstract: Can we get rid of passwords yet? They make for a poor user experience and users are notoriously bad with them. The advent of WebAuthn has brought a passwordless world closer, but where do we really stand? In this talk we'll explore the current user experience of WebAuthn and the requirements a user has to fulfil to authenticate without a password. We'll also explore the fallbacks and safeguards we can use to make the password experience better and more secure. By the end of the session you'll have a vision of how authentication could look in the future and a blueprint for how to build the best auth experience today. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Total ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScriptTotal ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScript

Presented at All Things Open 2023 Presented by Phil Nash - Sonar Title: Total ReDoS: The dangers of regex in JavaScript Abstract: Regular expressions are complicated and can be hard to learn. On top of that, they can also be a security risk; writing the wrong pattern can open your application up to denial of service attacks. One token out of place and you invite in the dreaded ReDoS. But how can a regular expression cause this? In this talk we’ll track down the patterns that can cause this trouble, explain why they are an issue and propose ways to fix them now and avoid them in the future. Together we’ll demystify these powerful search patterns and keep your application safe from expressions that behave in a way that is anything but regular. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?

Presented at All Things Open 2023 Presented by Karl Mozurkewich - Storj Title: What Does Real World Mass Adoption of Decentralized Tech Look Like? Abstract: We delve into the transformative potential of decentralized technology. Beginning with a brief overview of the rise of centralization with the advent of the internet and the counter-shift marked by blockchain we explore the intrinsic characteristics of decentralized and distributed systems, such as trustless operations, peer-to-peer networks, and enterprise application scalability. Various sectors, including finance, supply chains, media and entertainment, data science and cloud infrastructure are on the brink of disruption. The societal implications are vast, with the potential for greater individual empowerment, a greener planet and more viable resource utilization, but concerns about data security persist. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openopen sourceato2023
Generate Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
● generate the invoices
○ updateProgress each time an
invoice generated
JOB PROGRESS
1 .2
Generate Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
● generate the invoices
○ updateProgress each time an
invoice generated
JOB PROGRESS
1 .4
Generate Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
● generate the invoices
○ updateProgress each time an
invoice generated
JOB PROGRESS
1 .6
Generate Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
● generate the invoices
○ updateProgress each time an
invoice generated
JOB PROGRESS
1 .8

Recommended for you

How to Write & Deploy a Smart Contract
How to Write & Deploy a Smart ContractHow to Write & Deploy a Smart Contract
How to Write & Deploy a Smart Contract

Presented at All Things Open 2023 Presented by Anastasia Lalamentik - Kaleido Title: How to Write & Deploy a Smart Contract Abstract: In this talk, Anastasia Lalamentik, Full Stack Engineer at Kaleido, will walk through how Ethereum smart contracts work and go over related concepts like gas fees, the Ethereum Virtual Machine (EVM), the block explorer, and the Solidity programming language. This is vital to anyone who wants to build a blockchain app and is a great introduction to blockchain technology for newcomers to the space. By the end of the talk, attendees will better understand how to: - Write a simple smart contract - Deploy their smart contract to an Ethereum test network through the latest tools like Hardhat and the MetaMask wallet - Test interactions with their deployed smart contract and ensure that everything is working properly Additionally, participants will get to interact with Anastasia's deployed smart contract at the end of the talk. Anastasia’s past talks have attracted and have been attended by a diverse group of participants with a range of experience in the space. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow

Presented at All Things Open 2023 Presented by Paul Brebner - Instaclustr (by Spot by NetApp) Title: Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Abstract: In this talk we’ll build a Drone delivery application, and then use it to do some Machine Learning “on the fly”. In the 1st part of the talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming data). With up to 2,000 (simulated) drones and deliveries in progress at once this application generates a vast flow of spatio-temporal data. In the 2nd part of the talk, we'll use this platform to explore Machine Learning (ML) over streaming and drifting Kafka data with TensorFlow to try and predict which shops will be busy in advance. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
DEI Challenges and Success
DEI Challenges and SuccessDEI Challenges and Success
DEI Challenges and Success

Presented at the All Things Open 2023 Inclusion and Diversity in Open Source Event Presented by Efraim Marquez-Arreaza - Red Hat Title: DEI Challenges and Success Abstract: In today's world, many companies and organizations have Diversity, Equity and Inclusion (DEI) communities. Red Hat Unidos is a DEI community focused on advocating for the Hispanic/Latine community. In this talk, we would like to share our challenges and success during the past 4-years and plans for the future. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Generate Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
● generate the invoices
○ updateProgress each time an
invoice generated
● updateProgress(0.9)
JOB PROGRESS
1 .9
Generate Invoices Job
(input, updateProgress) =>
● fetch data to generate
invoices
● updateProgress(0.1)
● generate the invoices
○ updateProgress each time an
invoice generated
● updateProgress(0.9)
● upload generated file to s3
● updateProgress(1)
JOB PROGRESS
1 1
view job progress
updateProgress(X) updates job progress
aggregate progress
across jobs requests job
progress
With background
jobs you can…
Support Simultaneous Users

Recommended for you

Supercharging tutorials with WebAssembly
Supercharging tutorials with WebAssemblySupercharging tutorials with WebAssembly
Supercharging tutorials with WebAssembly

Presented at All Things Open 2023 Presented by Robert Aboukhalil - CZI Title: Supercharging tutorials with WebAssembly Abstract: sandbox.bio is a free platform that features interactive command-line tutorials for bioinformatics. This talk is a deep-dive into how sandbox.bio was built, with a focus on how WebAssembly enabled bringing command-line tools like awk and grep to the web. Although these tools were originally written in C/C++, they all run directly in the browser, thanks to WebAssembly! And since the computations run on each user's computer, this makes the application highly scalable and cost-effective. Along the way, I'll discuss how WebAssembly works and how to get started using it in your own applications. The talk will also cover more advanced WebAssembly features such as threads and SIMD, and will end with a discussion of WebAssembly's benefits and pitfalls (it's a powerful technology, but it's not always the right tool!). Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Using SQL to Find Needles in Haystacks
Using SQL to Find Needles in HaystacksUsing SQL to Find Needles in Haystacks
Using SQL to Find Needles in Haystacks

Presented at All Things Open 2023 Presented by K.S. Bhaskar - YottaDB LLC Title: Using SQL to Find Needles in Haystacks Abstract: Database journal files capture every update to a database. A database of a few hundred GB can generate GBs worth of journal files every minute at busy times. Troubleshooting and forensices, especially of rare and intermittent problems, such as which process made what update and when, is an exercise of finding needles in haystacks. A similar problem exists with syslogs. A solution is to load the journal files and syslogs into a database, and use SQL to query the database. Bhaskar will present and demonstrate this with a 100% FOSS stack. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit InterceptConfiguration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit Intercept

The document discusses configuration security as a game of pursuit-evasion and intercept. It was presented by Wes Widner, Principal Engineer at Automox. The document includes a JSON policy snippet with an ID, statement, actions, effects, resources, and principal allowing the GetObject action on all objects in an S3 bucket for all principals. It has page numbers at the bottom indicating it is from a larger presentation.

all things openato2023open source
Before background jobs…
invoice report please!
invoice report please!
ERROR
please load X page!
ERROR
report.pdf
invoice report please!
please load X page!
working on that!
progress?
progress?
invoice report please!
working on that!
progress?
here you go!
With background
jobs you can…
Save Jobs for Later
invoice report please!
working on that!
progress?
progress?
invoice report please!
working on that!
progress?

Recommended for you

Scaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship ProgramScaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship Program

Presented at All Things Open 2023 Presented by Carol Huang & Mike Fix - Stripe Title: Scaling an Open Source Sponsorship Program Abstract: ​​We already know this: the open-source ecosystem needs further monetary investment from the companies that benefit most from it. Likewise, companies say they want to participate in these initiatives, but find it hard to dedicate resources to open source funding when there isn’t a clear ROI. This talk discusses how the Open Source Program Office at Stripe built a scalable, sustainable open source sponsorship model that aligns internal company incentives with those of open source maintainers and the community at large. We go over the unique “platformization” of our OSPO that allowed us to create multiple funding models, such as BYOB (Bring Your Own Budget), and share lessons learned from this experience as well as other OSPOs. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Build Developer Experience Teams for Open Source
Build Developer Experience Teams for Open SourceBuild Developer Experience Teams for Open Source
Build Developer Experience Teams for Open Source

Presented at All Things Open 2023 Presented by Arundeep Nagaraj - Amazon Web Services (AWS) Title: Build Developer Experience Teams for Open Source Abstract: Open Source has become the default strategy for many IT organizations and Enterprises. However, the constant challenge with Open Source leaders of these organizations has been - How is my product's developer experience? Is this the right metric to track? How can I scale my team to support our products better? How can I add automation to scale redundant workflows? If my product involves working with developers, how can I scale to the complexity of the requests and reduce Engineering bandwidth? The challenges within support of open source products continues to magnify depending on the end user persona whether they are consumers or contributors to your product. Consumers utilize your product, SDK's and API's and are blocked with using it or run into issues, whereas contributors are advanced users of your software that understands the codebase to provide a meaningful contribution back to the product. The answer to the above is to look at Open Source support as a first-class citizen of your corporate support strategy. To employ the right level of developer focused support as opposed to traditional infrastructure based support is key to scale to the amount of developers using your product. Supporting customers in the open involves more than pure support - building customer / developer experiences (DX) in the open (across platforms and communities) that pivots over the ability of your product's users or developers to be focused on the end-to-end value add. This helps with your active developer growth and retention of users. Key Takeaways: - IT leaders of Open Source will learn to employ strategies to build a DX team that engages on multiple platforms - Work on identifying accurate metrics for product and organization - Innovate on platforms such as Discord to build a bot and a dashboard - Ability to leverage customer feedback and iterate over the customer success flywheel - Distinguish between DX and Developer Advocacy (DA) Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Deploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache BeamDeploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache Beam

Presented at All Things Open 2023 Presented by Danny McCormick - Google Title: Deploying Models at Scale with Apache Beam Abstract: Apache Beam is an open source tool for building distributed scalable data pipelines. This talk will explore how Beam can be used to perform common machine learning tasks, with a heavy focus on running inference at scale. The talk will include a demo component showing how Beam can be used to deploy and update models efficiently on both CPUs and GPUs for inference workloads. An attendee can expect to leave this talk with a high level understanding of Beam, the challenges of deploying models at scale, and the ability to use Beam to easily parallelize their inference workloads. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
With background
jobs you can…
Have Less Timeouts/Errors
Communicating with an External Service
With no background job:
Browser Web Server
send out customer
emails, please!
TIMEOUT
��🏻♀
Mailgun
send these
emails, please!
��
Browser Web Server
send out customer
emails, please!
send these
emails, please!
in-progress
Mailgun
is it done yet?
nope! 30% there
is it done yet?
nope! 70% there
is it done yet?
yes - all emails
are sent!
��
Background
Service
send these
emails,
please!
Recap - With a background job you can…
speed things up
show progress
support simultaneous users
have less timeouts/errors
save jobs for later

Recommended for you

Sudo – Giving access while staying in control
Sudo – Giving access while staying in controlSudo – Giving access while staying in control
Sudo – Giving access while staying in control

Presented at All Things Open 2023 Presented by Peter Czanik - One Identity Title: Sudo – Giving access while staying in control Abstract: Sudo is used by millions to control and log administrator access to systems, but using the default configuration only, there are plenty of blind spots. Using the latest features in sudo let you watch some previously blind spots and control access to them. Here are four major new features, which arrived since the 1.9.0 release, allowing you see your blind spots: - configuring a working directory or chroot within sudo often makes full shell access redundant - JSON-formatted logs give you more details on events and are easier to act on - relays in sudo_logsrvd make session recording collection more secure and reliable - you can log and control sub-commands executed by the command run through sudo Let us take a closer look at each of these. Previously, there were quite a few situations where you had to give users full shell access through sudo. Typical examples include when you need to run a command from a given directory, or running commands in a chroot environment. You can now configure the working directory or the chroot directory and give access only to the command the user really needs. Logging is a central role of sudo, to see who did what on the system. Using JSON-formatted log messages gives you even more information about events. What is even more: structured logs are easier to act on. Setting up alerting for suspicious events is much easier when you have a single parser to configure for any kind of sudo logs. You can collect sudo logs not only by local syslog, but also by using sudo_logsrvd, the same application used to collect session recordings. Speaking of session recordings: instead of using a single central server, you can now have multiple levels of sudo_logsrvd relays between the client and the final destination. This allows session collection even if the central server is unavailable, providing you with additional security. It also makes your network configuration simpler. Finally, you can log sub-commands executed from the command started through sudo. You can see commands started from a shell. No more unnoticed shell access from text editors. Best of all: you can also intercept sub-commands. These are just a few of the most prominent features helping you to watch and control previous blind spots on your systems. See these and other possibilities in action in some live demos during our presentation. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsFortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML Applications

Presented at All Things Open 2023 Presented by Christine Abernathy - F5, Inc. Title: Fortifying the Future: Tackling Security Challenges in AI/ML Applications Abstract: As Artificial Intelligence (AI) and Machine Learning (ML) applications continue to surge, it is crucial to be aware of and address the security risks associated with these technologies. In this talk, Christine will explore AI/ML failure modes, threats, and mitigation strategies. She will guide you through the fundamentals of ML models then introduce you to key security challenges such as adversarial attacks, data poisoning, model inversion, model stealing, and membership inference attacks, using real-world examples to demonstrate their potential impact. Christine will also discuss privacy and ethical considerations in ML, touching upon techniques like federated learning and shedding light on the current regulatory landscape surrounding security risks. If you are developing AI/ML applications or incorporating AI/ML components into your technology stack, check out this talk. You will walk away with a deeper understanding of the current AI/ML security landscape and a toolkit to help you address these risks, enabling you to build safer, more secure, and privacy-aware applications. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openopen sourceato2023
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...

Presented at All Things Open 2023 Presented by Carlos Santana - AWS Title: Securing Cloud Resources Deployed with Control Planes on Kubernetes using Governance and Policy as Code Abstract: Are you concerned about the security of your cloud resources deployed on Kubernetes? Are you struggling to ensure compliance with regulatory requirements while managing your cloud infrastructure? If yes, then this talk is for you! We will discuss how to secure cloud resources deployed with Crossplane on Kubernetes using Governance and Policy as Code. We will explore how to leverage Governance and Policy as Code tools like Rego, Kyverno, and OPA to ensure security and compliance. By the end of this talk, you will have a better understanding of the challenges associated with securing cloud resources deployed with Crossplane or ACK on Kubernetes, the importance of Governance and Policy as Code in ensuring security and compliance, and why it is critical to use open source and open standards in these technologies. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023open source
Should you use a background job?
You might want a background job for…
CPU-intensive jobs Jobs communicating externally
I/O intensive jobs Scheduled jobs
Tips
Struggling with app responsiveness?
Try a background job.

Recommended for you

Building AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources codeBuilding AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources code

Presented at All Things Open 2023 Presented by benny Vasquez - AlmaLinux OS Foundation Title: Building AlmaLinux OS without RHEL sources code Abstract: AlmaLinux OS has historically been built to be an exact copy of RHEL, using RHEL's provided resources. With RedHat's shift away from sharing the full building blocks on git.centos.org, the AlmaLinux team has been hard at work to find a new, reliable path forward. Come hear about what we've been doing since June, and what we're planning for the future. Find more info about All Things Open: On the web: https://www.allthingsopen.org/ Twitter: https://twitter.com/AllThingsOpen LinkedIn: https://www.linkedin.com/company/all-things-open/ Instagram: https://www.instagram.com/allthingsopen/ Facebook: https://www.facebook.com/AllThingsOpen Mastodon: https://mastodon.social/@allthingsopen Threads: https://www.threads.net/@allthingsopen 2023 conference: https://2023.allthingsopen.org/

all things openato2023benny vasquez
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALLBLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL

Blockchain technology is transforming industries and reshaping the way we conduct business, manage data, and secure transactions. Whether you're new to blockchain or looking to deepen your knowledge, our guidebook, "Blockchain for Dummies", is your ultimate resource.

blockchainweb3blockchain technology
Password Rotation in 2024 is still Relevant
Password Rotation in 2024 is still RelevantPassword Rotation in 2024 is still Relevant
Password Rotation in 2024 is still Relevant

Password Rotation in 2024 is still Relevant

passwordmanagementrotation
Don’t recreate the wheel.
Use a library with a robust queueing system.
Does speed matter? It probably does.
Parallelize.
The job queue makes it easy to show users
progress. Do so!
Find the optimal number of workers and
optimal amount of resources per
worker(see next slide…)

Recommended for you

[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf

Kief Morris rethinks the infrastructure code delivery lifecycle, advocating for a shift towards composable infrastructure systems. We should shift to designing around deployable components rather than code modules, use more useful levels of abstraction, and drive design and deployment from applications rather than bottom-up, monolithic architecture and delivery.

infrastructure as codeclouddevops
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides

If you’ve ever had to analyze a map or GPS data, chances are you’ve encountered and even worked with coordinate systems. As historical data continually updates through GPS, understanding coordinate systems is increasingly crucial. However, not everyone knows why they exist or how to effectively use them for data-driven insights. During this webinar, you’ll learn exactly what coordinate systems are and how you can use FME to maintain and transform your data’s coordinate systems in an easy-to-digest way, accurately representing the geographical space that it exists within. During this webinar, you will have the chance to: - Enhance Your Understanding: Gain a clear overview of what coordinate systems are and their value - Learn Practical Applications: Why we need datams and projections, plus units between coordinate systems - Maximize with FME: Understand how FME handles coordinate systems, including a brief summary of the 3 main reprojectors - Custom Coordinate Systems: Learn how to work with FME and coordinate systems beyond what is natively supported - Look Ahead: Gain insights into where FME is headed with coordinate systems in the future Don’t miss the opportunity to improve the value you receive from your coordinate system data, ultimately allowing you to streamline your data analysis and maximize your time. See you there!

7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf

Solar Storms (Geo Magnetic Storms) are the motion of accelerated charged particles in the solar environment with high velocities due to the coronal mass ejection (CME).

solar storms
1X
1X
1X
$25 x 6 = $150
512 MB x 6 = 3GB
$50 x 3 = $150
1 GB x 3 3GB
1X
1X
1X
2X
2X
2X
$250 x 1 = $250
2.5 GB x 1 = 2.5 GB
Perf M
What about the PDF?
Fetch the invoice data from the database
Transform data to shape of PDF data
Put completed PDF somewhere client can access
Write the Invoice PDF for each customer
Combine all Invoice PDFs
Fetch list of customers with invoices from the database
Transform fetched data to shape of PDF data
Put completed PDF somewhere client can access
Write the Invoice PDF for each customer
Combine all Invoice PDFs
do not need to fetch all
invoice data
Fetch invoice data for customers fetching 1/10 amount of data
transforming 1/10 amount of data
writing PDF for
1/10 customers
worker dyno
worker dyno

Recommended for you

How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf

In the modern digital era, social media platforms have become integral to our daily lives. These platforms, including Facebook, Instagram, WhatsApp, and Snapchat, offer countless ways to connect, share, and communicate.

social media hackerfacebook hackerhire a instagram hacker
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...

Have you noticed the OpenSSF Scorecard badges on the official Dart and Flutter repos? It's Google's way of showing that they care about security. Practices such as pinning dependencies, branch protection, required reviews, continuous integration tests etc. are measured to provide a score and accompanying badge. You can do the same for your projects, and this presentation will show you how, with an emphasis on the unique challenges that come up when working with Dart and Flutter. The session will provide a walkthrough of the steps involved in securing a first repository, and then what it takes to repeat that process across an organization with multiple repos. It will also look at the ongoing maintenance involved once scorecards have been implemented, and how aspects of that maintenance can be better automated to minimize toil.

dartflutteropenssf
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy

Not so much to say

Thank You!
Lydia Cupery
lydia-cupery.com
Lydia Cupery

More Related Content

Similar to Scaling Web Applications with Background

Deploying Machine Learning in production without servers - #serverlessCPH
Deploying Machine Learning in production without servers - #serverlessCPHDeploying Machine Learning in production without servers - #serverlessCPH
Deploying Machine Learning in production without servers - #serverlessCPH
Damien Cavaillès
 
Automating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics Pipelines
Sadayuki Furuhashi
 
Computerized Accounting System
Computerized Accounting SystemComputerized Accounting System
Computerized Accounting System
Sabbir Ahmed
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
Mike Brittain
 
Service workers and their role in PWAs
Service workers and their role in PWAsService workers and their role in PWAs
Service workers and their role in PWAs
Ipsha Bhidonia
 
On the importance of done
On the importance of doneOn the importance of done
On the importance of done
Rob Purdie
 
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
DeeDee Kato
 
Office Add-ins community call-March 2019
Office Add-ins community call-March 2019Office Add-ins community call-March 2019
Office Add-ins community call-March 2019
Microsoft 365 Developer
 
Virtualization Commputing
Virtualization CommputingVirtualization Commputing
Virtualization Commputing
Hi-Techpoint
 
Serverless is more findev than devops
Serverless is more findev than devopsServerless is more findev than devops
Serverless is more findev than devops
Yan Cui
 
web, spa vs traditional - 2016
web, spa vs traditional - 2016web, spa vs traditional - 2016
web, spa vs traditional - 2016
Yauheni Nikanovich
 
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
VMware Tanzu
 
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
Puppet
 
Max Voloshin - "Organization of frontend development for products with micros...
Max Voloshin - "Organization of frontend development for products with micros...Max Voloshin - "Organization of frontend development for products with micros...
Max Voloshin - "Organization of frontend development for products with micros...
IT Event
 
Redesigning a large B2B website - The FusionCharts revamping story
Redesigning a large B2B website - The FusionCharts revamping storyRedesigning a large B2B website - The FusionCharts revamping story
Redesigning a large B2B website - The FusionCharts revamping story
FusionCharts
 
Water billing management system project report.pdf
Water billing management system project report.pdfWater billing management system project report.pdf
Water billing management system project report.pdf
Kamal Acharya
 
LITE 2018 – A Deep Dive Into the API [Iain Brown]
LITE 2018 – A Deep Dive Into the API [Iain Brown]LITE 2018 – A Deep Dive Into the API [Iain Brown]
LITE 2018 – A Deep Dive Into the API [Iain Brown]
getadministrate
 
APIfying an ERP - ongoing saga
APIfying an ERP - ongoing sagaAPIfying an ERP - ongoing saga
APIfying an ERP - ongoing saga
Marjukka Niinioja
 
Application Performance Lecture
Application Performance LectureApplication Performance Lecture
Application Performance Lecture
Vishwanath Ramdas
 
Web Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Web Performance, Scalability, and Testing Techniques - Boston PHP MeetupWeb Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Web Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Jonathan Klein
 

Similar to Scaling Web Applications with Background (20)

Deploying Machine Learning in production without servers - #serverlessCPH
Deploying Machine Learning in production without servers - #serverlessCPHDeploying Machine Learning in production without servers - #serverlessCPH
Deploying Machine Learning in production without servers - #serverlessCPH
 
Automating Workflows for Analytics Pipelines
Automating Workflows for Analytics PipelinesAutomating Workflows for Analytics Pipelines
Automating Workflows for Analytics Pipelines
 
Computerized Accounting System
Computerized Accounting SystemComputerized Accounting System
Computerized Accounting System
 
Continuous Delivery: The Dirty Details
Continuous Delivery: The Dirty DetailsContinuous Delivery: The Dirty Details
Continuous Delivery: The Dirty Details
 
Service workers and their role in PWAs
Service workers and their role in PWAsService workers and their role in PWAs
Service workers and their role in PWAs
 
On the importance of done
On the importance of doneOn the importance of done
On the importance of done
 
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
Sapphire 2013 Presentation - Streamlining SAP Transactions for Barcode Scanne...
 
Office Add-ins community call-March 2019
Office Add-ins community call-March 2019Office Add-ins community call-March 2019
Office Add-ins community call-March 2019
 
Virtualization Commputing
Virtualization CommputingVirtualization Commputing
Virtualization Commputing
 
Serverless is more findev than devops
Serverless is more findev than devopsServerless is more findev than devops
Serverless is more findev than devops
 
web, spa vs traditional - 2016
web, spa vs traditional - 2016web, spa vs traditional - 2016
web, spa vs traditional - 2016
 
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
From 10 Deploys Per Year to 4 Per Day at DBS Bank: How Pivotal Platform Can R...
 
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
PuppetConf 2017: Puppet & Google Cloud: From Nothing to Production in 10 minu...
 
Max Voloshin - "Organization of frontend development for products with micros...
Max Voloshin - "Organization of frontend development for products with micros...Max Voloshin - "Organization of frontend development for products with micros...
Max Voloshin - "Organization of frontend development for products with micros...
 
Redesigning a large B2B website - The FusionCharts revamping story
Redesigning a large B2B website - The FusionCharts revamping storyRedesigning a large B2B website - The FusionCharts revamping story
Redesigning a large B2B website - The FusionCharts revamping story
 
Water billing management system project report.pdf
Water billing management system project report.pdfWater billing management system project report.pdf
Water billing management system project report.pdf
 
LITE 2018 – A Deep Dive Into the API [Iain Brown]
LITE 2018 – A Deep Dive Into the API [Iain Brown]LITE 2018 – A Deep Dive Into the API [Iain Brown]
LITE 2018 – A Deep Dive Into the API [Iain Brown]
 
APIfying an ERP - ongoing saga
APIfying an ERP - ongoing sagaAPIfying an ERP - ongoing saga
APIfying an ERP - ongoing saga
 
Application Performance Lecture
Application Performance LectureApplication Performance Lecture
Application Performance Lecture
 
Web Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Web Performance, Scalability, and Testing Techniques - Boston PHP MeetupWeb Performance, Scalability, and Testing Techniques - Boston PHP Meetup
Web Performance, Scalability, and Testing Techniques - Boston PHP Meetup
 

More from All Things Open

Building Reliability - The Realities of Observability
Building Reliability - The Realities of ObservabilityBuilding Reliability - The Realities of Observability
Building Reliability - The Realities of Observability
All Things Open
 
Modern Database Best Practices
Modern Database Best PracticesModern Database Best Practices
Modern Database Best Practices
All Things Open
 
Open Source and Public Policy
Open Source and Public PolicyOpen Source and Public Policy
Open Source and Public Policy
All Things Open
 
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
All Things Open
 
The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil NashThe State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil Nash
All Things Open
 
Total ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScriptTotal ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScript
All Things Open
 
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?
All Things Open
 
How to Write & Deploy a Smart Contract
How to Write & Deploy a Smart ContractHow to Write & Deploy a Smart Contract
How to Write & Deploy a Smart Contract
All Things Open
 
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
All Things Open
 
DEI Challenges and Success
DEI Challenges and SuccessDEI Challenges and Success
DEI Challenges and Success
All Things Open
 
Supercharging tutorials with WebAssembly
Supercharging tutorials with WebAssemblySupercharging tutorials with WebAssembly
Supercharging tutorials with WebAssembly
All Things Open
 
Using SQL to Find Needles in Haystacks
Using SQL to Find Needles in HaystacksUsing SQL to Find Needles in Haystacks
Using SQL to Find Needles in Haystacks
All Things Open
 
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit InterceptConfiguration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit Intercept
All Things Open
 
Scaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship ProgramScaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship Program
All Things Open
 
Build Developer Experience Teams for Open Source
Build Developer Experience Teams for Open SourceBuild Developer Experience Teams for Open Source
Build Developer Experience Teams for Open Source
All Things Open
 
Deploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache BeamDeploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache Beam
All Things Open
 
Sudo – Giving access while staying in control
Sudo – Giving access while staying in controlSudo – Giving access while staying in control
Sudo – Giving access while staying in control
All Things Open
 
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsFortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
All Things Open
 
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
All Things Open
 
Building AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources codeBuilding AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources code
All Things Open
 

More from All Things Open (20)

Building Reliability - The Realities of Observability
Building Reliability - The Realities of ObservabilityBuilding Reliability - The Realities of Observability
Building Reliability - The Realities of Observability
 
Modern Database Best Practices
Modern Database Best PracticesModern Database Best Practices
Modern Database Best Practices
 
Open Source and Public Policy
Open Source and Public PolicyOpen Source and Public Policy
Open Source and Public Policy
 
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
Weaving Microservices into a Unified GraphQL Schema with graph-quilt - Ashpak...
 
The State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil NashThe State of Passwordless Auth on the Web - Phil Nash
The State of Passwordless Auth on the Web - Phil Nash
 
Total ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScriptTotal ReDoS: The dangers of regex in JavaScript
Total ReDoS: The dangers of regex in JavaScript
 
What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?What Does Real World Mass Adoption of Decentralized Tech Look Like?
What Does Real World Mass Adoption of Decentralized Tech Look Like?
 
How to Write & Deploy a Smart Contract
How to Write & Deploy a Smart ContractHow to Write & Deploy a Smart Contract
How to Write & Deploy a Smart Contract
 
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
Spinning Your Drones with Cadence Workflows, Apache Kafka and TensorFlow
 
DEI Challenges and Success
DEI Challenges and SuccessDEI Challenges and Success
DEI Challenges and Success
 
Supercharging tutorials with WebAssembly
Supercharging tutorials with WebAssemblySupercharging tutorials with WebAssembly
Supercharging tutorials with WebAssembly
 
Using SQL to Find Needles in Haystacks
Using SQL to Find Needles in HaystacksUsing SQL to Find Needles in Haystacks
Using SQL to Find Needles in Haystacks
 
Configuration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit InterceptConfiguration Security as a Game of Pursuit Intercept
Configuration Security as a Game of Pursuit Intercept
 
Scaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship ProgramScaling an Open Source Sponsorship Program
Scaling an Open Source Sponsorship Program
 
Build Developer Experience Teams for Open Source
Build Developer Experience Teams for Open SourceBuild Developer Experience Teams for Open Source
Build Developer Experience Teams for Open Source
 
Deploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache BeamDeploying Models at Scale with Apache Beam
Deploying Models at Scale with Apache Beam
 
Sudo – Giving access while staying in control
Sudo – Giving access while staying in controlSudo – Giving access while staying in control
Sudo – Giving access while staying in control
 
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML ApplicationsFortifying the Future: Tackling Security Challenges in AI/ML Applications
Fortifying the Future: Tackling Security Challenges in AI/ML Applications
 
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
Securing Cloud Resources Deployed with Control Planes on Kubernetes using Gov...
 
Building AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources codeBuilding AlmaLinux OS without RHEL sources code
Building AlmaLinux OS without RHEL sources code
 

Recently uploaded

BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALLBLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
Liveplex
 
Password Rotation in 2024 is still Relevant
Password Rotation in 2024 is still RelevantPassword Rotation in 2024 is still Relevant
Password Rotation in 2024 is still Relevant
Bert Blevins
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
Safe Software
 
7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf
Enterprise Wired
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
HackersList
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Chris Swan
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
Larry Smarr
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
ScyllaDB
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
BookNet Canada
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
KAMAL CHOUDHARY
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
Mark Billinghurst
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Bert Blevins
 
Manual | Product | Research Presentation
Manual | Product | Research PresentationManual | Product | Research Presentation
Manual | Product | Research Presentation
welrejdoall
 
Quality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of TimeQuality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of Time
Aurora Consulting
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
Matthew Sinclair
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
shanthidl1
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
SynapseIndia
 

Recently uploaded (20)

BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALLBLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
BLOCKCHAIN FOR DUMMIES: GUIDEBOOK FOR ALL
 
Password Rotation in 2024 is still Relevant
Password Rotation in 2024 is still RelevantPassword Rotation in 2024 is still Relevant
Password Rotation in 2024 is still Relevant
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
Coordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar SlidesCoordinate Systems in FME 101 - Webinar Slides
Coordinate Systems in FME 101 - Webinar Slides
 
7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf7 Most Powerful Solar Storms in the History of Earth.pdf
7 Most Powerful Solar Storms in the History of Earth.pdf
 
How Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdfHow Social Media Hackers Help You to See Your Wife's Message.pdf
How Social Media Hackers Help You to See Your Wife's Message.pdf
 
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
Fluttercon 2024: Showing that you care about security - OpenSSF Scorecards fo...
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
The Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive ComputingThe Rise of Supernetwork Data Intensive Computing
The Rise of Supernetwork Data Intensive Computing
 
Mitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing SystemsMitigating the Impact of State Management in Cloud Stream Processing Systems
Mitigating the Impact of State Management in Cloud Stream Processing Systems
 
Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024Details of description part II: Describing images in practice - Tech Forum 2024
Details of description part II: Describing images in practice - Tech Forum 2024
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Recent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS InfrastructureRecent Advancements in the NIST-JARVIS Infrastructure
Recent Advancements in the NIST-JARVIS Infrastructure
 
Research Directions for Cross Reality Interfaces
Research Directions for Cross Reality InterfacesResearch Directions for Cross Reality Interfaces
Research Directions for Cross Reality Interfaces
 
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly DetectionAdvanced Techniques for Cyber Security Analysis and Anomaly Detection
Advanced Techniques for Cyber Security Analysis and Anomaly Detection
 
Manual | Product | Research Presentation
Manual | Product | Research PresentationManual | Product | Research Presentation
Manual | Product | Research Presentation
 
Quality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of TimeQuality Patents: Patents That Stand the Test of Time
Quality Patents: Patents That Stand the Test of Time
 
20240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 202420240705 QFM024 Irresponsible AI Reading List June 2024
20240705 QFM024 Irresponsible AI Reading List June 2024
 
Cookies program to display the information though cookie creation
Cookies program to display the information though cookie creationCookies program to display the information though cookie creation
Cookies program to display the information though cookie creation
 
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptxRPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
RPA In Healthcare Benefits, Use Case, Trend And Challenges 2024.pptx
 

Scaling Web Applications with Background

  • 1. Scaling Web Applications with Background Jobs: Takeaways from Generating a Huge PDF Lydia Cupery All Things Open, 2023
  • 2. Hi! Nice to meet you 󰗝 Lydia Cupery lydia-cupery.com Lydia Cupery
  • 5. Motion design Physical Computing Download Invoices Fetch the invoice data from the database Transform data to shape of PDF data Put completed PDF somewhere client can access Write the Invoice PDF for each customer Combine all Invoice PDFs SERVER write customer invoices PDF reference to created PDF CLIENT
  • 7. Fetch the invoice data from the database Transform data Put completed PDF somewhere client can access Write the Invoice PDF for each customer Combine all Invoice PDFs
  • 8. Write the Invoice PDF for each customer Combine all Invoice PDFs
  • 9. First attempt - create one large PDF. Write each customer statement to the main pdf. It works but it takes way too long.
  • 10. Okay, what if we create all the PDFs at the same time? And then merge them? We’ll use a Promise.all and store all the individual PDFs in memory.
  • 11. What if we processed two or three invoices at a time, and write each of those as pdfs to the file system? Then, we could merge those PDFs together with pdf-lib.
  • 12. What if we used an external service? Is there any external service that could help us out with our bottlenecks? Write the Invoice PDF for each customer Combine all Invoice PDFs …perhaps for combining PDFs!
  • 13. Cuts down on the time to combine PDFs, but generating the individual PDFs and writing the PDFs to the file system still takes too long.
  • 14. Takes Too Long Generating & writing PDFs to the file systems takes too long. Generate too many pdfs in parallel, use a lot of memory, the dyno runs out of memory. Uses Too Much Memory VS
  • 15. What if we keep increasing memory, and upgrade our dynos to ones with more memory? invoice report please! report.pdf
  • 16. The system could still run out of memory and start failing as soon as more than one user tries to generate a PDF at once. invoice report please! invoice report please! ERROR please load X page! ERROR report.pdf
  • 17. Make it a Background Job!
  • 18. “Background jobs can dramatically improve the scalability of a web app by enabling it to offload slow or CPU-intensive tasks from its front-end.”
  • 19. Browser Web Server Background Service request invoices PDF schedule generate invoice PDF in-progress generate invoice PDF is it done yet? nope! is it done yet? nope! is it done yet? yes! here it is
  • 21. web process (web dyno) data store (redis) library to implement queue system on top of redis (BullMQ) worker processes (worker dynos)
  • 22. With a Background Job You Can…
  • 23. With background jobs you can… Speed Things Up
  • 24. Browser Web Server Background Service request invoices PDF schedule generate invoice PDF in-progress generate invoice PDF is it done yet? nope! is it done yet? nope! is it done yet? yes! here it is
  • 25. Sorted Customers with Invoices fetch invoice data generate invoices customerIds: A-E fetch invoice data generate invoices customerIds: F-O fetch invoice data generate invoices customerIds: P-Z combine generated invoices JOB QUEUE customers A-E customers F-O customers P-Z customer list/ WORKER_COUNT
  • 26. generate invoices customerIds: A-E generate invoices customerIds: F-O generate invoices customerIds: P-Z combine generated invoices JOB QUEUE fetch data (A-E) write invoices upload “batch invoices xxx - 1” fetch data (F-O) write invoices upload “batch invoices xxx - 2” fetch data (P-Z) write invoices upload “batch invoices xxx - 3” combine s3 files to “batch invoices xxx” upload “batch invoices xxx”
  • 27. Job Partial Job Partial Job Partial Job Combine Partial Job Outputs output output output
  • 28. With background jobs you can… Show Progress
  • 29. Progress Indication With no background job (assuming the server doesn’t time out or run out of memory) : Browser Web Server request invoice PDF here it is! generate pdf…
  • 31. Browser Web Server Background Service request invoices PDF schedule generate invoice PDF in-progress generate invoice PDF jobs is it done yet? nope! 30% there is it done yet? nope! 70% there is it done yet? yes! here it is
  • 32. Generate Partial Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) JOB PROGRESS 1 .1
  • 33. Generate Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) ● generate the invoices ○ updateProgress each time an invoice generated JOB PROGRESS 1 .2
  • 34. Generate Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) ● generate the invoices ○ updateProgress each time an invoice generated JOB PROGRESS 1 .4
  • 35. Generate Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) ● generate the invoices ○ updateProgress each time an invoice generated JOB PROGRESS 1 .6
  • 36. Generate Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) ● generate the invoices ○ updateProgress each time an invoice generated JOB PROGRESS 1 .8
  • 37. Generate Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) ● generate the invoices ○ updateProgress each time an invoice generated ● updateProgress(0.9) JOB PROGRESS 1 .9
  • 38. Generate Invoices Job (input, updateProgress) => ● fetch data to generate invoices ● updateProgress(0.1) ● generate the invoices ○ updateProgress each time an invoice generated ● updateProgress(0.9) ● upload generated file to s3 ● updateProgress(1) JOB PROGRESS 1 1
  • 39. view job progress updateProgress(X) updates job progress aggregate progress across jobs requests job progress
  • 40. With background jobs you can… Support Simultaneous Users
  • 41. Before background jobs… invoice report please! invoice report please! ERROR please load X page! ERROR report.pdf
  • 42. invoice report please! please load X page! working on that! progress? progress? invoice report please! working on that! progress? here you go!
  • 43. With background jobs you can… Save Jobs for Later
  • 44. invoice report please! working on that! progress? progress? invoice report please! working on that! progress?
  • 45. With background jobs you can… Have Less Timeouts/Errors
  • 46. Communicating with an External Service With no background job: Browser Web Server send out customer emails, please! TIMEOUT ��🏻♀ Mailgun send these emails, please! ��
  • 47. Browser Web Server send out customer emails, please! send these emails, please! in-progress Mailgun is it done yet? nope! 30% there is it done yet? nope! 70% there is it done yet? yes - all emails are sent! �� Background Service send these emails, please!
  • 48. Recap - With a background job you can… speed things up show progress support simultaneous users have less timeouts/errors save jobs for later
  • 49. Should you use a background job?
  • 50. You might want a background job for… CPU-intensive jobs Jobs communicating externally I/O intensive jobs Scheduled jobs
  • 51. Tips
  • 52. Struggling with app responsiveness? Try a background job.
  • 53. Don’t recreate the wheel. Use a library with a robust queueing system.
  • 54. Does speed matter? It probably does. Parallelize.
  • 55. The job queue makes it easy to show users progress. Do so!
  • 56. Find the optimal number of workers and optimal amount of resources per worker(see next slide…)
  • 57. 1X 1X 1X $25 x 6 = $150 512 MB x 6 = 3GB $50 x 3 = $150 1 GB x 3 3GB 1X 1X 1X 2X 2X 2X $250 x 1 = $250 2.5 GB x 1 = 2.5 GB Perf M
  • 59. Fetch the invoice data from the database Transform data to shape of PDF data Put completed PDF somewhere client can access Write the Invoice PDF for each customer Combine all Invoice PDFs
  • 60. Fetch list of customers with invoices from the database Transform fetched data to shape of PDF data Put completed PDF somewhere client can access Write the Invoice PDF for each customer Combine all Invoice PDFs do not need to fetch all invoice data Fetch invoice data for customers fetching 1/10 amount of data transforming 1/10 amount of data writing PDF for 1/10 customers worker dyno worker dyno