Background Tasks Without a Separate Service: Hangfire for ASP.NET - KCDC - July 2019

Editor's Notes

1. Stack Overflow question about threading, it's about threading, but I think it's relevant when it comes to background processing Just celebrated the 10 year anniversary of this sarcastic, worthless, but accepted answer
2. Time based job scheduling "cron" is a unix command In windows the equivalent is a "scheduled task" Cloud providers have similar capabilities: Azure Scheduler, AWS Scheduled Tasks, and so on Cron itself has a specific syntax, but often when people say "cron job" they mean "execute some process on a schedule" Cron is tried and true and stands the test of time, possibly one of the best pieces of software ever created. So I can't possibly say that you shouldn't use cron or scheduler But the issue is that it will typically require a separate process to be created/deployed, which can complicate your deployment You could have cron just hit an HTTP endpoint in your ASP.NET application, but this could tie up HTTP and even a dirt simple script like "cron curl" is still a separate thing that has to be remembered, deployed, updated, separately from your asp.net app Also, with cron, it's just a scheduler, it has no state. It can't do fire-and-forget, continuations, retries, etc by itself
3. Another approach is to use a message broker, a queueing system, a service bus, the actor model, pub/sub, etc These are tools that are meant to pass messages around between processes and/or distribute work amongst multiple nodes/worker processes And once again, these are fine pieces of software being used to great effect. These tools all have state, so they're able to do things like retries. They support various models of scaling. There's lot of monitoring tools and ecosystems around these. These are fine tools, and I can't tell you that you shouldn't use them either! But these are relatively heavy solutions. Rabbit and Kafka require deployments of their own, the others might as well. May require implemented specific interfaces, base classes, APIs, that require some coupling. Again, for good reason, because these are often meant to support complex scenarios, multiple-state background processing, long running multi-step processes, broadcasting, etc.
4. One or more of these drawbacks
5. This is the elevator pitch for hangfire It addresses the drawbacks of the other methods And carves out its own niche in background processing Easy to get started for simple tasks, easy to scale for handling a lot of them
6. The client and server can be in the same application
7. The way it scales is going to look familiar to many of you. It's similar to how you would scale session with ASPNET As you scale out your asp.net application, hangfire will scale with it IF YOU WANT, you can also deploy servers individually Hangfire is NOT coupled to asp.net, it can run in a console app or windows service too For Job Storage, default hangfire uses SQL Server BUT there are extensions to use other databases for job storage This is where Couchbase can come in, provide faster access to job data AND can easily scale horizontally for massive job processing. And by the way Couchbase can handle session storage, caching, user profile, etc.
8. Hangfire ships with a built-in UI Dashboard to show a graph of job history List of recurring jobs View job history Jobs can be manually started List of servers
9. Fire off a background task that will run IMMEDIATELY and ONLY ONCE Some background process that sends an email Or executes against a slow API Something that you don't want or need to block for Notice how you use Hangfire here: BackgroundJob.Enqueue and you give it a C# expression You don't need to implement an IJob interface or anything like that. Don't take this as a challenge, but Hangfire has not choked on any expression that I've thrown at it SO FAR Also notice that it returns a jobId, because….
10. You can create a continuation. These will execute after a given job is finished. So, let's think about an amazon purchase. The first job might be to verify inventory. The next job would be payment processing. The next job would be shipping. The next job would be email notification. Etc. It makes sense to execute in the background, but it doesn't make sense to execute in parallel. So, continuations.
11. Recurring task, which is mainly what I use hangfire for You want to run some code every night Update a cache record every minute Process Azure Cognitive services every couple of hours Again, it's just RecurringJob, AddOrUpdate (which you would run on startup, usually) Supply a C# expression And then supply a Cron expression. Literally the same cron syntax. There are some convenience methods like Cron.Daily, but it's just returning a string
12. Delayed jobs: send an email reminder in 7 days Batches: a group of background tasks created atomically that are considered a single entity Batch continuations: continuations, but after a batch The paid version of Hangfire, Hangfire Pro
13. Exceptions can happen in a background task too Hangfire will log these exceptions, and will retry up to 10 times by default with an increasing delay You can view some of the exception information You can re-queue the job Retry up to 10 times automatically If you DON'T want a retry, you can specify that with a C# attribute or you can specify it globally.
14. ASP.NET applications, depending on how you deploy them, may not start until the first user hits your site, so hangfire may not start right away And the processes may be "recycled" after a period of inactivity For an on-premises site, you will need to do some additional configuration to keep the process running For ASP.NET Core, if you run them in startup, recurring jobs should start right away For Azure, one workaround is the "Always On" setting, which is not available for free sites You can always fall back to a separate process. Or you can setup a cron job to ping your site.
15. Localhost only access by default If you want anyone not on localhost to have access, you have to create an authorization filter
16. They say you only remember 3 things from any presentation, so here they are
17. Easy to get running with rich background tasks
18. Easy to scale horizontally, just like Couchbase
19. Cron is everywhere, but here's one tutorial about the syntax Topshelf is a service hosting framework for .net There are a lot of options, Hanselman did a pretty good rundown back in 2014
20. Hangfire dot io, great documentation There are a ton of hangfire extensions Hangfire.Couchbase project, which gives you the ability to use Hangfire with Couchbase Hangfire pro, which comes with more features, dedicated support, priced VERY reasonably
21. If anything looks interesting to you, you have questions or feedback, come talk to me afterwards I want to hear from you! My boss says I have to listen to you, it's my job. So now's your chance :) ----------- * Background jobs are processed by a dedicated pool of worker threads that run inside Hangfire Server subsystem, You can specify the number of worker threads * You don’t want to consume additional Thread Pool threads with background processing – Hangfire Server uses custom, separate and limited thread pool. * You are using Web Farm or Web Garden and don’t want to face with synchronization issues – Hangfire Server is Web Garden/Web Farm friendly by default. * Hangfire can process multiple queues. If you want to prioritize your jobs or split the processing across your servers