PersistentJobs

Does not work yet, don't try.

Background jobs stored in application's Entity Framework Core database, without external services.

Goals

• Uses Entity Framework Core for storing the jobs
• No external services (e.g. Kafka, MQTT)
• Tiny library with dependencies to .NET framework
• Not used for performance sensitive tasks
• Queue processing can be triggered automatically in hosted service
• Queue processing can be triggered manually with API endpoint

Use case examples

• Email queue with retry mechanism and max parallelization
• Resize images on background
• Send invoice at given date and time
• Run cleanup commands on defined cron schedule
• Check failed payments in subscription system

Defined behavior

Two types of persistent jobs:

• Deferred jobs
  • Single static method call
  • Takes optional input value and injected services
  • Ability to retry
  • Wait between failures
  • Max parallelization per method
  • Schedule execution after certain timestamp
  • Atomic: Can be queued within database transaction
  • Ignores methods not defined in the source
  • Does not delete non-existing deferred methods
• Cron jobs
  • Hourly, daily, minutely, and with custom function
  • Cron job works by creating deferred jobs on demand
  • All deferred jobs can be cron jobs
  • Can be dedfined in code and manually in database
  • Can be disabled from database
  • Source defined cron jobs are recreated on start
  • Ignores cron jobs not defined in the source
  • Does not delete non-existing cron jobs

How to define a job

1. Partial class if you use source generators
2. Single public static method for your work
3. First parameter optionally named "input"
   • Input is serialized to the database
4. Second, third, fourth etc. parameters are injected services
5. Last parameter can be CancellationToken
6. Decorate the method with [CreateDeferred] or with [Deferred]

public static Task SendEmail(
    string input, // Serialized to database, must be named "input"
    IEmailSender sender, // Injected service 1.
    DbContext context, // Injected service 2., ...
    CancellationToken cancellationToken = default
)
{
    // Your code...
    return Task.CompletedTask;
}

Usage with source generator attribute CreateDeferred

When decorated with [CreateDeferred] it generates a corresponding public static method with Deferred suffix:

public partial class Worker
{
    [CreateDeferred]
    public static Task SendEmail(
        string input,
        IEmailSender sender,
        CancellationToken cancellationToken = default
    )
    {
        // Your code...
        return Task.CompletedTask;
    }

    // Following method is generated by the source generator:
    public static Deferred SendEmailDeferred(
        string input,
        DbContext dbContext
    )
    {
        // Returns `Deferred` which allows to query is it ready? What is the
        // output value? What are the exceptions? It does not allow to await
        // for the task to finish.
    }

}

Generated method stores the call to the database, in effect deferring it's execution until the DeferredQueue executes it.

It's notable that it omits the IEmailSender and other paramters in original method, because it replaces the function with a version which just stores the task in the database.

Usage with plain Deferred attribute

It's also rather easy to use deferred execution without the source generator. One only need to call DeferredQueue.Enqueue with public static method decorated with DeferredAttribute, e.g.

public class Worker
{
    [Deferred]
    public static Task SendEmail(
        string input,
        IEmailSender sender,
        CancellationToken cancellationToken = default
    )
    {
        // Your code...
        return Task.CompletedTask;
    }

    public static Deferred SendEmailDeferred(
        string input,
        DbContext dbContext
    )
    {
        DeferredQueue.Enqueue(dbContext, SendEmail, input);
    }

}

Notes

• Stream success or exceptions from Deferred?
• Channel for incoming Deferred tasks instead of polling?
• The signatures are a bit odd as they work with given DbContext. This is intentional as the caller should handle transactions and saving the changes themselves. It means though that repositories are also odd, as they don't own the DbContext, but merely uses the given ones.