Scheduling, Batch, or Continuation

Batch or Continuation Jobs

Batch jobs is not supported in Shift, and no further plan to do so. However this batch type of jobs can be done in a single job.

public class BatchMail 
{
    public static void SendAll(int number)
    {
        var mailList = new List<int>();
        for (var i=0; i < number; i++)
        {
            var mailID = GenerateMail(i); //Generate your mail and return a unique mailID
            mailList.Add(mailID);
        }
        //All mails successfully generated
        SendMails(mailList); //Send all the mails at once to the mailserver
    }
}

//Use the Shift client to add the `BatchMail.SendAll` job to Shift server queue
shiftClient.Add( () => BatchMail.SendAll(100) );

The same workaround to batch type jobs can also be used for continuation or chain type jobs, such as task.ContinueWith(...). Wrap a continuation jobs in one runnable job and send to Shift.

Scheduling and Recurring Jobs

Shift does not support scheduling or recurring jobs. Shift is not a cron manager or scheduler. The best practice is to use another library, such as NCron to schedule adding jobs to Shift.

Shift has a goal to allow predictable maximum amount of work among Shift server processes, using the setting MaxRunnableJobs, we know each Shift server can run at most 100 jobs by default. For jobs which must run successfully at the scheduled time makes it much harder to set limit on the Shift server. If the scheduled jobs run despite there are maximum processes running, then the maximum load becomes unpredictable. If we don't run scheduled jobs when maximum limit is reached then the scheduled jobs become unpredictable and unreliable.

Jobs with specific scheduled time generally need to be "scheduled" to run by a single global scheduler or cron manager. Here is the general workflow:

• User schedule several jobs.
• Scheduler through polling schedule jobs to run.

With multiple Shift servers, this workflow can be a problem:

• User schedule several jobs.
• Two Shift servers are running. Which Shift server should schedule and run the jobs? For example the first server through polling retrieve most of the scheduled jobs. The first server will schedule and run most of the jobs and the 2nd maybe by chance pick up few left over scheduled jobs not claimed by the 1st server. This violates a predictable and good distribution of works goal, also one Shift server process will always be overworked.

Putting a maximum scheduled jobs limit to each Shift servers can resolve the distribution problem, but it introduces a reliability problem because the scheduled jobs must run. Putting limit means some scheduled jobs will not be picked up and scheduled to run.

The workaround to scheduled or recurring jobs is to use NCron to schedule jobs, including recurring ones. Example below is adapted from NCron introduction wiki.

public class MyJobCron : NCron.CronJob 
{ 
    public override void Execute() 
    { 
        shiftClient.Add("My Job", () => MyJob.Run());  //When cron job is executed, it calls the static Shift client to run My Job in Shift server.
    } 
}

class Program 
{ 
    static void Main(string[] args) 
    { 
        Bootstrap.Init(args, ServiceSetup); //Bootstrap the Cron manager
    }

    static void ServiceSetup(SchedulingService service)
    {
        service.Hourly().Run<MyJobCron>(); //Send MyJob.Run() to Shift every hour
    }
}