Job scheduler (cron)

What is a job scheduler?

A job scheduler is software with one task: start other work at a moment you fixed in advance. A nightly export at 02:00, a reminder run every weekday at 06:00, a sync with the webshop every ten minutes. You give it a time and a command, and it keeps the appointment.

Cron is the original, and old enough that it set the vocabulary everyone still uses. It runs in the background on Unix and Linux systems and reads a table called a crontab, each line a time pattern followed by a command. Everything since borrowed the shape: the recurrence trigger behind a scheduled cloud flow, the timer trigger in Azure Functions, a SQL Server Agent job, the schedule field on an Airflow DAG.

One property causes most of the surprises. In its plain form a scheduler knows the clock and nothing else: not whether the previous run finished, not whether the source system is up, not whether yesterday's file arrived. It starts the job because it is 06:00, and the rest is your design.

Reading a cron expression

Five fields, separated by spaces, in a fixed order: minute, hour, day of month, month, day of week. POSIX sets the ranges at minute 0 to 59, hour 0 to 23, day of month 1 to 31, month 1 to 12, and day of week 0 to 6 with 0 as Sunday. Linux cron also accepts 7 for Sunday and names such as Mon. An asterisk means every value, a comma makes a list, a hyphen a range, a slash a step.

So 0 6 * * 1-5 reads as: minute 0, hour 6, every day of the month, every month, Monday through Friday. Six in the morning on weekdays. One more to practise on: */15 * * * * is every fifteen minutes.

A trap in the standard catches almost everyone. Fill in both day of month and day of week and cron combines them with OR, not AND. The crontab manual's own example, 30 4 1,15 * 5, runs at 04:30 on the 1st and the 15th, and on top of that every Friday. Cron cannot say the 15th only when it is a Friday, so that condition goes inside the job. Count the fields too: Azure Functions uses NCRONTAB, which adds seconds at the front, so 0 30 9 * * 1-5 is 09:30 on weekdays.

Questions a schedule raises once it is running

Which clock is 06:00?
Server local time, UTC, or a named time zone. Pick the named zone: it is the only one that survives the daylight saving change on its own. Azure Functions runs expressions in UTC unless you set WEBSITE_TIME_ZONE, and Airflow follows the zone you name, so 0 0 * * * in US/Eastern runs at 04:00 UTC during daylight saving time and at 05:00 otherwise. Classic cron only matches the wall clock, and its manual is blunt: jobs in the missing hour never run, jobs in the repeated hour run twice. Keep the 02:00 to 03:00 window empty.

What if the previous run is still going?
The clock fires again regardless, and two copies of the same import writing to the same table is a data problem, not a speed problem. Some platforms prevent it: the Azure Functions timer trigger runs on one instance even when the app scales out, and does not fire again while an invocation is running. If yours does not, have the job take a lock at the start and exit when it cannot get one.

What if nothing was running at 06:00?
Skip or catch up, and you need to know which one you have. The Logic Apps Recurrence trigger skips the missed moments and picks up at the next interval. Airflow catches up with catchup=True, creating a run for every interval that has not run since the last one, and ships with catchup_by_default=False. Catch up when the work belongs to a period, such as a daily aggregate. Skip when the work is about the current state, such as a stock alert, where yesterday's warnings help nobody.

What if it fails at 03:00?
A scheduler starts things, it does not watch them. The Azure Functions timer trigger does not retry after a failure: the function is called again at the next scheduled time. Decide per job whether it retries itself after a delay or whether the next run picks up the leftover work, which only holds when the job is safe to repeat.

How do you know it ran at all?
Alerting on failure only catches jobs that ran and failed. When the scheduler itself is stopped, nothing fails and nothing alerts, and you hear it from a customer asking where the report is. Add an alert on absence: the job reports that it finished, and a separate check raises the alarm when that signal has not arrived by 06:30. It is the control that catches silence, and usually the one missing.

A nightly chain that breaks

A wholesaler has four jobs, each scheduled an hour after the one before: export yesterday's orders from the ERP at 02:00, load that file into the data warehouse at 03:00, run the transformations at 04:00, refresh the Power BI report at 06:00.

This works for a year. Then order volume doubles, the export goes from forty minutes to eighty, and it finishes at 03:20. The 03:00 load still finds a file, reads it and succeeds, because last night's file is sitting where it always sits. At 06:00 the report refreshes with a full day missing and all four jobs report success. Nothing failed, so nothing alerted. Those hourly gaps were never a design: they were a guess about how long each step takes, written into the clock, and the guess gets worse as the data grows.

Scheduler or orchestrator: what starts the next job?

With a scheduler, the clock starts every job. Four schedules are four separate appointments, the load starts at 03:00 whether the export finished or not, and neither job knows the other one exists.

With an orchestrator you keep one schedule for the chain, at 02:00, and the previous job finishing is what starts the next. The export ending at 03:20 starts the load at 03:20. If the export fails, the load never starts, the report keeps yesterday's numbers instead of publishing a half-empty day, and you get one alert naming the step that failed. Airflow, Dagster and Fabric Data Pipelines are the usual tools, and the data orchestration entry covers how they model dependencies.

One job on a clock is a schedule. Three jobs where each waits for the previous is a dependency in disguise, and you have found one the moment you widen a gap because a step got slower.

What to watch out for with scheduled jobs

Nobody knows what runs at night
Write down every scheduled job in one list: what it does, the expression and its time zone, where it runs, who owns it, and what breaks if it does not run. That takes an afternoon. Most lists turn up a job owned by somebody who left, and the same list is what makes a server migration survivable.

Frequency is a bill
A trigger every ten minutes is 144 runs a day and more than four thousand a month before anything useful has happened. For AI work each wake-up is a round of model calls, which the ambient agent entry works through, and Microsoft warns that extra executions on a consumption plan can cost significantly more. Hourly is usually enough, and when it is not, you want an event trigger rather than a shorter interval.

The job on somebody's laptop
Windows Task Scheduler on a PC under a desk is a real scheduler, with a real dependency on that PC being awake and that colleague still working here. If the job matters, it belongs on a server.