Background jobs and cron for Python without Redis or RabbitMQ
Run scheduled tasks and background workers in Django apps, scrapers, and CLIs using only SQLite—no broker, no daemon, zero dependencies.
- LinkLoot access
- Free
- Provider costs
- Unknown
What you get from it
What it does
tinyjobs brings Celery-style task queuing to Python projects that don't need the infrastructure overhead. Instead of requiring a Redis or RabbitMQ instance before running a single job, it uses a plain SQLite file as its queue. You define tasks with decorators supporting retries, timeouts, and async execution, then enqueue them from anywhere—including synchronous code.
The library handles the lifecycle: queued → running → success or failure. When you need current status, call job.refresh() explicitly; attribute access never triggers I/O silently. Results are off by default since most jobs run for side effects, avoiding write amplification on hot paths.
Terminal jobs get pruned automatically. Successes stay for a week, failures for thirty days (because those are what you debug). Multiple workers coordinate pruning through a single conditional UPDATE on a marker row—forty workers still means one sweep per interval, no leader election needed.
Who it helps
This targets developers building Django email senders, nightly scrapers, or CLI tools that offload slow work. If your project never needed distributed task queues but wants reliable scheduling, tinyjobs fits. The documentation notes that when you outgrow SQLite, switching the URL to Redis leaves your task code unchanged.
Getting started
Install via pip, then initialize with a SQLite path:
app = TinyJobs("sqlite:///jobs.db")
Define tasks with decorators like @app.task(max_attempts=5, timeout=300). Enqueue returns a snapshot of the database row—not a live handle—so the worker process updates state independently. Use job.wait(timeout=30) to block until completion, or await job.awaited(timeout=30) from async contexts.
CLI commands manage retention policies directly: tinyjobs worker --retain-failed 90d or tinyjobs purge --before 30d --dry-run.
Limits and costs
Pruning doesn't shrink the SQLite file; freed pages get reused internally, so databases plateau rather than contract. Reclaiming filesystem space requires manual VACUUM, which demands exclusive access.
MIT-licensed open source. No pricing information provided in the packet.
Source links
Discussion
Share practical experience, questions, or warnings with the community.
Sign in to join the discussion and vote on comments.
Sign in