Batch Processing System

A Batch Processing System is a data processing system that performs work in batches (executing programs without manual intervention).

• AKA: Offline Processing System, Batch Computation System.
• Context:
  • It can (typically) require Batch Processing Elements, such as:
    • It can need input data collection into processing batches.
    • It can involve job scheduling for batch execution.
    • It can manage resource allocation for batch operations.
  • It can (typically) provide Batch Functions, such as:
    • It can execute batch jobs through automated sequences.
    • It can process data batches through predefined workflows.
    • It can generate batch outputs through scheduled operations.
  • It can (often) handle Batch Challenges, such as:
    • It can manage processing load through workload distribution.
    • It can ensure job completion through error recovery.
    • It can optimize resource usage through batch scheduling.
  • It can range from being a Simple Batch System to being a Complex Batch System, depending on its processing complexity.
  • It can range from being a Sequential Batch System to being a Parallel Batch System, depending on its execution model.
  • It can range from being a Single-Node Batch System to being a Distributed Batch System, depending on its system architecture.
  • ...
• Example(s):
  • Enterprise Batch Systems, such as:
    • ETL Processing Systems, such as:
      • Nightly Data Load for data warehouses.
      • Periodic Data Integration for business intelligence.
    • Report Generation Systems, such as:
      • Monthly Financial Report for accounting systems.
      • Weekly Analytics Summary for business metrics.
  • Big Data Batch Systems, such as:
    • Hadoop-based Systems, such as:
      • MapReduce Cluster for large-scale processing.
      • Hive System for data warehouse queries.
    • Spark-based Systems, such as:
      • Spark Batch Job for in-memory processing.
      • Data Lake Processing for large datasets.
  • Scientific Batch Systems, such as:
    • Simulation Processings, such as:
      • Weather Model Computation for forecast generation.
      • Molecular Dynamics Simulation for scientific research.
    • Data Analysis Batches, such as:
      • Genomic Data Processing for sequence analysis.
      • Astronomical Data Analysis for sky surveys.
  • Industrial Batch Systems, such as:
    • Manufacturing Processes, such as:
      • Production Schedule Execution for factory automation.
      • Quality Control Analysis for product testing.
    • Maintenance Systems, such as:
      • System Backup Process for data protection.
      • Log Analysis Batch for system monitoring.
  • ...
• Counter-Example(s):
  • Online Processing Systems, which handle immediate requests.
  • Interactive Systems, which require user interaction.
  • Real-Time Systems, which process continuous data streams.
  • Transaction Systems, which handle individual transactions.
• See: Batch Processing, Job Scheduling, Workload Management, Resource Allocation, Processing Pipeline.

References

2014

• (Wikipedia, 2014) ⇒ http://en.wikipedia.org/wiki/batch_processing Retrieved:2014年1月23日.
  • Template:More footnotes

    Batch processing is execution of a series of programs ("jobs") on a computer without manual intervention.

    Jobs are set up so they can be run to completion without manual intervention. So, all input data are preselected through scripts, command-line parameters, or job control language. This is in contrast to "online" or interactive programs which prompt the user for such input.

    A program takes a set of data files as input, processes the data, and produces a set of output data files. This operating environment is termed as "batch processing" because the input data are collected into batches of files and are processed in batches by the program.

• Concept
• Processing System
• Batch Processing
• Quality Silver