Skip to content

Mining Job API

The Job API provides the backbone for executing specific operations as jobs that can parallelize operations into different tasks. When the Mining API Server is set up as a cluster, these tasks are automatically distributed for parallel execution.


Configuration

The following excerpt of a Mining API Server application configuration YML file shows the default job-api configuration. Use it as the baseline for standalone setups before enabling cluster-specific behavior. The property descriptions below explain which values are required for naming, logging, and shared job result storage.

1
2
3
4
5
6
7
8
9
job-api:
    cluster:
        name: myName
        mode: standalone
        nodeNameStrategy: hostname
    log:
        logFolder: logs
        logFilePrefix: job-
    jobResultFolder: jobResults

The following configuration properties are available for the Spring application:

Parameter

Description and Values

name

The name that should be assigned to the cluster. Only mining-api-server instances having the same cluster name are eligible to join a cluster.

mode

The actual execution mode. The default value is standalone.
Valid values are: 

  • standalone: This is the default setting when not explicitly providing a mode and will not create any cluster.

  • default: The default cluster mode, which can be used to create a cluster based on the TCP-IP or Multicast configuration.

  • aws: The cluster mode must be used together with the AWS-specific configuration to create a cluster on AWS.

nodeNameStrategy

The strategy is to assign a name to the cluster node, which can also be accessed in the log configuration. The default value is hostname.
Valid values are:

  • random-uuid: Every application startup will assign a new random UUID as the node name.

  • internal-node-id: Assigns the Hazelcast member node ID, which will change with every application startup.

  • ip: Uses the primary outbound IP address of the system.

  • hostname: The default strategy uses the hostname of the local system.

  • ec2-instance-id: When running on EC2, this uses the EC2 instance ID.

logFolder

Configures the base directory where all log files should be written to. Defaults to the logs directory where Java is currently running.

logFilePrefix

The prefix for all job-specific log files. The default value is job-.

jobResultFolder

Configures the base directory where the result files of Jobs are written to. Defaults to the jobResults directory where Java is currently running.

Warning

The logFolder and jobResultFolder must be accessible by all nodes of a cluster. Therefore, when running in clustered mode, these two properties must point to a shared file storage like an EFS on AWS.

Info

All properties with the exception of name are optional.

Multicast

Multicast is the easiest approach to create a cluster, but it may not be possible in all network environments.

job-api:
    cluster:
        name: my-cluster
        mode: default
        network:
            port: 5710
            portAutoIncrement: true
            multicast:
                enableLoopback: true
                group: 224.2.2.3
                port: 54327
                connectionTimeout: 2
                ttl: 32
                trustedInterfaces: 10.15.169.36,10.27.248.120

Parameter

Description and Values

port (network)

The local port to use for the current process. Defaults to 5701 when not provided.

portAutoIncrement

When set to true the port will be automatically incremented until reaching one that is not already in use. Otherwise startup will fail. Defaults to true when not provided.

enableLoopback

If this is set to true then the loopback interface of the machine will be treated as a valid network interface during the node discovery. Defaults to false when not provided.

group

The address on which to send and receive multicast packets. Defaults to 224.2.2.3 when not provided.

port (multicast)

The port on which to send and receive multicast packets. Defaults to 54327 when not provided.

connectionTimeout

The time in seconds a node will wait for a multicast response from a different node before assigning itself to a master of its own cluster. Defaults to 2 when not provided.

ttl

The TTL of the multicast packets. Every hop between different networks decrements this value. Defaults to 32 when not provided.

trustedInterfaces

A comma-separated list of IPs that are eligible for the multicast lookup. When not provided, all machines available in the network are eligible to be included in the cluster.

Info

All mining-api-server instances with the same name property (for example, "my-cluster" as above) will be included in the same cluster by the multicast discovery process.

TCP-IP

In most network environments, the TCP-IP configuration is recommended if multicast is not available.

job-api:
    cluster:
        name: my-cluster
        mode: default
        network:
            port: 5710
            portAutoIncrement: true
            tcpIp:
                connectionTimeout: 5
                members: 10.15.169.36,10.27.248.120

Parameter

Description and Values

port

The local port to use for the current process. Defaults to 5701 when not provided.

portAutoIncrement

When set to true the port will be automatically incremented until reaching one that is not already in use. Otherwise startup will fail. Defaults to true when not provided.

connectionTimeout

The time in seconds to wait for a response from one of the configured cluster members before giving up. Defaults to 5 when not provided.

members

A comma-separated list of IPs of all members that should participate in the cluster.

AWS

Creating a cluster on AWS is supported for EC2 machines (with and without Auto Scaling).

job-api:
    cluster:
        name: my-cluster
        mode: aws
        network:
            port: 5710
            portAutoIncrement: true
            aws:
                region: myRegion
                portRange: 5710-5715

Parameter

Description and Values

`accessKey`
`secretKey`

Access and secret keys of your AWS account. If not set, iamRole is used.

`region`

The region where members are running. Default is the current region.

`hostHeader`

Either ec2, ecs, or the URL of an EC2/ECS API endpoint. Automatically detected by default.

`portRange`

A range of ports to locate members in the format <lower-port>-<upper-port>. The default value is 5701-5708.

`connectionTimeout`
readTimeout

The connection and read timeouts in seconds when making a call to the AWS API. The default value is 10.

`connectionRetries`

Number of retries while connecting to the AWS API. The default value is 3.

Info

All of the AWS and EC2 configuration properties are optional. If none of them have been provided, then the IAM role assigned to the EC2 instance will be used to form a cluster with all members running in the same region as that instance.

EC2

The IAM role needs to have the ec2:DescribeInstances permission to be able to form a cluster.

job-api:
    cluster:
        name: my-cluster
        mode: aws
        network:
            port: 5710
            portAutoIncrement: true
            aws:
                portRange: 5710-5715
                ec2:
                    tagKey: my-ec2-instance-tag-key
                    tagValue: my-ec2-instance-tag-value

The following are EC2-specific properties, which must be contained in a ec2 subsection (all of these are optional):

Parameter

Description and Values

`iamRole`

The IAM role attached to the EC2 instance is used to fetch credentials if accessKey/secretKey is not specified. If not set, the default IAM role attached to the EC2 instance is used.

securityGroupName

The filter to identify only EC2 instances with the given security group.

`tagKey`
`tagValue`

The filter to identify only EC2 Instances with the given tagKey/tagValue.

Info

All of the AWS and EC2 configuration properties are optional. If none of them have been provided, then the IAM role assigned to the EC2 instance will be used to form a cluster with all members running in the same region as that instance.

EC2 Auto Scaling

The Job API is fully integrated into the AWS EC2 Auto Scaling life cycle.

job-api:
    cluster:
        name: my-cluster
        mode: aws
        network:
            port: 5710
            portAutoIncrement: true
            aws:
                portRange: 5710-5715
                ec2:
                    tagKey: my-ec2-instance-tag-key
                    tagValue: my-ec2-instance-tag-value
                    autoscaling:
                        lifecycleQueueName: my-lifecycle-queue
                        activeJobsTimeout: 60
                        memberShutdownTimeout: 600

In addition to the default EC2 configuration properties mentioned above, the following additional parameters can be adjusted for Auto Scaling:

Parameter

Description and Values

`lifecycleQueueName`

The name of the SQS (Simple Queue Service) queue that will receive the life cycle hook notifications of the Auto Scaling group.

Info

This is a required parameter for Auto Scaling.

activeJobsTimeout

When the Job API receives a machine termination notification from the SQS queue, it will wait for this duration (in seconds) for all currently running jobs on this instance to finish, while no longer accepting new ones. If there are still active jobs after the timeout, the AWS Autoscaler will be notified to keep the EC2 instance running, transitioning it back into service, and it will also accept new jobs again. Please note that the value chosen here should be smaller than the default timeout of the life cycle hook attached to the Auto Scaling group in AWS.

The default is 60 seconds, which is also the minimum possible value.

membersShutdownTimeout

During the shutdown process, when no jobs are active anymore on this member, it will gracefully detach itself from the cluster. If this takes longer than the configured timeout (in seconds), it will notify the AWS Autoscaler to transition the instance back into service. Please note that when choosing too small a value, the EC2 instance will transition back into service, but can no longer accept any jobs or tasks, as Hazelcast is already in a shutdown state.

The default is 600 seconds, with 60 being the minimum possible value.

Info

The lifecycleQueueName parameter is a required non-optional parameter for Auto Scaling to avoid compromising the cluster integrity and losing data.

Logging

The aforementioned configuration properties logFolder and logFilePrefix can be used to influence the log behavior.

The Job API provides multiple system properties, pattern parameters for trace logging, and context variables that can be used in the log4j configuration. These are:

System Property

Description

`job-api.cluster.nodeName`

Contains the resolved name of the cluster node as resolved by the configured nodeNameStrategy .

`job-api.log.logFolder`

Contains the final resolved log directory, which depends on the mode configuration property. In standalone mode, it will contain the raw value as provided in the logFolder configuration property, while in any cluster mode, is comprised of the logFolder configuration property and the cluster node name <logFolder-property-value>/<resolved-node-name>.

`job-api.log.logFilePrefix`

Contains the value of the logFilePrefix configuration property as-is. 

Pattern Parameter

Description

`traceId`

Provides the ID of the current active tracing.

`spanId`

Provides the ID of the current active span.

`parentId`

Provides the ID of the parent span.

Context Variables

Description

`job-id`

Provides the ID of the job or task that triggered the log output.

Info

All system properties are populated at the earliest possible time when Spring actually creates the beans of the Job API. They are therefore not yet present during the Spring core initialization.

Configuration

When running in a clustered environment, the logFolder configuration property should point to a shared file storage like an EFS on AWS.

The following is a best practices example log4j2 configuration that shows how to use these properties and parameters:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
    <Appenders>
        <Null name="NullAppender" />
        <Console name="Console" target="SYSTEM_OUT">
            <PatternLayout pattern="%d{HH:mm:ss.SSS} [%X{traceId},%X{spanId},%X{parentId}] [%t] %-5level %logger{36} - %msg%n" />
        </Console>

        <Routing name="RollingFile">
            <Routes pattern="$${sys:job-api.log.logFolder}">
                <Route key="$${sys:job-api.log.logFolder}" ref="NullAppender" /> <!-- Log to NullAppender when system property not set yet, Console appender will still log it -->
                <Route>
                    <RollingFile name="RollingFile-${sys:job-api.cluster.nodeName}" fileName="${sys:job-api.log.logFolder}/job-api.log"
                        filePattern="${sys:job-api.log.logFolder}/job-api-%i.log.gz">
                        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%X{traceId},%X{spanId},%X{parentId}] [%t] %-5level %logger{36} - %msg%n" />
                        <Policies>
                            <SizeBasedTriggeringPolicy size="250 MB"/>
                        </Policies>
                    </RollingFile>
                </Route>
            </Routes>
        </Routing>

        <Routing name="job">
            <Routes pattern="$${ctx:job-id}">
                <Route key="$${ctx:job-id}" ref="Console" /> <!-- Log to console when the job-id not set -->
                <Route>
                    <File name="jobAppender-${ctx:job-id}"
                        fileName="${sys:job-api.log.logFolder}/${sys:job-api.log.logFilePrefix}${ctx:job-id}.log">
                        <Filters>
                            <ThresholdFilter level="trace" onMatch="accept" />
                        </Filters>
                        <PatternLayout pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
                    </File>
                </Route>
            </Routes>
        </Routing>
    </Appenders>

    <Loggers>
        <Root level="info">
            <AppenderRef ref="Console" />
            <AppenderRef ref="RollingFile" />
        </Root>

        <!-- Your logger config -->
        <Logger name="my.job.logger" level="info">
            <AppenderRef ref="job"/>
        </Logger>
    </Loggers>

</Configuration>

The main point of interest is routing configurations. With these, one can achieve conditional logging depending on specific system properties or context variables. 

The RollingFile Routing will create a log routing based on the job-api.log.logFolder system property. This is the main log file.
If the system property has not yet been set, all logs will be routed to the NullAppender appender, but are still handled by the Console appender as that is listed in the Root logger. You can also use a different appender in case these should be logged to a fixed file instead. If the system property has been set during the Job API initialization, then all log output is handled by the default Route below.
In this example, a RollingFile appender is used that receives its unique name based on the job-api.cluster.nodeName system property containing the cluster node name. It uses the job-api.log.logFolder system property to define the log directory. Depending on the configuration mode, the log file location will be resolved to the following:

  • In standalone mode: <logFolder-config-value>/job-api.log , that is. logs/job-api.log with the default configuration.

  • In any cluster mode: <logFolder-config-value>/<resolved-node-name>/job-api.log , that is. logs/my-hostname/job-api.log with the default logFolder configuration and nodeNameStrategy resolving the node name to my-hostname .

Additionally, it showcases how to use the trace logging pattern parameters via %X{traceId}, %X{spanId} and %X{parentId}.

The job Routing shows how to route log output based on the job-id context variable. The logic closely resembles the RollingFile Routing, with the difference that it interacts with the context variable instead of the system property. This context variable is set by every job being executed, which enables you to separate the log output created by the job and its tasks into separate log files in addition to the main log file. Depending on the configured mode the log file location will be resolved to the following:

  • In standalone mode: <logFolder-config-value>/<logFilePrefix-config-value><job-id>.log , that is. logs/job-1234.log with the default configuration.

  • In any cluster mode: <logFolder-config-value>/<resolved-node-name>/<logFilePrefix-config-value><job-id>.log , that is. logs/my-hostname/job-1234.log with the default logFolder and logFilePrefix configuration and nodeNameStrategy resolving the node name to my-hostname .


REST endpoints

There are currently four REST endpoints located in JobController that can be used to request job information, their results, or to cancel them. Those endpoints are also documented through the Swagger UI (for example, when running the API server locally, http://localhost:8080/swagger-ui.html#/job-controller/).

Method URI Parameter Status Code Description
GET /api/v1/jobs query 200 This endpoint can be used to get the JobInformation of all jobs that match the provided RSQL query. See below for more details on RSQL.
GET /api/v1/jobs/{jobId}/info jobId 200, 404 This endpoint returns the JobInformation of the job with the provided UUID. Status code 404 is returned if the job does not exist.
GET /api/v1/jobs/{jobId}/cancel jobId 200, 404 If the job produced a result like a CSV file or a job-specific Object, then this can be queried with this endpoint. This endpoint supports dynamic MIME type handling, depending on the information provided by the job implementation. Status code 404 is returned if the job does not exist. Status code 204 is returned if the job has no result.
GET /api/v1/jobs/{jobId}/result jobId 200, 404, 204 If the job produced a result like a CSV file or a job-specific Object, then this can be queried with this endpoint. This endpoint supports dynamic MIME type handling, depending on the information provided by the job implementation. Status code 404 is returned if the job does not exist. Status code 204 is returned if the job has no result.
GET /api/v1/jobs/{jobId}/log jobId 202, 404, 403 This endpoint returns the String representation of JobLog  that matches the provided jobId  if the user is authorized to access the job. Status code 404 is returned if the job does not exist or if the JobLog  is empty. Status code 403 is returned if the user is not authorized to access the job.
Info

The parameter jobId is a path variable containing the UUID of the job, while query is a request parameter.

Info

Status code 404 is returned if the requested job does not exist. Status code 204 is returned if the requested job has no result.