Skip to content

Environment setup

This chapter focuses on Code Discovery & Mining server configuration, setting up the server connection, and configuring logging. It includes detailed information about user and access management for system administrators.

Info

Code Discovery & Mining is not compatible with the IBM AIX operating system and the IBM JRE.


Postgres setup

The following section covers the setup for Postgres. It summarizes the base database tasks that Mining depends on, including cluster initialization, server startup, and initial user creation. Use the platform-specific steps below to prepare a Postgres instance before connecting the rest of the Mining components.

Windows

Perform the steps below to set up Postgres on a Windows machine.

Step 1:
Download and unpack server binaries.

Step 2:
Create and initialize a new database cluster.

  • Choose a folder for creating a new database "cluster" (). This will contain all the databases managed by a server instance along with its configuration.

  • Initialize the data cluster. From the command prompt, run the code below. Note that and should contain the absolute paths.
    <pgsql>\bin\initdb --no-locale –-encoding=UTF8 <pgdata>

  • After initialization, you will find the following configuration files in . (No change is required in these files when starting the server locally.)

    • postgresql.confContains basic server settings like the TCP port (5432 by default), load, and memory limits.

    • pg_hba.confDefines access restrictions. By default, administrative access is allowed without authentication from localhost only.

Step 3:
Start the database server.

  • From a command prompt, run:
    <pgsql>\bin\postgres -D <pgdata>
    The server log will be printed to the terminal in real-time.

    Alternatively, the server can be started using a batch file as:
    <pgsql>\bin\postgres.exe -D <pgdata>

Step 4:
Create an application user and a database.

  • With the server up and running, from a command prompt, run:
    <pgsql>/bin/psql postgres
    The server admin prompt shown below will be displayed:
    postgres=+

  • From this command prompt,

    • Create a role "mining" as below:
      CREATE ROLE mining WITH LOGIN;

    • Then create the database "mining" with the role created above:
      CREATE DATABASE mining WITH OWNER = mining;

Linux

Set up using Docker PostgreSQL image

Step 1:

Pull the latest or version 15 and above Postgres image using the command:

1
2
3
docker pull postgres:15.3
OR
docker pull postgres:latest

Step 2:

Create a Postgres container, copy the backup into it, and start the container as described below:

1
2
3
docker create --name ${pg_container} -e POSTGRES_PASSWORD=mining -p 5432:5432 postgres
docker cp ${jobWorkspace}/docker/backup_postgres ${pg_container}:/backup_postgres
docker start ${pg_container}

Step 3:

Delete and recreate the database and role:

1
2
3
4
docker exec --user postgres ${pg_container} psql -c \"DROP DATABASE IF EXISTS mining;\"
docker exec --user postgres ${pg_container} psql -c \"DROP ROLE IF EXISTS mining;\"
docker exec --user postgres ${pg_container} psql -c \"CREATE ROLE mining WITH LOGIN PASSWORD 'Worx2000';\"
docker exec --user postgres ${pg_container} psql -c \"CREATE DATABASE mining WITH OWNER=mining;\"

Step 4:

Restore the backup using the following command:

docker exec ${pg_container} pg_restore -U postgres -d mining backup_postgres

Prerequisites (if Postgres image is not used)

Follow the steps below to manually install Postgres on a Linux machine:

Step 1:
Log in as a root user.

Step 2:
Install Postgres.

  1. Install the repository RPM:
    dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat

  2. Disable the built-in PostgreSQL module:
    dnf -qy module disable postgresql

  3. Install PostgreSQL:
    dnf install -y postgresql15-server

Step 3:
A pgsql-15 folder is visible in the "usr" directory.

Step 4:
Create a directory "postgresql" and a data directory "data15" within postgresql and assign the necessary permissions.

1
2
3
mkdir /usr/postgresql
mkdir /usr/postgresql/data15
chmod -R 777 /usr/postgresql/data15

Step 5:
Copy the pgsql-15 folder and paste it into the postgresql directory:
cp -r /usr/pgsql-15 /usr/postgresql/pgsql-15

Step 6:
Log in as Postgres user, set the library path, and initialize the database cluster/directory:

1
2
3
4
5
su - postgres

export LD_LIBRARY_PATH=/usr/pgsql-15/lib:/usr/postgresql/pgsql-15/lib

/usr/postgresql/pgsql-15/bin/initdb -D /usr/postgresql/data15

Step 7:
After initialization, you will find the following configuration files in data15:

  • postgresql.conf

  • pg_hba.conf

Step 8:
Open postgresql.conf, uncomment the following lines and add the IP address of the host machine against listen_addresses:

1
2
3
4
5
# - Connection Settings -

listen_addresses = '<ip address>'

port = 5432

Step 9:
Open pg_hba.conf and add another IP address of the host machine under the section "IPv4 local connections":

1
2
3
# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             <ip address>/32         trust

Set up Postgres via Groovy script

Perform the following steps to set up Postgres via a Groovy Script.

Step 1:
Create a directory "postgres" inside the jobWorkspace and modify the permissions (chmod -R 777) of that directory.

Step 2:
Create a container with username and password, mount the volume from "<jobWorkspace>/postgres" to "/var/lib/postgresql/data15", and start the container with port 5432 and image postgres.

docker create --name postgres -e POSTGRES_USER=postgres -e POSTGRES_PASSWORD=postgres -v <jobWorkspace>/postgres:/var/lib/postgresql/

Step 3:
Copy the data directory (data15) from the host system and paste it into the container.

docker cp /usr/postgresql/data15 postgres:/var/lib/postgresql/data15

Step 4:
Start the Postgres container.

Step 5:
Verify that Postgres is running.

1
2
3
4
5
#check if postgres is running
docker exec postgres pg_isready

#The following output should be displayed
/var/run/postgresql:5432 - accepting connections

Step 6:
Create the application user and database as follows.

docker exec --user postgres postgres psql -c \"CREATE ROLE mining WITH LOGIN PASSWORD 'Worx2000';\"
docker exec --user postgres postgres psql -c \"CREATE DATABASE mining WITH OWNER=mining;\"

Step 7:
Add the JDBC URL in the command to start the API server.

postgres.datasource.jdbc-url=jdbc:postgresql://<ip_address>:5432/mining

AWS EC2 Instance

Add the following commands to the Postgres launch template in the User Data section:

sudo -i

tee /etc/yum.repos.d/pgdg.repo<<EOF

[pgdg15]

name=PostgreSQL 15 for RHEL/CentOS 7 - x86_64

baseurl=https://download.postgresql.org/pub/repos/yum/15/redhat/rhel-7-x86_64

enabled=1

gpgcheck=0

EOF

yum update -y

yum install postgresql15 postgresql15-server -y

mkdir /usr/postgresql

mkdir /usr/postgresql/data15

chmod -R 777 /usr/postgresql/data15

cp -r /usr/pgsql-15 /usr/postgresql/pgsql-15

chown -R postgres:postgres /usr/postgresql/data15

echo export LD_LIBRARY_PATH=/usr/pgsql-15/lib:/usr/postgresql/pgsql-15/lib\&/usr/postgresql/pgsql-15/bin/initdb -D /usr/postgresql/data15 | su - postgres

echo /usr/postgresql/pgsql-15/bin/pg_ctl -D /usr/postgresql/data15 -l logfilenewnew start\&/usr/postgresql/pgsql-15/bin/pg_isready | su - postgres

/usr/postgresql/pgsql-15/bin/psql -U postgres -c "CREATE ROLE mining WITH LOGIN PASSWORD 'Worx2000'";

/usr/postgresql/pgsql-15/bin/psql -U postgres -c "CREATE DATABASE mining WITH OWNER=mining";

Code Discovery & Mining API server configuration

This section gives an overview of the fundamentals of Mining API server configuration. It explains the default packaging model, the main startup options, and where to place customer-specific overrides. Use it to understand which settings must be supplied externally and which ones can remain at their defaults.

Configuration

The Mining API Server is delivered as a self-contained JAR file with a built-in default configuration. It is usually configured to customer specifications during the initial setup by the Deloitte project team.

Configuration

On Linux hosts, the fontconfig package must be installed as a requirement for the Excel file generation during Discovery.

Starting the Mining API server

The Mining API Server JAR only requires the Java JRE to run, and can be executed via command line interface (CLI) with the following command:

java -jar mining-api-server-dist-<version>.jar

The server configuration can be adjusted by appending command-line options, for example, for application and log configuration.

Application configuration

The application's configuration can be adjusted by providing a customized YML file via the jar command line parameter --spring.config.additional-location:

java -jar mining-api-server-dist-<version>.jar --spring.config.additional-location=/path/to/application-custom.yml

Configuration options range from server details, database connection, and spring profile parameters to Job API (see Mining Job API) and Keycloak settings.

Configuration

The default YML configuration can be extracted from the server JAR file at mining-api-server-dist-<version>.jar\BOOT-INF\classes\application-default.yml and used as a template.

To simplify updating your Mining version, it is recommended to remove everything from the default YML, except for the configuration values that you need to customize. 
For example, to configure the database URL and credentials, the following YML file content should suffice:

1
2
3
4
5
spring:
    datasource:
        url: jdbc:postgres:remote:<host>:<port>/mining
        username: <username>
        password: <password>

Task thread limit

For testing and debugging, it is possible to limit the number of concurrent tasks for the job execution.
In the application configuration YML file, adjust the block maximumLocalTaskThreads: under job-api::

job-api:
    maximumLocalTaskThreads: 1

Job thread limit

A job may initiate another job and wait for its completion. Therefore, the configurable number of concurrent jobs has been constrained to a minimum of 4. This helps to avoid a job causing a deadlock when another job is waiting or cannot run in parallel.  To change the limit on the number of concurrent jobs, adjust the following property in the configuration YML file:

job-api:
    maximumLocalJobThreads: 4
Warning

Setting the thread count below 4 will fail to start the server, and the corresponding error message: "APPLICATION FAILED TO START".

Discovery cache sizes

To optimize memory usage for Discover Code and Discover Metrics executions, the number of entries kept in memory for various tasks can be configured. To adjust cache sizes, adjust the following properties in the configuration YML file:

configuration:
    # maximum number of entries in the Discovery parse result cache (0 disables cache)
    discovery-parse-result-cache-size: 10000
    # maximum number of entries in the Discovery lightweight parse result cache (0 disables cache)
    discovery-lightweight-parse-result-cache-size: 1000
    # maximum number of entries in the clustered Discovery module repository cache (0 disables cache)
    discovery-module-repository-cache-size: 200000
    # maximum number of entries in the Discovery source object cache (0 disables cache)
    discovery-source-object-cache-size: 100000
    # maximum number of bytes in the Discovery source code cache (0 disables cache)
    discovery-source-code-cache-size: 2147483648
    # true uses strong keys and values in caches, false weak keys and soft values
    discovery-cache-strong-references: false

To restore the previous behavior, use the following setting, since caches use strong references:

configuration:
    discovery-cache-strong-references: true
Configuration

Reduce the result cache size if you are encountering OutOfMemory errors during Discovery.

Discovery upload size limit

During Discover Code operations, the project source code is uploaded in the background. The source upload has a default limit of 10,000,000,000 uncompressed bytes. The value is configurable to enable projects to upload archives that contain larger amounts of data via the discovery-source-upload-maximum-size parameter in the configuration YML file:

1
2
3
4
configuration:
    # maximum number of uncompressed bytes that may be read in a Discovery source upload to prevent Zip Bomb attacks.
    # If the value is 0 or negative, the default is used
    discovery-source-upload-maximum-size: 10000000000

Call chain export options

Maximum CSV export lines

The CSV export of call chains is limited in the spring configuration to 5,000,000 lines by default. The value can be configured in the application.yaml file in the configuration section to improve call chain export performance:

1
2
3
4
configuration:
    # maximum number of CSV lines that can be exported by each single call chain job
    # A value less than 1 means that no limit is used for the CSV export
    call-chain-maximum-csv-export-lines: 5000000
Maximum export threads

It is possible to specify the maximum number of task threads for call chain computation.
The default value is 8. A value less than 1 will allow the runtime to use all available processors.

The value can be configured in the application.yaml file in the configuration section to improve call chain export performance:

1
2
3
4
5
configuration:
    # Maximum number of task threads to be used for the computation of call chains
    # A value less than 0 means that all available processors are used
    # This can significantly slow down other running jobs and cause the UI to be unresponsive while call chains are computed
    call-chain-maximum-export-threads: 8

The Mining Web UI can be configured to prompt the user to consent to cookies before logging in.
The mining.cookieId property must be set at the time of deployment to enable or disable this feature.

Info

Cookie IDs are generated specifically for each client and are provided to the respective Deloitte project admin.

This property can be set in three ways:

Option 1:
Adjust the configuration YML file.

mining:
    cookieId: <your-ID>

Option 2:
Pass the property as a start parameter in the server startup command.

java -jar mining-api-server-dist-<version>.jar --mining.cookieId=<your-ID>

Option 3:
Set the property as an environment variable.

env MINING_COOKIEID=<your-ID> java -jar mining-api-server-dist.jar
Warning

For non-production environments, the cookie consent prompt can be disabled by assigning the value "DISABLED", for example. -mining.cookieId=DISABLED.

Authentication

The primary authentication approach is by default Keycloak authentication. However, for users opting for legacy authentication, they can activate the legacy authentication profile by appending --spring.profiles.active=legacy-auth when running the project. Pass the property as a start command in the api server startup command.

--spring.profiles.active=legacy-auth --mining.user.username=<your-Username> --mining.user.password=<your-Password> --mining.user.token=<your-LegacyToken>

Alternatively, it can be configured in the application configuration YML file:

1
2
3
4
5
6
7
8
mining:
user:
username: <YourUsername>
password: <YourPassword>
token: <YourLegacyToken>
spring:
    profile: 
active: legacy-auth

Set category for GenAI knowledge-service retrieval

You are allowed to specify the category/index to be used for querying the knowledge service by setting its value to the mining.genAI.annotationKnowledgeCategory property when running the project. This property is used for querying the OpenSearch database in the backend. The default value is VARIABLE_DESCRIPTION. This property can be passed along when starting the API server with the following command:

--mining.genAI.annotationKnowledgeCategory=<yourValue>

Alternatively, it can be configured in the application configuration YML file:

mining:
    taxonomies:
        technicalTaxonomyCategoryName: Technical Taxonomies
        defaultTaxonomyCategoryName: Business Taxonomies
    cookieId:
    genAI:
        url:
        apiKey:
        plugIn:
        responseFormat: JSON # or TEXT, ensure availablity with the above plugin
        maxNewTokens:
        temperature:
        doSample:
        tokenRateLimit:
        maxParallelTasks:
        pathSegment:
            # This is expecting the path segment including the "/"
            moduleDescription:
            annotationDescription:
        semanticSearchUrl: # Points to GenAI backend where Semantic Search Plugin is deployed
        promptServiceUrl:
        knowledgeService:
            url: # Points to GenAI backend where the Knowledge Service is hosted
        retrieverTopK:
        # maximum number of prompts in the prompt cache
        prompt-cache-size: 100
        # the time in seconds after which the prompt cache is invalidated
        prompt-cache-duration: 30
        annotationKnowledgeCategory: VARIABLE_DESCRIPTION

Introduced LLM response

The response format (for example, TEXT, JSON) of the configured model can now be specified via the responseFormat parameter inside the application.yaml. The default response format for Annotation Description generation is now JSON.

Warning

Make sure that the plugin specified in your application.yaml file supports JSON response format

plugIn: azure gpt 3.5 # ensure availability of the below 'responseFormat' for the configured plugin
responseFormat: JSON # or TEXT, ensure availablity with the above plugin

Important: If the specified plugin does not support the JSON response format, make sure to change the responseFormat to “TEXT” in the application.yaml file.

responseFormat: TEXT

For example, check Azure AI for endpoints hosted on Azure Cloud.

The following models (and newer) should support JSON mode:

  • gpt-35-turbo(1106)

  • gpt-35-turbo(0125)

  • gpt-4(1106-Preview)

  • gpt-4(0125-Preview)

Keycloak configuration

When the Mining API Server is to be used in conjunction with Keycloak, the authorized-access spring profile needs to be used. Otherwise, the server will fall back on the integrated default authentication without Keycloak. The application YML file should therefore contain the following content:

1
2
3
4
5
6
7
spring:
    profiles:
        active: authorized-access
keycloak:
    auth-server-url: http://your-auth-server
keycloak.lookup-timeout : Set lookup timeout (default 1800 seconds)
configuration.keycloakUserNameCacheDuration : For time after which the cache will be invalidated (default 1800 seconds)

The HTML Display name and the Redirect URIs in the Keycloak Realm Client need to be adjusted to point to the Mining API server. This can be done in your Realm's client configuration via the Keycloak Admin UI.
The following example shows the required configuration for the HTML display name, backend, and web clients, assuming that the Mining API server is deployed to http://127.0.0.1:8080.

General configuration

env setup General configuration

Backend client configuration

env setup Backend client configuration

Web client configuration

env setup client configuration

Info

Note that the URI http://127.0.0.1:4200 is also listed in the setting above. Port 4200 is the preconfigured default port for the Mining UI.

To support the Mining access matrix (see Access Matrix), a client-specific role mapping needs to be configured. Keycloak supports the integration of various identity providers and user federation systems. Therefore, your Mining setup can be interconnected with your Single-Sign-On (SSO) or Active Directory (AD) system.

For detailed information about individual options and Keycloak configurations, please refer to the official Keycloak documentation:

Keycloak version 22 upgrade

Building a Docker image

Step 1:
Log in to the Docker registry

docker login {registryName}

Step 2:
Build the Docker image with the upgraded Keycloak jar (with the help of a Docker File) and tag it with the build version

docker build -t {registryName}/keycloak:{mxBuildVersion} .

Step 3:
Push the newly built image to the Docker registry

docker.image("{registryName}/keycloak:{mxBuildVersion}").push()

Keycloak URL

The /auth from the keycloak URL is removed and the new URL is changed to http://:8180/

Keycloak directory path

The path of the Keycloak directory to create/assign roles and users via the Keycloak Admin CLI has been changed to /opt/keycloak/bin/kcadm.sh

Connecting to an AWS Aurora Postgres database

You can use AWS Aurora as the "postgres" database. When authenticating with username/password, you can use the standard PostgreSQL driver, and no special configuration is needed. However, when you want to use AWS IAM authentication to access the database, you must use a special JDBC driver provided by AWS and a special configuration.

The documentation on the JDBC driver can be found here:

Note that setting the "classpath" argument on the command line does not have the desired effect for spring-boot jars. To load the JDBC driver, you must use the "loader.path" system property like so:

java -Dloader.path="/path/to/aws-jdbc-driver.jar" -cp mining-api-server.jar org.springframework.boot.loader.PropertiesLauncher
Info

You must put the -Dloader.path argument before the class argument (org.springframework.boot.loader.PropertiesLauncher), otherwise it doesn't work.

When the driver is loaded properly, you can use this configuration to set up IAM authentication:

postgres:
    enabled: true
    datasource:
        driver-class-name: software.amazon.jdbc.Driver
        jdbc-url: jdbc:aws-wrapper:postgresql://<host>:<port>/mining
        username: <username>
        connection-test-query: select 1
        minimum-idle: 50
        maximum-pool-size: 150
        data-source-properties:
            wrapperPlugins: iam,efm
            wrapperDialect: aurora-pg
            iamDefaultPort: 6160
            max-lifetime: 840000
            idle-timeout: 900000

Log configuration

Logging can be configured by providing a customized log4j2.xml file via the java system property -Dlogging.config:

java -Dlogging.config=/path/to/log4j2-custom.xml -jar mining-api-server-dist-<version>.jar
Configuration

The default log configuration can be extracted from the server JAR file at mining-api-server-dist-<version>.jar\BOOT-INF\classes\log4j2.xml and used as a template.

Info

The existence of a log file depends on the configured log level in the log4j2.xml file. For example, if the log level is set to OFF or ERROR, no log file is created when only INFO events are triggered.

After a job has been executed from Eclipse (for example, mining > Analysis > Calculate Control Flow), the corresponding server log can be retrieved from the Logs column in Job View:

env setup After been executed from Eclipse (for example,

Clicking the Logs link will trigger a Windows Explorer prompt to save the log file to your desired location.

Temporary log directory

When executing Discovery jobs (Discover Code, Discover Metrics, etc.), the log output from these jobs is recorded in a temporary file, which is then automatically downloaded to the Eclipse workspace after the job finishes.
The location of the temporary folder can be reconfigured with the following adjustments:

In the application configuration YML file, add a child block log: under job-api::

1
2
3
4
job-api:
    log:
        logFolder: my/custom/location
        logFilePrefix: "customPrefix-"

A matching change must be applied to the log configuration. In the log configuration XML file, search for the appender with name="jobAppender-${ctx:job-id:-unknown}" and modify it as follows:

<Route>
    <File name="jobAppender-${ctx:job-id:-unknown}"
        fileName="my/custom/location/customPrefix-${ctx:job-id:-unknown}.log">
        <Filters>
            <ThresholdFilter level="trace" onMatch="accept" />
        </Filters>
        <PatternLayout
            pattern="%d{HH:mm:ss.SSS} [%t] %-5level %logger{36} - %msg%n" />
    </File>
</Route>

Mining Extensions

Mining extensions are Java classes that implement a certain interface and register some additional functionality for the Mining product. They are provided as .jar files by the Mining development team.
Extensions are registered automatically at runtime if they are put on the classpath of the Mining API server.

The following example outlines how to integrate the Decompose export extension.
The API server start command needs to be adjusted to include the library:

java -Dloader.path="db-cutter-mining-extensions-<version>.jar" -cp mining-api-server-dist.jar org.springframework.boot.loader.launch.PropertiesLauncher

Additional launch parameters and escaping might apply, especially in the case of remote or automated deployments.


Mining API server connection

To work with most Mining functions, your local project must be connected to the Mining server. This can be done with or without Keycloak.

To connect to the server, navigate to Window > Preferences > mining and perform the steps for Option A: Connect with Keycloak or Option B: Connect without Keycloak, as the case may be:

Option A: Connect with Keycloak

Step 1:
In the project or workspace Preferences, enter the correct API server URL and select Manage Tokens on the mining tab.

env setup 2 showing project workspace Preferences, enter correct server select

env setup screenshot 4

Step 2:
After selecting Manage Tokens , a browser opens automatically. Log on to that browser, and then click the + New Token . Enter a description (optional) in the Create New Token window. Then click Save .

env setup 5 showing After selecting Manage Tokens browser opens automatically.

Within a couple of seconds, the “Token Created” window appears with the access token. Copy the access token and close the window.

env setup 6 showing Within couple seconds, “Token Created” window appears

Step 3: Paste the access token in the Preferences window (see Step 1 above), click Apply , and then click the TEST API Connection . A “Server connection successful” message will be displayed along with information on the connected user, the response time, and the server API version.

env setup screenshot 7

Option B: Connect without Keycloak

Step 1:
Enter the URL of the API server you wish to connect with, and then enter the access token ff93340a-af6c-4fcd-90bb-426448fa1597 in the Access token field.

Step 2:
Click Apply .

To check whether the connection to the server has been established, you can click Test API Connection . If the connection exists, the “Server connection successful” message will be displayed. The message also shows the connected user, the response time, and the server API version.

Step 3:
Click Apply and Close .


Logging

Logging helps you verify that the Eclipse-based tooling and server connection are working as expected. The sections below show where Mining writes diagnostic information and how you can review it inside the IDE. Use these logs first when you need to investigate warnings, errors, or connectivity issues.

Workspace log

Logs are written to the Workspace Log. It is located in your Eclipse workspace in the .metadata folder and is named .log.

Error log view

The error messages and warnings listed in the Workspace Log are also shown in the Error Log view in Eclipse. To display the Error Log, navigate to Window > Show View > Error Log and click Open .

error-log-complete.png

To view details, double-click the corresponding entry.

error-log-complete-2.png


User management without Keycloak - Information for system administrators

The following sections provide information for system administrators. They describe the built-in user and token model that is available when Keycloak is not part of the deployment. Use this information to review default access and to provision additional users in older or simplified environments.

User and OAuth access

The database schema comes loaded with an administrator (Username: admin) user with the following access token:

admin user access token
6c2000c1-d9a8-4d87-92f0-0a11139a037e

If multiple miners will be using the Mining API server, additional users should be created according to the instructions in the section below.

Creating a new user

Warning

This section is for OrientDB only and is deprecated for the newest version of Code Discovery & Mining

Complete the following steps to create a new user:

Step 1:

Generate the user password hash by starting the Mining server with the argument "hashpw"

java -jar mining-api-server.jar hashpw

Step 2:

When prompted, enter the user password to hash. Find the hashed password in the next line.

1
2
3
4
5
6
2019-06-19 10:52:44.205  INFO 21816 --- [           main] o.s.b.w.embedded.tomcat.TomcatWebServer  : Tomcat started on port(s): 8088 (http) with context path ''
2019-06-19 10:52:44.210  INFO 21816 --- [           main] i.mining.server.MiningApiApplication     : Started MiningApiApplication in 7.792 seconds (JVM running for 8.308)
Enter password to hash:
mysecret
$2a$10$zboBJSQesNhHf3Mmqkq1G.JH2Q0cUEu.eJeSk6ef.k7HFPf5S4p4e
2019-06-19 10:52:52.633 DEBUG 21816 --- [           main] ConfigServletWebServerApplicationContext : Closing ...

Step 3:

In a web browser, enter the URL on which the database is running, followed by a colon, followed by the port (always 2480), for example. localhost:2480.

Step 4:

In the window that opens:

  • For Database, select mining from the drop-down list

  • Enter the credentials of the database and root user

  • Click CONNECT

orientdb.png

Step 5:

An OrientDB window opens. In the top field, create the new user by executing the following SQL with the new user name and the password hash that you created before.

insert into user set userId="<username>", password="$2a$10$zboBJSQesNhHf3Mmqkq1G.JH2Q0cUEu.eJeSk6ef.k7HFPf5S4p4e"

Step 6:

Click RUN .

Step 7:

Verify the creation by running select from user. Now the user is displayed in the user table.

select-from-user.png

Creating the access token

Follow these steps to create an access token:

Step 1:

Create the access token by using the OAuth service with Postman as follows. On the landing page, click GET STARTED .

Step 2:

The download page will open. From the drop-down menu, select Windows 64-bit.

Step 3:

On the page that opens, beneath "Untitled Request", select POST from the drop-down menu and enter localhost:8080.

Step 4:

Navigate to the Authorization tab and select Basic Auth.

Step 5:

Navigate to the BODY tab and select the x-www-form-urlencoded radio button.

Postman Settings
POST method on http://<server:port>/oauth/token

Authorization
Type is Basic Auth
Username: mining-plugin-client
Password: mining-client-secret

Body
Type is x-www-form-urlencoded

Key:Value
username:<userId>
password:<password NOT hashed>
grant_type:password

Successful result:

1
2
3
4
5
6
7
{
    "access_token": "f4be410d-996e-455d-b95e-19465d525ad9",
    "token_type": "bearer",
    "refresh_token": "2cd06648-0742-497e-a945-458bedcb62d4",
    "expires_in": 315571179,
    "scope": "read write trust"
}

Step 6:

Use the provided access token in the mining-plugin preferences or on any mining-server call as url argument "access_token".

Example:

http://server:port/?access_token=f4be410d-996e-455d-b95e-19465d525ad9


Identity and access management IAM with Keycloak

The IAM feature allows administrators to manage user privileges by means of role-based access control (RBAC). While Identity Management covers authentication (for example, signup, login, password reset, etc.), access management deals with the authorization of the access levels to features in the Mining platform.

Authentication takes place via Keycloak. Keycloak is an external tool that is integrated into Mining. For more information, see the official Keycloak documentation at https://www.keycloak.org/docs/latest/.

Users can only be created via Keycloak; however, the user "Admin" ("god mode" admin) can access the page for creating users in Keycloak via the Global menu in the web UI (Global > Users).

Roles and definitions

The following table describes the available roles and access levels.

Role Description
Admin The highest role in the user hierarchy, can also understood as "super user", "root", or "god mode". A general characteristic of admin role is that this role holds rights to manipulate data structures (for example, executing SQL commands). A user with a "admin" role has no restrictions and must be assigned to users wisely.
Client Admin The Client Admin role provides similar rights to an Admin, but is scoped to a defined client. It can be understood as an Account Manager. Like the Admin role, one of the features of the Client Admin is the right to manipulate data structures of projects, such as modifying annotation categories or taxonomies.
Manager The Manager role is provided for the project leads and allows manipulation of the mining baseline data. This is typically a person from the business function, that is, a person who is capable of understanding and approving the mined data.
Editor Users with the Editor role have basic editing rights needed to work with Mining effectively, such as creating annotations, assigning taxonomies, creating descriptions, etc.
Viewer The Viewer role is effectively a read-only role that can access and view features but can not add or modify data. This role is designed to allow an insight into the delivery and service process as a spectator, and to ensure that this user cannot edit data.

Access matrix

The following matrix represents the various access levels and permissions.

Project Nature Client Admin
(Restricted to specific client)
Admin Manager
(Restricted to assigned project)
Editor
(Restricted to assigned project)
Viewer
(Restricted to assigned project)
Users
Assign "Admin" role
(via Keycloak)
n/a ❌ ❌ ❌ ❌ ❌
Assign "Client Admin" role n/a ✅ ✅ ❌ ❌ ❌
Clients
See/Read n/a Only those clients from which projects the current user is assigned to ✅ ✅ ✅ ✅
Add, Edit, Delete n/a ❌ ✅ ❌ ❌ ❌
Projects
See/Read n/a All of specific client ✅ ✅ ✅ ✅
Add, Edit, Delete n/a ✅ ✅ ❌ ❌ ❌
Assign Members with Regular Roles
(Manager, Editor, Viewer)
n/a ✅ ✅ ❌ ❌ ❌
Override Project Nature on User
(for example, access Mining features in a Discovery-only project)
n/a ❌ ✅ ❌ ❌ ❌
Legacy Discovery Import
Server-side Import Discovery Excel/CSV
Legacy
Mining ✅ ✅ ✅ ❌ ❌
Discovery on Mining
Discover Code
on selection
Discovery Mining ✅ ✅ ✅ ❌ ❌
Discover Metrics
on selection
Discovery Mining ✅ ✅ ✅ ❌ ❌
Discover DNA
on selection
Discovery Mining ✅ ✅ ✅ ❌ ❌
Upload Source Code Files
(Advanced)
Discovery Mining ✅ ✅ ✅ ❌ ❌
Download Source Code Files
(Advanced)
Discovery Mining ✅ ✅ ✅ ✅ ✅
Upload Configuration
(Advanced)
Discovery Mining ✅ ✅ ✅ ❌ ❌
Download Configuration
(Advanced)
Discovery Mining ✅ ✅ ✅ ✅ ✅
Export Discovery Excel/CSV Discovery Mining ✅ ✅ ✅ ❌ ❌
Export Summary Effort Excel Discovery Mining ✅ ✅ ✅ ❌ ❌
Annotations
See/Read Annotations on Code Mining ✅ ✅ ✅ ✅ ✅
Add, Edit, Delete Annotations on Code Mining ✅ ✅ ✅ ✅ ❌
Set status to "Approved" Mining ✅ ✅ ✅ ❌ ❌
Maintain Annotation Categories (project specific) Mining ✅ ✅ ✅ ❌ ❌
Bulk Deletion Mining ✅ ✅ ✅ ❌ ❌
Data Dictionary
See/Read Entries Mining ✅ ✅ ✅ ✅ ✅
Add, Edit, Delete Entries Mining ✅ ✅ ✅ ✅ ❌
Validate Text Selection Mining ✅ ✅ ✅ ✅ ❌
Bulk Deletion Mining ✅ ✅ ✅ ✅ ❌
Module Description
See/Read Mining ✅ ✅ ✅ ✅ ✅
Edit/Delete Mining ✅ ✅ ✅ ✅ ❌
Taxonomy
See/Read Taxonomies Mining ✅ ✅ ✅ ✅ ✅
Add, Edit, Delete Taxonomies Mining ✅ ✅ ✅ ❌ ❌
Assign, Bulk Assign / Propagate Taxonomy to Module Mining ✅ ✅ ✅ ❌ ❌
Job Framework
See Jobs and Results Mining All of specific client ✅ ✅ ✅ ❌
Cancel/Delete Job Mining All of specific client ✅ ✅ ✅ ❌
Entity Meta Model (Custom Properties)
See/Read/"Use" properties Mining ✅ ✅ ✅ ✅ ✅
Create/edit/delete properties Mining ❌ ✅ ❌ ❌ ❌
Automated Analyses
Candidate Item Identification Mining ✅ ✅ ✅ ❌ ❌
Technical Taxonomy Mining ✅ ✅ ✅ ❌ ❌
Module Description Mining ✅ ✅ ✅ ❌ ❌
GenAI
Bulk Generation of Annotation Descriptions Mining ✅ ✅ ✅ ❌ ❌
Generate Single Reachability Block Description Mining ✅ ✅ ✅ ❌ ❌
Bulk Generate Reachability Block Description Mining ✅ ✅ ✅ ❌ ❌
Bulk Generate Module Description Mining ✅ ✅ ✅ ❌ ❌
Reachability Analysis
Initiate Reachability Analysis Mining ✅ ✅ ✅ ❌ ❌
Edit Name and Description of Single Reachability Block Mining ✅ ✅ ✅ ✅ ❌
Activate/ Inactivate Single Reachability Block Mining ✅ ✅ ✅ ✅ ❌
Merge / Un-Merge Reachability Blocks Mining ✅ ✅ ✅ ✅ ❌
Bulk Inactivate Reachability Blocks Mining ✅ ✅ ✅ ❌ ❌
Download CSV Data from RB Table View Mining ✅ ✅ ✅ ✅ ✅
Dismiss Removed Reachability Blocks Mining ✅ ✅ ✅ ✅ ❌
Sort Reachability blocks in Overview screen (Table View) Mining ✅ ✅ ✅ ✅ ✅
Download Graph Views for Single Reachability Block Mining ✅ ✅ ✅ ✅ ✅
Taxonomy Assignment to Modules within Reachability Blocks Mining ✅ ✅ ✅ ❌ ❌
Recalculate Reachability Blocks Mining ✅ ✅ ✅ ❌ ❌
Reveal Neighbors of Reachability Blocks Mining ✅ ✅ ✅ ✅ ❌
Others
Initiate AST/Control Flow Calculation Mining ✅ ✅ ✅ ✅ ✅
SQL REST-Endpoints n/a ❌ ✅ ❌ ❌ ❌
Feature Console/Toggle n/a ❌ ✅ ❌ ❌ ❌
Export
(for example, CSV, Export Extensions)
Mining ✅ ✅ ✅ ✅ ✅

AWS Mining setup

This chapter outlines the recommended setup and configurations for Mining with Amazon Web Services (AWS) components. The goal is to separate the API, database, identity, and shared storage layers so the environment can scale and be managed independently. The EC2 instances are designed to be launched from reusable machine images and connected through the AWS services listed below.

  • A cluster of EC2 instances with Elastic Block Stores (EBS) for Mining API servers

  • One or more EC2 instances with Elastic Block Stores (EBS) for the Postgres database

  • An Elastic File System (EFS) as a shared location for Mining server artifacts and logs

  • Auto Scaling groups to launch above EC2 instances as required from Amazon Machine Images

  • An EC2 instance hosting KeyCloak

  • An Application Load Balancer

AWS architecture diagram

The following diagram provides an overview of a suggested Discovery and Mining AWS setup:

env setup following diagram provides overview suggested Discovery Mining

Info

All compute node instances form one Hazelcast compute cluster with significant traffic between them. Auto scaling for KeyCloak is an option, but one instance is expected to be sufficient. The database cluster uses Hazelcast for replication across all nodes with significant traffic between them.

Mining IDE

Configuration

Java 11 or 17 (or higher) must be installed on the machine hosting the Mining IDE. The machine must be able to make a network connection to the Mining API server, KeyCloak server and Postgres database instances.

The Eclipse-based Mining IDE is delivered in the form of an archive file in ZIP format, which needs to be extracted.
The ZIP archive contains all of the necessary files and executables. As far as installation goes, this is the only step.

To start the Mining IDE, a user needs to log in to the machine and run the Eclipse executable, eclipse.exe.
The required innoWake.lic license file is bundled into the ZIP archive in the appropriate location.

When setting up a Windows machine image in AWS, it is recommended that the default TCP RDP port 3389 be open so a user can log in. The user will need the Administrator login and password from AWS to be able to log in at first. Afterwards, they can create additional users if needed.

The Mining IDE will connect to the Mining server and authenticate with the KeyCloak server. Therefore, it requires the proper Security Group configuration in AWS.

Keycloak server

Configuration

Java 8 (or higher) must be installed on the machine before starting the KeyCloak application, as it runs in the JVM.

The KeyCloak server is delivered in the form of an archive file in ZIP format, which needs to be extracted. The archive includes all necessary files and configurations needed.
To extract the file, transfer the archive file to the machine and run the following command via an appropriate CLI:

unzip keycloak*.zip -d /path/to/preferred/directory

KeyCloak exposes the admin console UI on port 8080 by default. This port can be changed if needed by adjusting the following line in the file /standalone/configuration/standalone.xml:

<socket-binding name="http" port="${jboss.http.port:<port>}"/>
Configuration

If port 8080 is changed it needs to be reflected in the Mining server’s Spring configuration. This configuration value can be passed in as an argument when starting the Mining server: --keycloak.auth-server-url=http://<url>:<port>/auth.

The ZIP archive includes a Mining theme and extension for the KeyCloak configuration, which need to be moved to dedicated locations:

  • Copy mining-keycloak-theme-*.jar to /standalone/deployments/

  • Copy mining-keycloak-extension-*.jar to /standalone/deployments/

The Mining theme needs to be selected manually once after setup via the KeyCloak Admin UI.

Start KeyCloak by executing the following command via an appropriate CLI:

<KEYCLOAK-INSTALLATION-DIRECTOR>/bin/standalone.bat -b=0.0.0.0
Info

The standalone.bat/sh should start with the -b=0.0.0.0 parameter to allow non-local connections, that is, connections from another machine.

Warning

KeyCloak uses an embedded H2 (in-memory) database by default. While H2 can be persisted to disk it is highly recommended to connect this to a persistent RDBMS. KeyCloak’s documentation best describes this setup: https://www.keycloak.org/docs/latest/server_installation/index.html#_database The choice and setup of the RDBMS is left to the administrator.

A set of commands needs to be executed in the <KEYCLOAK-INSTALLATION-DIRECTOR>/bin directory to set up the server.
The following commands use default values for the admin user, password, and server port. Please adjust these appropriately where needed. The admin user needs to be set up manually so they will be able to log in to the KeyCloak UI and perform the configuration for any identity management integration.

Configuration

If these default values are changed, they need to be reflected in the Mining server’s Spring configuration. This configuration value can be passed in as an argument when starting the Mining server: --keycloak.auth-server-url=http://<url>:<port>/auth.

# Add Keycloak administrator 
add-user-keycloak.bat --user admin --password Worx2000 

# Setup realm 
kcadm.bat config credentials --server http://localhost:8180/auth --realm master --user admin --password Worx2000 

kcadm.bat create realms -s realm=mining -s id=mining -s enabled=true -s loginTheme=innowake-theme

# Setup clients 
# Change ${API_SERVER_URL} with the actual URL of where the API server is running 
kcadm.bat create clients -r mining -s id=backend -s clientId=backend -s directAccessGrantsEnabled=true -s publicClient=true -s redirectUris+="$API_SERVER_URL/*" -s webOrigins='["*"]' -s adminUrl=$API_SERVER_URL -s baseUrl=$API_SERVER_URL -s enabled=true 

kcadm.bat create clients -r mining -s id=web -s clientId=web -s directAccessGrantsEnabled=true -s publicClient=true -s redirectUris+="$API_SERVER_URL/*" -s webOrigins='["*"]' -s baseUrl="$API_SERVER_URL/" -s enabled=true 

kcadm.bat create clients -r mining -s id=eclipse -s clientId=eclipse -s directAccessGrantsEnabled=true -s publicClient=true -s redirectUris+="urn:ietf:wg:oauth:2.0:oob" -s redirectUris+="http://localhost" -s webOrigins='[]' -s enabled=true 

# Create Mining administrator 
kcadm.bat create users -r mining -s username=mining-admin -s enabled=true 

kcadm.bat set-password -r mining --username mining-admin --new-password Worx2000 

kcadm.bat create roles -r mining -s name=admin 

kcadm.bat add-roles --username mining-admin --rolename admin -r mining

After this has been set up, the web application should be available by navigating to the admin console in a browser at <IP of KeyCloak server>:<port>/auth. Log in with the admin credentials used in the command above.

The Deloitte Mining product uses KeyCloak for most of its base functionality. For further details, please refer to KeyCloak's documentation (https://www.keycloak.org/documentation).

Single sign on (SSO)

Info

This section covers the basic setup for SSO. It is possible that further steps are required for your internal systems.

A proper SSL setup of the backend application is mandatory.
The following points need to be considered:

  • Self-signed certificates are prohibited.

  • A DNS record needs to be requested to acquire a public signed certificate.

  • On AWS, an application load balancer can provide SSL termination and a static URL.

  • The URL the SSO solution should redirect to after login has to be activated upon request.

  • The SSO administrator will then provide configuration parameters (for example, "client ID", "client secret"), which need to be specified in the KeyCloak Admin UI.

The SSO settings in KeyCloak itself can be configured entirely via the KeyCloak Admin UI.

Mining API server

Configuration

Java 17 must be installed on the machine before starting the application server, as it runs in the JVM.

The KeyCloak server is delivered in the form of an archive file in ZIP format, which needs to be extracted. The archive includes all necessary files and configuration needed.
To extract the file, transfer the archive file to the machine and run the following command via an appropriate CLI:

unzip mining-api-server*.zip -d /path/to/preferred/directory

Inside the archive is the Java Spring Boot JAR and a Spring configuration file application-config.yml. The application-config.yml file needs to be updated with the correct IP addresses and ports for the Postgres database and the KeyCloak server. It also needs to be updated with appropriate performance settings based on the machine’s resources.
Settings to be modified in the Spring config are:

1
2
3
4
5
spring.datasource.hikari.maximum-pool-size: #vCPU
job-api.maximumLocalTaskThreads: 2*#vCPU
configuration.discoverCodeDbThreads: #vCPU/2
configuration.discoverMetricsDbThreads: #vCPU/2
keycloak.auth-server-url: <URL to KeyCloak server>:<KeyCloak port>
Warning

KeyCloak and the Postgres database must be up and running, configured, and available for connection before starting the Mining API server; otherwise, it will fail to start.

Start the Mining API server by executing the following command via an appropriate CLI:

java -jar --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -Xms<40%> -Xmx<85%> ./mining-api-server*.jar
Info

Percentages values here are based on the machine’s total memory, assuming the machine is dedicated to running this server. Example for a machine with 128GB total memory: -Xms50G -Xmx100G.

Example start command for a machine with 64vCPU and 128GB RAM

1
2
3
4
5
6
7
8
java -jar --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -Xms50G -Xmx100G mining-api-server*.jar \ 
--spring.profiles.active=authorized-access \ 
--spring.datasource.hikari.maximum-pool-size=64 \ 
--job-api.maximumLocalThreadTasks=128 \ 
--configuration.discoverCodeDbThreads=32 \ 
--configuration.discoverMetricsDbThreads=32 \ 
--keycloak.auth-server-url=http://localhost:8080/auth \ 
--spring.datasource.url=jdbc:postgres:remote:localhost:2424/mining

Alternatively, to the inline Spring configuration options, Deloitte can provide a default spring-configuration.yml file that can be bundled within the Mining API server zip file. This file contains the full properties used for Spring and can be modified as mentioned above. To use the Spring configuration file, adjust the jar command:

java -jar --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED -Xms50G -Xmx100G mining-api-server*.jar -spring.config.location=<path-to-custom-config-file>

In addition, the server start can be set up as a service in a Linux system by creating a systemd service. This provides benefits like automatic restart and service control via systemctl.
Instructions for this step are intentionally left out, as it is within the administrator's discretion to configure system services.

Clustered Mining environment with AWS ASG

This section will outline the setup of a full-fledged auto-scaled Mining API server cluster on EC2, with full integration of the API server into the Auto scaling life cycle for graceful instance termination and central log and job result storage via an EFS (Elastic File System).

Simple Notification Service (SNS)

An SNS is required to enable the Auto Scaling group to send life cycle messages to the Simple Queue Service (SQS) queue created in the next step.

Step 1:

In the Amazon SNS Console, choose Topics and select Create Topic.

Step 2:

Type select Standard.
Give the topic a meaningful name click Create Topic to finalize the SNS setup.

Simple Queue Service (SQS)

A queue will be used by the Auto Scaling group to send life cycle notifications to be processed by the Mining API server instances.

Step 1:

In the Amazon SQS Console, select Create Queue.

Step 2:

Type select Standard.

Step 3:

Under Configuration, customize the SQS queue according to your needs.

Step 4:

Under Access policy, enter your AWS account number and the ARN of the IAM role that will be assigned to the EC2 instances.
Required permissions in addition to the default clustering permissions are "sqs:GetQueueUrl", "sqs:ReceiveMessage" and "sqs:DeleteMessage".

Step 5:

Apply further configurations as needed (for example, encryption). Then select Create Queue.

Step 6:

In the queue overview page, select Subscribe to Amazon SNS topic and select the previously created SNS topic.

Elastic File System (EFS)

An EFS (Elastic File System) storage mounted into each EC2 instance enables us to store all log files created by all instances at a central location, even after they are terminated.
A centralized EFS for logging is required for Discovery if the API server is clustered. This way the log files can be returned to the Mining IDE once a code scanning job has finished.

Step 1:

Create a dedicated EFS and customize it according to your needs (automatic backups, life cycle management, encryption, etc.).

Step 2:

For the network access settings, ensure that the same subnet is selected that will also be used by your cluster or Auto Scaling setup. Otherwise, the EC2 machines won't be able to mount the EFS storage. Remove the default security groups that may have already been applied, and select a security group that is different from the one that will be used for the EC2 instances.

Info

The security group used for the EFS needs an inbound rule that allows NFS traffic with the source being the security group used for the EC2 machines. While the security group for the EC2 machines needs an outbound rule allowing NFS traffic with the destination being the security group of the EFS.

Info

An EFS storage can be mounted in different ways. Choose your created file system and click Attach to see the possible commands. When mounting via DNS you have to make sure that DNS lookups are enabled and working in your VPC.

Amazon Machine Image (AMI)

Every machine started by the Auto Scaling Group will use the same AMI that must already contain all required resources for the machine to participate.

Step 1:

Set up a machine as described above in the Mining API Server.

All resources must be copied to the instance, like the Mining API server jar, application config YML, and log4j configuration. The application config must contain the following properties:

  • tagKey  and tagValue  that contain a unique key and value pair to make sure that only machines using the same key/value pair are eligible to form a cluster (the values chosen here are required later on when creating the launch template). Use a meaningful value, especially for tagValue.
Warning

If you do not set this tag, then all api-server instances on all machines in the region of your AWS account will be used to form a cluster, which is not recommended for this Auto Scaling setup.

  • lifecycleQueueName: Provide the name of the SQS queue created previously.

  • jobResultFolder: Since all job results should be stored in the centralized EFS storage, provide the path of the mount point where the EFS storage will be mounted to during the EC2 instance launch.

  • logFolder: Since all logs should be stored in the centralized EFS storage, provide the path of the mount point where the EFS storage will be mounted to during the EC2 instance launch.

Example configuration in YML application config

...
job-api:
    jobResultFolder: /mnt/efs/jobResults
    cluster:
        name: mining-api-server-cluster
        mode: aws
        network:
            port: 5710
            portAutoIncrement: true
            aws:
                portRange: 5710-5715
                ec2:
                    tagKey: api-server-cluster
                    tagValue: mining-autoscaling-cluster
                    autoscaling:
                        lifecycleQueueName: mining-api-server-queue
    log:
        logFolder: /mnt/efs/logs
...

Step 2:

Provide any other AWS properties you may need for your setup as described in Mining Job API - AWS.

Step 3:

In the Amazon EC2 Instances Console, select the machine that you have set up. Choose Actions → Image and templates → Create image to create your Mining Server AMI.

Launch template

The launch template will be used by the Auto Scaling group to launch the EC2 instances.

Step 1:

Create a Launch Template for the AMI created above.

Step 2:

For Resource tags, add the following, while making sure to set Resource types to both Instances and Volumes:

  • Key="Name" and Value="".

  • Additionally the tagKey and tagValue pair that you already entered in the application config above during your AMI setup. Based on the example above Key="api-server-cluster" and Value="mining-autoscaling-cluster"

Step 3:

For Advanced details, set the following:

  • IAM instance profile should be set to the cluster and SQS-capable IAM role used during the SQS queue creation.

  • In User data, enter everything that should be executed when a new EC2 instance is launched.

Example Java launch command for the Mining API server

1
2
3
4
5
6
7
#!/bin/bash
sudo yum install -y amazon-efs-utils
sudo mkdir /mnt/efs
sudo mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport <your-efs-ip>:/ /mnt/efs
sudo chown ec2-user:ec2-user /mnt/efs
cd /home/ec2-user
java -Dlogging.config=log4j2.xml -jar mining-api-server-dist-21.0.0.jar --spring.config.location=application-cluster.yml

This installs the Amazon EFS utils, creates a mount point, actually mounts the EFS storage, and changes the owner of the mount point to ensure that the API server can write to the mount point. The mount point used here is "/mnt/efs", as already configured in the application config above. The IP "10.195.130.75" for the mount command should be replaced with the IP of your previously created EFS storage.
Finally, we switch to the directory that contains all Mining API server resources and execute the Java command to start it. The example above expects an explicit log4j2 configuration to be present in this location, along with the spring configuration file.

Step 4:
Finalize the launch template by customizing it according to your needs.

Load balancer

Before creating the Auto Scaling group, we will create a load balancer. Technically, this could also be directly created as part of the Auto Scaling group creation, but this way more fine-grained configuration can be done.

Step 1:

Create an Application Load Balancer and configure the name, scheme, HTTP port, availability zones with subnets, and security groups according to your needs.

Step 2:

Give the target group a descriptive name, set Target Type to Instance, set the protocol to HTTP, and the port to the one where your API server is configured to listen on (8080 by default).

Step 3:

Don't register any target and directly click Next: Review, and finally Create.

Auto Scaling Group

The final step is to create the Auto Scaling group for the Mining API server cluster.

Step 1:

Create an Auto Scaling group and choose the launch template created in the previous steps. Make sure to select Latest for the Version. This way, the auto scaling group will always automatically pick up the latest changes made to the launch template. Then click Next.

Step 2:

For Network, select the same subnet chosen during the load balancer creation.

Step 3:

Select Attach to an existing load balancer and choose the one created in the previous step.

Step 4:

Check Enable group metrics collection within CloudWatch and click Next.

Step 5:

Enter the capacities for your Auto Scaling group in the Group size section. Make sure that the Desired capacity and Minimum capacity should have a minimum value of 1. Otherwise, no machine will be started.

Keep Scaling policies at None and click Next.

Step 6:

Click Add Notification and select the SNS topic created in the previous step.

Step 7:

Finalize the creation of your Auto Scaling group by customizing it according to your needs, for example, with additional tags.

Step 8:

Create the policies that decide when to scale out/in the number of possible machines in the Auto Scaling group.
Ensure the following settings when creating policies:

  • For the Policy type, select Simple scaling.

  • Either use an existing CloudWatch alarm or create a new one.

  • To take the action, select Add 1 capacity units for the scale-out policy and Remove 1 capacity units for the scale-in policy.

Warning

Whatever type of policy you define, you have to make sure that your policy never reduces more than 1 machine per execution. This ensures that no more than one machine is being terminated at a time. Otherwise, the Hazelcast cluster may get into a corrupt state. Data may be corrupted when two machines are being terminated that both hold the primary and backup copies of a specific data structure.

Step 9:

Create the life cycle hooks that will send the life cycle events of the Auto Scaling group to your SQS queue, so that the API server can react. The type of life cycle hooks required cannot be created from within the AWS management console, but only from a Command Line Interface (CLI).

Access one of the EC2 instances initialized by the Auto Scaling group via SSH, open a CLI and and enter the following two commands to create two life cycle hooks.

1
2
3
aws autoscaling put-lifecycle-hook --lifecycle-hook-name <your-lifecycle-hook-name> --auto-scaling-group-name <your-autoscaling-group-name> --lifecycle-transition autoscaling:EC2_INSTANCE_LAUNCHING --notification-target-arn <your-SQS-queue-ARN> --role-arn <IAM-role-ARN> --region <your-region> --default-result CONTINUE 

aws autoscaling put-lifecycle-hook --lifecycle-hook-name <your-lifecycle-hook-name> --auto-scaling-group-name <your-autoscaling-group-name> --lifecycle-transition autoscaling:EC2_INSTANCE_TERMINATING --notification-target-arn <your-SQS-queue-ARN> --role-arn <IAM-role-ARN> --region <your-region> --default-result ABANDON

The first command is for the EC2_INSTANCE_LAUNCHING event when additional instances are about to be launched, while the second one handles the EC2_INSTANCE_TERMINATING event when an instance is no longer required and should be terminated.

Replace the command parameters:

  • <your-life cycle-hook-name>: How the life cycle hook should be called. For example, "Launch_Api-Server_Instance" and "Terminate-Api-Server_Instance".

  • <your-autoscaling-group-name>: The name of the Auto Scaling group created above.

  • <your-SQS-queue-ARN>: The ARN of the SQS queue created above.

  • <IAM-role-ARN>: The ARN of an IAM role that must have a trust policy to the EC2 Auto Scaling Service and the permissions "sqs:SendMessage", "sqs:GetQueueUrl", and "sns:Publish". This must not be the same IAM role that is used by the EC2 instances of the Auto Scaling group or the commands will not succeed.

  • <your-region>: The AWS region.

In the Auto Scaling group console, select your group and navigate to Instance management. You should now see the two created life cycle hooks attached to the Auto Scaling group.

Step 10:

In the Amazon SQS Console, select your queue and click Send and Receive Messages.
Select Poll for messages, and you should see at least two messages that have automatically been sent during the creation of the life cycle hooks for testing purposes.

Step 11:

Your Auto Scaling environment is now set up.
You can access the Mining UI and attach the Mining IDE to the Mining API server via the DNS name assigned to the load balancer.

Example specifications

Mining API server

Component

Specifications

EC2 Linux AMI

  • c5.12xlarge (pre auto scaling)

    • 48vCPU, 96GB memory
  • c5.12xlarge (pre auto scaling)

    • 16vCPU, 32GB memory

EBS

  • gp2

  • 100GB

Security Group

  • Port 8080: HTTP requests in VPC (For Mining IDE, KeyCloak, Postgres, Users, etc.)

Keycloak

Component

Specifications

EC2 Linux AMI

  • t3a.large
    • 2vCPU, 8GB memory

Amazon RDS for PostgreSQL

  • db.t3.medium
    • 2vCPU, 4GB memory

EBS

  • gp2

  • 100GB

Security Group

  • Port 8080: HTTP requests to the admin console for configuration

  • Other ports may be necessary for SSL, SSO redirect, etc.

Mining IDE

Component

Specifications

EC2 Windows AMI

  • m5.2xlarge
    • 8vCPU, 32GB memory

EBS

  • io2

  • 150GB

  • 5,000 IOPS

  • needed for storing source code files and moving them to the workspace

Security Group

  • Port: 3389 for RDP (Remote Desktop) access

  • Needs to be able to connect to Mining API server and KeyCloak server

OrientDB (deprecated)

Component

Specifications

EC2 Linux AMI

  • m5d.8xlarge
    • 32vCPU, 256GB memory

Instance store

  • 2 x 300GB, NVMe SSD

Security Group

  • Port 2480: HTTP requests in VPC

  • Port 2424: All requests from Mining API server, none otherwise

Backups

  • Scheduled job that exports the database to a dedicated S3 location

EFS

  • Standard Storage 350GB/month.

    • Can be modified to configure infrequent access if needed to reduce costs.
  • Used for commonly mounted logging locations of the various services (Postgres logs, Mining API server logs, KeyCloak, etc.).

    • This needs to be configured for each application to write its logs to this common mounted location instead of a local EBS or instance storage.

S3

  • Standard Storage 300GB/month.
    • Can be modified for the POST/GET requests count.

Approximate monthly cost estimate

Based on the AWS Pricing Calculator (https://calculator.aws/), using:

  • Pricing Model: EC2 instance savings plan

  • Reservation Term: 1 year

  • Payment option: No Upfront

Mining Service AWS Service Type/Size Monthly Cost (as of 01/2021)
Mining API Server EC2 c5.9xlarge $700
Mining API Server Storage EBS gp2, 100GB (x2) $20
KeyCloak Server EC2 t3a.large $35
KeyCloak Server Persistence RDS db.t3.medium $50
Common Logs EFS 350GB $105
DB Backup Storage S3 500GB $10
OrientDB (deprecated) EC2 m5.8xlarge $830
Total $1,750