Skip to content

Mining Extensions

Mining Extensions are Java classes that implement a certain interface and register some additional functionality for the mining product. The extensions are registered automatically at runtime if they are put on the classpath of the mining-api-server (and they are located in the correct package).

We currently provide extension points to:

  • export mining data to a custom format

  • add additional custom pages to the Mining UI

  • provide additional MetricsContributor for Discover metrics process


Target audience

Target audience is Mining and Adoption Pod DevOps team members, as well as mining power-users and delivery professionals. These readers typically configure, extend, or operationalize Mining beyond standard end-user workflows. The remaining sections assume familiarity with Java projects, server configuration, and Mining deployment concepts.


Implementing Mining Extensions

To implement a mining extension, it is recommended to create a new Java project. Any simple Maven project that compiles to a jar file will do.

Configuring projects

You need to add the "mining-shared" dependency:

1
2
3
4
5
<dependency>
    <groupId>innowake.products.mining</groupId>
    <artifactId>mining-shared</artifactId>
    <version>99.9.99-TRUNK-MINING-SNAPSHOT</version>
</dependency>

To access mining data within your application (for example, to implement an export extension), you will also want to include the "mining-data" dependency:

1
2
3
4
5
    <dependency>
        <groupId>innowake.products.mining</groupId>
        <artifactId>mining-data</artifactId>
        <version>99.9.99-TRUNK-MINING-SNAPSHOT</version>
    </dependency>
Warning

Disclaimer: There is currently no stable, public API available to access mining data. Therefore, include and use the "minig-data" project with the expectation that your project may break with a future update. A stable, approved API may come in the future.

Loading extensions

To load your extension into the mining-api-server, you need to compile your extension to a jar file and then add the jar file (and all of its dependencies) to the classpath of mining-api-server. All extensions that are underneath the package innowake.mining.extensions will then be automatically registered.

To add your extension to the classpath, use the loader.path system property (or the LOADER_PATH environment variable) and launch the application using the org.springframework.boot.loader.launch.PropertiesLauncher class. More information on the topic can be found here.

Example launch options from the command-line:

    java -Dloader.path="/path/to/my-extension.jar,/path/to/my-extension-dependency.jar" -cp mining-api-server-99.9.99-TRUNK-MINING-SNAPSHOT.jar org.springframework.boot.loader.launch.PropertiesLauncher

Use the below command for mining versions before 99.9.0-alpha-202409260030:

java -Dloader.path="/path/to/my-extension.jar,/path/to/my-extension-dependency.jar" -cp mining-api-server-99.9.99-TRUNK-MINING-SNAPSHOT.jar org.springframework.boot.loader.PropertiesLauncher
Info

On the command line, use the -cp option, not the -jar option to specify the mining-api-server jar file. Otherwise, the main class argument (org.springframework.boot.loader.launch.PropertiesLauncher) will be ignored.

How to ensure the extension was loaded?

Add a @PostConstruct  method to your extension class in which you log the successful loading:

1
2
3
4
@PostConstruct
public void init() {
    logger.info("Extension XYZ loaded successfully.");
}

That way, you can check the Mining server logs to ensure that everything was set up correctly.

Showing extensions on the UI pages

You can show your extension on the export page, optionally. A "Mining Export Extension" will show itself on the export page by default, whereas a "Mining Job Extension" will not show. This option can be changed according to the requirement by overriding the method getShowOnPage() of the ShowOnPageExtension interface, inside your extension class, as shown below.

Info

Currently, Mining supports showing UI extensions only on the Export page and Customizable Tables.

1
2
3
4
    @Override
    public ShowOnPage getShowOnPage() {
       return new ShowOnPage(true, "Extension label", "Extension Category", "Extension location");
    }

It returns a ShowOnPage object, which can be created by providing values for the parameters :

  • boolean show: This accepts a boolean value. true if the extension should be shown on the page, false otherwise

  • String label: This accepts the string value for the label of the extension

  • String category: This accepts the string value for the category of the extension, defaults to 'Extension' if null or empty

  • String usage: This accepts the string value, which should be one of the Usage values from the Usages class if it needs to be shown in the table or 'ExportPage' to show in the export page, default to 'ExportPage' if null or empty.

    • Use the following mapping for the locations
1
2
3
4
5
6
7
8
'ExportPage' or Empty  ->  Export page in UI
Usages.MINING_UI_MODULES_TABLE -> Modules Reporting Page
Usages.MINING_UI_ANNOTATIONS_TABLE -> Annotation Reporting Page
Usages.MINING_UI_REACHABILITY_TABLE -> individual RB Table
Usages.MINING_UI_REACHABILITY_BLOCKS_OVERVIEW_TABLE -> Reachability Overview table
Usages.MINING_UI_DATADICTIONARY_TABLE -> Data dictionary Reporting Page

You can see the section "Pages / Locations on Mining UI" in the Usagses.Java (Mining-shared) for other usages that can be used.

Mining Extension types

The following extensions can currently be registered – each description also includes a sample implementation. Together, they cover the main extension patterns for adding export, job, or UI-related functionality. Review the type that matches the behavior you want to introduce before starting implementation.

Mining Export Extension

Info

Eclipse Project with example implementation: mining-export-extension-example.zip.

This kind of extension contributes an export format. The export can be triggered via an HTTP request to:

GET http://localhost:8080/api/v1/projects/1/export/<formatIdentifier>?<options>

Below is an example for a simple export extension:

package innowake.mining.extensions.export;

import innowake.mining.shared.extensions.MiningExportExtension;

@Service
public class HelloWorldExporter implements MiningExportExtension {

    @Override
    public String getFormatIdentifier() {
        return "hello-world";
    }

    @Override
    public String getDescription() {
        return "A simple extension for demonstration purposes";
    }

    @Override
    public ExportValue export(Long projectId, Map<String, List<String>> parameters) throws ExportException {
        return new ExportValue() {

            @Override
            public InputStream getInputStream() {
                return new ByteArrayInputStream("Hello, World".getBytes(StandardCharsets.UTF_8));
            }

            @Override
            public String getFileName() {
                return "hello.csv";
            }

            @Override
            public String getContentType() {
                return "text/csv";
            }
        };
    }

    @Override
    public ShowOnPage getShowOnPage() {
    return new ShowOnPage(true, "Extension label", "Extension Category", "Extension location");
    }
}

Mining Job Extension

Info

Eclipse Project with example implementation: mining-job-extension-example.zip.

This kind of extension contributes the implementation of a Job that can be executed on a mining project and runs on the server. Unlike an export extension, a job is executed asynchronously ("in background") - its status can be observed via the Job REST API. The Job API also provides a way to cancel a Job. The Job can optionally produce a result. These feature make it the most flexible type of extension that is suitable for long-running and computationally intensive tasks.

The implementation of a Job Extension must provide a factory method that creates a Job instance. The returned instance will then be scheduled for execution via the job framework. It can be triggered via an HTTP request to:

POST http://localhost:8080/api/v2/projects/<projectId>/job-extensions/<extensionId>

Optional query parameters are passed in the parameters  Map to the createJob() method of your extension (see below). The request body is passed as the inputDataargument and may contain arbitrary data (or it can be empty, if the job requires no input data). This allows your extension to implement file upload / import functionality.

You can implement the getParameterDescriptions() and getUploadDescription() methods to describe the parameters that can be passed to your extension, respectively, and the file that can be uploaded. This is especially useful when also overriding getShowOnExportPage() to show the extension on the Export page of the Mining UI, as it will allow users to interactively input the parameters or choose a file through the UI. Note that even when not implementing  getParameterDescriptions() or getUploadDescription() it is still possible to pass parameters or upload a file when manually sending the HTTP request or using the mining-cli.

Below is an example for a simple job extension:

package innowake.mining.extensions.job;

@Service
public class HelloWorldJobExtension implements MiningJobExtension<String> {

    private static class HelloWorldJob extends Job<String> {

        private final String greeting;

        public HelloWorldJob(final String greeting) {
            this.greeting = greeting;
        }

        @Override
        protected Result<String> run(ProgressMonitor progressMonitor) {
            return new Result<>("Hello " + greeting);
        }
    }

    @Autowired
    private ObjectMapper objectMapper;

    @Override
    public String getIdentifier() {
        return "hello-world";
    }

    @Override
    public String getDescription() {
        return "Hello World Example Job";
    }

    @Override
    public Job<String> createJob(final Long projectId, final Map<String, List<String>> parameters, HttpEntity<byte[]> inputData) {
        /* read parameters and apply defaults */
        final String greeting = Optional.ofNullable(parameters.get("greeting")).map(t -> t.get(0)).orElse("World");

        /* check uploaded file and parse JSON */
        if ( ! inputData.hasBody()) {
            throw new IllegalArgumentException("Input data is mandatory and must contain the data to be imported in JSON format.");
        }
        final MyData data = objectMapper.readValue(inputData.getBody(), MyData.class);

        /* MyData could now be passed to the Job (not shown) */

        return new HelloWorldJob(greeting);
    }

    @Override
    public NatureType getRequiredNature() {
        return MINING;
    }

    @Override
    public RoleType getRequiredRole() {
        return VIEWER;
    }

    @Override
    public ShowOnPage getShowOnPage() {
        return new ShowOnPage(true, "Extension label", "Extension Category", "Extension location");
    }

    @Override
    public List<ParameterDescription> getParameterDescriptions() {
        return Arrays.asList(
                new ParameterDescription("greeting", /* name */
                        "The greeting to be sent by the example extension", /* description */
                        ParameterDescription.ParameterType.STRING, /* parameter type */
                        false, /* mandatory parameter yes/no */
                        "World", /* default value for non-mandatory parameters */
                        false, /* parameter can be given multiple times yes/no */
                        "World", /* example value show to the user */
                        Collections.emptyList()) /* possible values - if specified allows to chose the values from a list, otherwise any value can be entered */
        );
    }

    @Override
    public UploadDescription getUploadDescription() {
        return UploadDescription.with("JSON data file", /* short descriptive name */
                "JSON file containing data to be imported", /* description */
                true, /* file upload is mandatory yes/no */
                ".json,application/json"); /* possible file types (can be omitted to allow any type) */
    }
}

Mining UI Extension

Info

Eclipse Project with example implementation: mining-ui-extension-example.zip.

This kind of extension contributes a new page within the Mining UI. The page will be displayed in the navigation menu on Mining UI. It can optionally receive the id of the currently selected project and client.

/*
 * Copyright (c) 2021 innoWake gmbh Germany. All rights reserved.
 */
package innowake.mining.extensions.ui;

import innowake.mining.shared.extensions.MiningWebUiExtension;

@Service
public class HelloWorldExtension implements MiningWebUiExtension {

    @Override
    public String getName() {
        return "Hello World";
    }

    @Override
    public Kind getKind() {
        /* Note: IFRAME is the only supported kind as of writing this */
        return Kind.IFRAME;
    }

    @Value("${server.servlet.context-path:}")
    private String servletContextPath;

    @Override
    public Map<Property, String> getProperties() {
        final Map<Property, String> props = new HashMap<>();

        final String prefix = "/".equals(servletContextPath) ? "" : servletContextPath; // Adding this prefix is not required if the Iframe is served externally.       
        props.put(Property.IFRAME_SRC, prefix + "/my/extension-url/index.html?clientId=${clientId}&projectId=${projectId}");

        return props;
    }

}

When using a path as IFRAME_SRC (as in the example above) then the static resources for your extension will be served from the classpath (from one of the default locations used by Spring Boot). You can also embed an external web page in the iFrame by specifying its full URL (for example. "http://my-other-domain.com/my/extension"). 

In order to serve the resources (HTML etc.) for your extension from the classpath, you need an additional configuration. The default security configuration of mining-api-server will set the X-Frame-Options: DENY header on all resources. This will effectively prevent your extension from functioning since it is displayed in an iframe. Also, we enforce Authentication which may also cause problems for your static resources.

To fix this, add another class to your extension project:

import static org.springframework.security.web.util.matcher.AntPathRequestMatcher.antMatcher;

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.annotation.Order;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.web.SecurityFilterChain;

@Configuration
@EnableWebSecurity
public class AuthorizedExampleExtensionConfig {

    @Bean
    @Order(1)
    public SecurityFilterChain configure(final HttpSecurity http) throws Exception {
        return http
                .securityMatcher(antMatcher("/my/extension-url/**"))
                .headers(headers -> headers.frameOptions().sameOrigin())
                .authorizeHttpRequests(authorize -> authorize.requestMatchers(antMatcher("/my/extension-url/**"))
                        .permitAll())
                .build();
    }
}

Now you can put the resources for your extension under "src/main/resources/static/my/extension-url" (or whatever path you configured as IFRAME_SRC).

Please see the attached Eclipse project for a working example.

Discovery Contributor

This extension enables you to provide custom contributors for Discovery. These contributors allow you to discover new types of Modules from source code, or to extend the already supported Modules with additional information. This will be helpful for clients who have different language dialects or need to support any client specific implementation.

Info

The class implementing DiscoveryContributor or DiscoveryContributorFromSource has to be public and needs to be annotated with @Component. It also needs to be underneath the innowake.mining.extensions package.

Here is an example for an extension that extends the already discovered Java classes with additional dependency information based on a customer-specific annotation:

/**
 * Example contributor extension which creates additional dependecies from Java classes (JAVA_TYPE) to Stored Procedures (SQL_STORED_PROCEDURE)
 * based on an "@StoredProcedure" annotation.
 */
@Component
public class JavaMetricsContributorExtension implements DiscoveryContributor {

    private static final Pattern PATTERN = Pattern.compile("@StoredProcedure\\(\"(\\w+)\"\\)");

    @Override
    public void contribute(final DiscoveryBuilder builder, final DiscoveryContext context) {
        /* anchorTo() is used to "attach" to existing Modules which were already discovered by another contributor.
         * In this case, the built-in Java support of Discovery has already discovered the Java classes (they have the type JAVA_TYPE).
         * So our contributor extension can simply "anchor to" the existing Modules and extend them with additional information.
         *
         * To do this, we use deferAction(). Discovery processes files in parallel for optimal performance. Using deferAction(), we can schedule
         * some code to be executed after processing of all files is completed. This is ideal for our use-case, because we want our custom
         * contributor to run after all Java classes and Stored Procedures have been discovered. */
        builder.anchorTo(new ModuleFilter().setTypes(ModuleType.JAVA_TYPE))
                .deferAction("analyzeStoredProcedures"); /* must match method name of method annotated with @DeferredAction */
    }

    @DeferredAction
    public void analyzeStoredProcedures(final Module module, final ModuleBuilder moduleBuilder, final SourceObject sourceObject) {
        String content = sourceObject.getContent();
        /* the SourceObject is for the entire COMPILATION_UNIT (i.e. .java file) but this deferred action runs separately on each class
         * since one COMPILATION_UNIT can contain multiple classes, we make sure to only look at the part of the file that contains the current class */
        final ModuleLocation classLocation = module.getLocation();
        if (classLocation != null) {
            content = content.substring(classLocation.getOffset(), classLocation.getLength());
        }
        final Matcher matcher = PATTERN.matcher(content);
        if (matcher.find()) {
            final String storedProcedure = matcher.toMatchResult().group(1);
            moduleBuilder.declareDependency(Relationship.CALLS, new ModuleFilter().setNames(storedProcedure).setTypes(ModuleType.SQL_STORED_PROCEDURE))
                    .setBinding(Binding.LATE);
        }
    }
}

Metrics Contributor

This extension enables to provide custom Metrics contributors. This will be helpful for clients who have different dialects or to support any client specific implementation.

Info

The class implementing MetricsContributorExtension has to be public and needs to be annotated with @Component. It also needs to be underneath the innowake.mining.extensions package.

@Component
public class JavaMetricsContributorExtension implements MetricsContributorExtension {

  @Override
  public boolean accept(final ModelArtifact entry, final Phase phase) {
    return entry.getType() == ResolveTarget.JAVA_COMPILATION_UNIT;
  }

  @Override
  public MetricsContributor init(final Parameters parameters, final SourceObjectDao sourceObjectDao, final @Nullable TimedWorker timedWorker) {
    return new JavaMetricsContributor(sourceObjectDao, parameters);
  }
}

class JavaMetricsContributor implements MetricsContributor {

  private static final Pattern PATTERN = Pattern.compile("@StoredProcedure\\(\"(\\w+)\"\\)");

  private final SourceObjectDao sourceObjectDao;
  private final Parameters parameters;

  public JavaMetricsContributorTest(final SourceObjectDao sourceObjectDao, final ContributorParameters parameters) {
    this.sourceObjectDao = sourceObjectDao;
    this.parameters = parameters;
  }

  @Override
  public List<ModelArtifact> collectModules(final ModelArtifact entry) throws DiscoveryException {
    return Collections.singletonList(entry);
  }

  @Override
  public void calculateGenericMetrics(final ModelArtifact artifact) throws DiscoveryException {
  }

  @Override
  public void calculateDependentMetrics(final ModelArtifact artifact) throws DiscoveryException {
    final String content = getSourceObject(artifact).getContent();
    final Matcher matcher = PATTERN.matcher(content);
    if (matcher.find()) {
      final String storedProcedure = matcher.toMatchResult().group(1);
      artifact.addDependency(new ModelDependency().setLateBinding()
          .setTarget(new ModelArtifact().setName(storedProcedure).setType(ResolveTarget.SQL_STORED_PROCEDURE).validate()).validate());
    }
  }

  private SourceObject getSourceObject(final ModelArtifact artifact) throws DiscoveryException {
    final String path = artifact.getPath().orElseThrow(() -> new DiscoveryException("Entry file is not present: " + artifact.getName()));

    final SourceObject sourceObject = Assert.assertNotNull(sourceObjectDao).findByPath(Assert.assertNotNull(parameters).getProjectId(), path);
    if (sourceObject == null) {
      throw new DiscoveryException("Java object is unresolvable: " + path);
    }
    return sourceObject;
  }
}

Technical Taxonomy identification

If a technical taxonomy has to be identified for a module based on some patterns that cannot be achieved through Taxonomy propagation (or) through the existing technical taxonomy logic, then you can write a custom logic that should implement TaxonomyIdentifier interface. Although, this interface is not introduced specifically as a mining extension, one can still use it like all other extension.

Info

The class implementing TaxonomyIdentifier<ModuleBasePojo> has to be public and needs to be annotated with @Component. It also needs to be underneath the innowake.mining.extensions package. At the time of writing, this can take AstNodePojo , AstModel , DependencyModel .

Let's say you need to identify all the Cobol modules that have a CICS statement in them as the "Access CICS" taxonomy, then:

@Component
public class CicsTaxonomyIdentifier implements TaxonomyIdentifier<ModuleBasePojo> {
    private final ModuleService moduleService;

    @Override
    public List<Tuple2<Name, TypeName>> identify(final ModuleBasePojo moduleBasePojo) {
        if (moduleBasePojo.getTechnology() == Technology.COBOL) {
            return moduleService.findStatements(q -> q.ofModule(moduleBasePojo.identity())).stream()
                                    .map(StatementPojo::getType)
                                    .filter(StatementType.EXEC_CICS::equals)
                                    .findFirst()
                                        .stream()
                                        .map(type -> new Tuple2<>(Name.ACCESS_CICS, TypeName.PROGRAM_TYPE))
                                    .collect(Collectors.toList());
        }
    return Collections.emptyList();
    }
}

GenAI Context Retriever

Before a prompt is sent to a GenAI model, additional context is added to the prompt by context retrievers. It is possible to add custom context retrievers with a mining extension.

To do so you need to add a class implementing one of the ContextRetriever interfaces to the innowake.mining.server.genai.context package:

  • GenAiAnnotationContextRetriever

  • GenAiModuleContextRetriever

  • GenAiFunctionalBlockContextRetriever

  • GenAiReachabilityBlockContextRetriever

  • GenAiDataDictionaryContextRetriever

It is also possible to define multiple retrievers for the same kind of entity. Each retriever is supposed to return the context of the previously executed retriever + the newly retrieved context.

Example

If you wanted to add the custom properties of an annotation to the annotation description prompt, the context retriever could look like this:

package innowake.mining.server.genai.context;

import innowake.mining.shared.entities.AnnotationPojoPrototype;
import org.springframework.stereotype.Service;

@Service
public class ExampleAnnotationContextRetriever implements GenAiAnnotationContextRetriever {

    @Override
    public AiGenerationContext retrieve(final AnnotationContext context) {
        final AiGenerationContext previousContext = context.getGenerationContext();
        final AnnotationPojoPrototype annotationPojoPrototype = context.getAnnotation();
        if (annotationPojoPrototype.customProperties.isDefined()) {
            /* Return context containing context from previous retrievers + new context */
            return new AiGenerationContext(previousContext.getAdditionalPromptContext()
                    + annotationPojoPrototype.customProperties.get().toString());
        }
        /* No custom properties are available -> we return the previous context */
        return context.getGenerationContext();
    }
}

The String value that gets passed to the AiGenerationContext constructor will get added to the prompt.

Request UUID

  • If you don't use GenAI capabilities for the context retrieval you can ignore this

  • The context object (f.e. AnnotationContext) which gets passed to retrieve has a method getRequestUUID

  • Purpose: associate the context retrieval with the description generation for token usage tracing purposes

  • If you access f.e. the Knowledge Service in your context retriever, you should add the UUID to the request as part of the requestMetadata map at key "uuid"