Mining Extensions
Mining Extensions are Java classes that implement a certain interface and register some additional functionality for the mining product. The extensions are registered automatically at runtime if they are put on the classpath of the mining-api-server (and they are located in the correct package).
We currently provide extension points to:
-
export mining data to a custom format
-
add additional custom pages to the Mining UI
-
provide additional MetricsContributor for Discover metrics process
Target audience
Target audience is Mining and Adoption Pod DevOps team members, as well as mining power-users and delivery professionals. These readers typically configure, extend, or operationalize Mining beyond standard end-user workflows. The remaining sections assume familiarity with Java projects, server configuration, and Mining deployment concepts.
Implementing Mining Extensions
To implement a mining extension, it is recommended to create a new Java project. Any simple Maven project that compiles to a jar file will do.
Configuring projects
You need to add the "mining-shared" dependency:
| <dependency>
<groupId>innowake.products.mining</groupId>
<artifactId>mining-shared</artifactId>
<version>99.9.99-TRUNK-MINING-SNAPSHOT</version>
</dependency>
|
To access mining data within your application (for example, to implement an export extension), you will also want to include the "mining-data" dependency:
| <dependency>
<groupId>innowake.products.mining</groupId>
<artifactId>mining-data</artifactId>
<version>99.9.99-TRUNK-MINING-SNAPSHOT</version>
</dependency>
|
Warning
Disclaimer: There is currently no stable, public API available to access mining data. Therefore, include and use the "minig-data" project with the expectation that your project may break with a future update. A stable, approved API may come in the future.
Loading extensions
To load your extension into the mining-api-server, you need to compile your extension to a jar file and then add the jar file (and all of its dependencies) to the classpath of mining-api-server. All extensions that are underneath the package innowake.mining.extensions will then be automatically registered.
To add your extension to the classpath, use the loader.path system property (or the LOADER_PATH environment variable) and launch the application using the org.springframework.boot.loader.launch.PropertiesLauncher class. More information on the topic can be found here.
Example launch options from the command-line:
| java -Dloader.path="/path/to/my-extension.jar,/path/to/my-extension-dependency.jar" -cp mining-api-server-99.9.99-TRUNK-MINING-SNAPSHOT.jar org.springframework.boot.loader.launch.PropertiesLauncher
|
Use the below command for mining versions before 99.9.0-alpha-202409260030:
| java -Dloader.path="/path/to/my-extension.jar,/path/to/my-extension-dependency.jar" -cp mining-api-server-99.9.99-TRUNK-MINING-SNAPSHOT.jar org.springframework.boot.loader.PropertiesLauncher
|
Info
On the command line, use the -cp option, not the -jar option to specify the mining-api-server jar file. Otherwise, the main class argument (org.springframework.boot.loader.launch.PropertiesLauncher) will be ignored.
How to ensure the extension was loaded?
Add a @PostConstruct method to your extension class in which you log the successful loading:
| @PostConstruct
public void init() {
logger.info("Extension XYZ loaded successfully.");
}
|
That way, you can check the Mining server logs to ensure that everything was set up correctly.
Showing extensions on the UI pages
You can show your extension on the export page, optionally. A "Mining Export Extension" will show itself on the export page by default, whereas a "Mining Job Extension" will not show. This option can be changed according to the requirement by overriding the method getShowOnPage() of the ShowOnPageExtension interface, inside your extension class, as shown below.
Info
Currently, Mining supports showing UI extensions only on the Export page and Customizable Tables.
| @Override
public ShowOnPage getShowOnPage() {
return new ShowOnPage(true, "Extension label", "Extension Category", "Extension location");
}
|
It returns a ShowOnPage object, which can be created by providing values for the parameters :
-
boolean show: This accepts a boolean value. true if the extension should be shown on the page, false otherwise
-
String label: This accepts the string value for the label of the extension
-
String category: This accepts the string value for the category of the extension, defaults to 'Extension' if null or empty
-
String usage: This accepts the string value, which should be one of the Usage values from the Usages class if it needs to be shown in the table or 'ExportPage' to show in the export page, default to 'ExportPage' if null or empty.
- Use the following mapping for the locations
| 'ExportPage' or Empty -> Export page in UI
Usages.MINING_UI_MODULES_TABLE -> Modules Reporting Page
Usages.MINING_UI_ANNOTATIONS_TABLE -> Annotation Reporting Page
Usages.MINING_UI_REACHABILITY_TABLE -> individual RB Table
Usages.MINING_UI_REACHABILITY_BLOCKS_OVERVIEW_TABLE -> Reachability Overview table
Usages.MINING_UI_DATADICTIONARY_TABLE -> Data dictionary Reporting Page
You can see the section "Pages / Locations on Mining UI" in the Usagses.Java (Mining-shared) for other usages that can be used.
|
Mining Extension types
The following extensions can currently be registered – each description also includes a sample implementation. Together, they cover the main extension patterns for adding export, job, or UI-related functionality. Review the type that matches the behavior you want to introduce before starting implementation.
Mining Export Extension
Info
Eclipse Project with example implementation: mining-export-extension-example.zip.
This kind of extension contributes an export format. The export can be triggered via an HTTP request to:
| GET http://localhost:8080/api/v1/projects/1/export/<formatIdentifier>?<options>
|
Below is an example for a simple export extension:
| package innowake.mining.extensions.export;
import innowake.mining.shared.extensions.MiningExportExtension;
@Service
public class HelloWorldExporter implements MiningExportExtension {
@Override
public String getFormatIdentifier() {
return "hello-world";
}
@Override
public String getDescription() {
return "A simple extension for demonstration purposes";
}
@Override
public ExportValue export(Long projectId, Map<String, List<String>> parameters) throws ExportException {
return new ExportValue() {
@Override
public InputStream getInputStream() {
return new ByteArrayInputStream("Hello, World".getBytes(StandardCharsets.UTF_8));
}
@Override
public String getFileName() {
return "hello.csv";
}
@Override
public String getContentType() {
return "text/csv";
}
};
}
@Override
public ShowOnPage getShowOnPage() {
return new ShowOnPage(true, "Extension label", "Extension Category", "Extension location");
}
}
|
Mining Job Extension
Info
Eclipse Project with example implementation: mining-job-extension-example.zip.
This kind of extension contributes the implementation of a Job that can be executed on a mining project and runs on the server. Unlike an export extension, a job is executed asynchronously ("in background") - its status can be observed via the Job REST API. The Job API also provides a way to cancel a Job. The Job can optionally produce a result. These feature make it the most flexible type of extension that is suitable for long-running and computationally intensive tasks.
The implementation of a Job Extension must provide a factory method that creates a Job instance. The returned instance will then be scheduled for execution via the job framework. It can be triggered via an HTTP request to:
| POST http://localhost:8080/api/v2/projects/<projectId>/job-extensions/<extensionId>
|
Optional query parameters are passed in the parameters Map to the createJob() method of your extension (see below). The request body is passed as the inputDataargument and may contain arbitrary data (or it can be empty, if the job requires no input data). This allows your extension to implement file upload / import functionality.
You can implement the getParameterDescriptions() and getUploadDescription() methods to describe the parameters that can be passed to your extension, respectively, and the file that can be uploaded. This is especially useful when also overriding getShowOnExportPage() to show the extension on the Export page of the Mining UI, as it will allow users to interactively input the parameters or choose a file through the UI. Note that even when not implementing getParameterDescriptions() or getUploadDescription() it is still possible to pass parameters or upload a file when manually sending the HTTP request or using the mining-cli.
Below is an example for a simple job extension:
| package innowake.mining.extensions.job;
@Service
public class HelloWorldJobExtension implements MiningJobExtension<String> {
private static class HelloWorldJob extends Job<String> {
private final String greeting;
public HelloWorldJob(final String greeting) {
this.greeting = greeting;
}
@Override
protected Result<String> run(ProgressMonitor progressMonitor) {
return new Result<>("Hello " + greeting);
}
}
@Autowired
private ObjectMapper objectMapper;
@Override
public String getIdentifier() {
return "hello-world";
}
@Override
public String getDescription() {
return "Hello World Example Job";
}
@Override
public Job<String> createJob(final Long projectId, final Map<String, List<String>> parameters, HttpEntity<byte[]> inputData) {
/* read parameters and apply defaults */
final String greeting = Optional.ofNullable(parameters.get("greeting")).map(t -> t.get(0)).orElse("World");
/* check uploaded file and parse JSON */
if ( ! inputData.hasBody()) {
throw new IllegalArgumentException("Input data is mandatory and must contain the data to be imported in JSON format.");
}
final MyData data = objectMapper.readValue(inputData.getBody(), MyData.class);
/* MyData could now be passed to the Job (not shown) */
return new HelloWorldJob(greeting);
}
@Override
public NatureType getRequiredNature() {
return MINING;
}
@Override
public RoleType getRequiredRole() {
return VIEWER;
}
@Override
public ShowOnPage getShowOnPage() {
return new ShowOnPage(true, "Extension label", "Extension Category", "Extension location");
}
@Override
public List<ParameterDescription> getParameterDescriptions() {
return Arrays.asList(
new ParameterDescription("greeting", /* name */
"The greeting to be sent by the example extension", /* description */
ParameterDescription.ParameterType.STRING, /* parameter type */
false, /* mandatory parameter yes/no */
"World", /* default value for non-mandatory parameters */
false, /* parameter can be given multiple times yes/no */
"World", /* example value show to the user */
Collections.emptyList()) /* possible values - if specified allows to chose the values from a list, otherwise any value can be entered */
);
}
@Override
public UploadDescription getUploadDescription() {
return UploadDescription.with("JSON data file", /* short descriptive name */
"JSON file containing data to be imported", /* description */
true, /* file upload is mandatory yes/no */
".json,application/json"); /* possible file types (can be omitted to allow any type) */
}
}
|
Mining UI Extension
Info
Eclipse Project with example implementation: mining-ui-extension-example.zip.
This kind of extension contributes a new page within the Mining UI. The page will be displayed in the navigation menu on Mining UI. It can optionally receive the id of the currently selected project and client.
| /*
* Copyright (c) 2021 innoWake gmbh Germany. All rights reserved.
*/
package innowake.mining.extensions.ui;
import innowake.mining.shared.extensions.MiningWebUiExtension;
@Service
public class HelloWorldExtension implements MiningWebUiExtension {
@Override
public String getName() {
return "Hello World";
}
@Override
public Kind getKind() {
/* Note: IFRAME is the only supported kind as of writing this */
return Kind.IFRAME;
}
@Value("${server.servlet.context-path:}")
private String servletContextPath;
@Override
public Map<Property, String> getProperties() {
final Map<Property, String> props = new HashMap<>();
final String prefix = "/".equals(servletContextPath) ? "" : servletContextPath; // Adding this prefix is not required if the Iframe is served externally.
props.put(Property.IFRAME_SRC, prefix + "/my/extension-url/index.html?clientId=${clientId}&projectId=${projectId}");
return props;
}
}
|
When using a path as IFRAME_SRC (as in the example above) then the static resources for your extension will be served from the classpath (from one of the default locations used by Spring Boot). You can also embed an external web page in the iFrame by specifying its full URL (for example. "http://my-other-domain.com/my/extension").
In order to serve the resources (HTML etc.) for your extension from the classpath, you need an additional configuration. The default security configuration of mining-api-server will set the X-Frame-Options: DENY header on all resources. This will effectively prevent your extension from functioning since it is displayed in an iframe. Also, we enforce Authentication which may also cause problems for your static resources.
To fix this, add another class to your extension project:
| import static org.springframework.security.web.util.matcher.AntPathRequestMatcher.antMatcher;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.annotation.Order;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.configuration.EnableWebSecurity;
import org.springframework.security.web.SecurityFilterChain;
@Configuration
@EnableWebSecurity
public class AuthorizedExampleExtensionConfig {
@Bean
@Order(1)
public SecurityFilterChain configure(final HttpSecurity http) throws Exception {
return http
.securityMatcher(antMatcher("/my/extension-url/**"))
.headers(headers -> headers.frameOptions().sameOrigin())
.authorizeHttpRequests(authorize -> authorize.requestMatchers(antMatcher("/my/extension-url/**"))
.permitAll())
.build();
}
}
|
Now you can put the resources for your extension under "src/main/resources/static/my/extension-url" (or whatever path you configured as IFRAME_SRC).
Please see the attached Eclipse project for a working example.
Discovery Contributor
This extension enables you to provide custom contributors for Discovery. These contributors allow you to discover new types of Modules from source code, or to extend the already supported Modules with additional information. This will be helpful for clients who have different language dialects or need to support any client specific implementation.
Info
The class implementing DiscoveryContributor or DiscoveryContributorFromSource has to be public and needs to be annotated with @Component. It also needs to be underneath the innowake.mining.extensions package.
Here is an example for an extension that extends the already discovered Java classes with additional dependency information based on a customer-specific annotation:
| /**
* Example contributor extension which creates additional dependecies from Java classes (JAVA_TYPE) to Stored Procedures (SQL_STORED_PROCEDURE)
* based on an "@StoredProcedure" annotation.
*/
@Component
public class JavaMetricsContributorExtension implements DiscoveryContributor {
private static final Pattern PATTERN = Pattern.compile("@StoredProcedure\\(\"(\\w+)\"\\)");
@Override
public void contribute(final DiscoveryBuilder builder, final DiscoveryContext context) {
/* anchorTo() is used to "attach" to existing Modules which were already discovered by another contributor.
* In this case, the built-in Java support of Discovery has already discovered the Java classes (they have the type JAVA_TYPE).
* So our contributor extension can simply "anchor to" the existing Modules and extend them with additional information.
*
* To do this, we use deferAction(). Discovery processes files in parallel for optimal performance. Using deferAction(), we can schedule
* some code to be executed after processing of all files is completed. This is ideal for our use-case, because we want our custom
* contributor to run after all Java classes and Stored Procedures have been discovered. */
builder.anchorTo(new ModuleFilter().setTypes(ModuleType.JAVA_TYPE))
.deferAction("analyzeStoredProcedures"); /* must match method name of method annotated with @DeferredAction */
}
@DeferredAction
public void analyzeStoredProcedures(final Module module, final ModuleBuilder moduleBuilder, final SourceObject sourceObject) {
String content = sourceObject.getContent();
/* the SourceObject is for the entire COMPILATION_UNIT (i.e. .java file) but this deferred action runs separately on each class
* since one COMPILATION_UNIT can contain multiple classes, we make sure to only look at the part of the file that contains the current class */
final ModuleLocation classLocation = module.getLocation();
if (classLocation != null) {
content = content.substring(classLocation.getOffset(), classLocation.getLength());
}
final Matcher matcher = PATTERN.matcher(content);
if (matcher.find()) {
final String storedProcedure = matcher.toMatchResult().group(1);
moduleBuilder.declareDependency(Relationship.CALLS, new ModuleFilter().setNames(storedProcedure).setTypes(ModuleType.SQL_STORED_PROCEDURE))
.setBinding(Binding.LATE);
}
}
}
|
Metrics Contributor
This extension enables to provide custom Metrics contributors. This will be helpful for clients who have different dialects or to support any client specific implementation.
Info
The class implementing MetricsContributorExtension has to be public and needs to be annotated with @Component. It also needs to be underneath the innowake.mining.extensions package.
| @Component
public class JavaMetricsContributorExtension implements MetricsContributorExtension {
@Override
public boolean accept(final ModelArtifact entry, final Phase phase) {
return entry.getType() == ResolveTarget.JAVA_COMPILATION_UNIT;
}
@Override
public MetricsContributor init(final Parameters parameters, final SourceObjectDao sourceObjectDao, final @Nullable TimedWorker timedWorker) {
return new JavaMetricsContributor(sourceObjectDao, parameters);
}
}
class JavaMetricsContributor implements MetricsContributor {
private static final Pattern PATTERN = Pattern.compile("@StoredProcedure\\(\"(\\w+)\"\\)");
private final SourceObjectDao sourceObjectDao;
private final Parameters parameters;
public JavaMetricsContributorTest(final SourceObjectDao sourceObjectDao, final ContributorParameters parameters) {
this.sourceObjectDao = sourceObjectDao;
this.parameters = parameters;
}
@Override
public List<ModelArtifact> collectModules(final ModelArtifact entry) throws DiscoveryException {
return Collections.singletonList(entry);
}
@Override
public void calculateGenericMetrics(final ModelArtifact artifact) throws DiscoveryException {
}
@Override
public void calculateDependentMetrics(final ModelArtifact artifact) throws DiscoveryException {
final String content = getSourceObject(artifact).getContent();
final Matcher matcher = PATTERN.matcher(content);
if (matcher.find()) {
final String storedProcedure = matcher.toMatchResult().group(1);
artifact.addDependency(new ModelDependency().setLateBinding()
.setTarget(new ModelArtifact().setName(storedProcedure).setType(ResolveTarget.SQL_STORED_PROCEDURE).validate()).validate());
}
}
private SourceObject getSourceObject(final ModelArtifact artifact) throws DiscoveryException {
final String path = artifact.getPath().orElseThrow(() -> new DiscoveryException("Entry file is not present: " + artifact.getName()));
final SourceObject sourceObject = Assert.assertNotNull(sourceObjectDao).findByPath(Assert.assertNotNull(parameters).getProjectId(), path);
if (sourceObject == null) {
throw new DiscoveryException("Java object is unresolvable: " + path);
}
return sourceObject;
}
}
|
Technical Taxonomy identification
If a technical taxonomy has to be identified for a module based on some patterns that cannot be achieved through Taxonomy propagation (or) through the existing technical taxonomy logic, then you can write a custom logic that should implement TaxonomyIdentifier interface. Although, this interface is not introduced specifically as a mining extension, one can still use it like all other extension.
Info
The class implementing TaxonomyIdentifier<ModuleBasePojo> has to be public and needs to be annotated with @Component. It also needs to be underneath the innowake.mining.extensions package. At the time of writing, this can take AstNodePojo , AstModel , DependencyModel .
Let's say you need to identify all the Cobol modules that have a CICS statement in them as the "Access CICS" taxonomy, then:
| @Component
public class CicsTaxonomyIdentifier implements TaxonomyIdentifier<ModuleBasePojo> {
private final ModuleService moduleService;
@Override
public List<Tuple2<Name, TypeName>> identify(final ModuleBasePojo moduleBasePojo) {
if (moduleBasePojo.getTechnology() == Technology.COBOL) {
return moduleService.findStatements(q -> q.ofModule(moduleBasePojo.identity())).stream()
.map(StatementPojo::getType)
.filter(StatementType.EXEC_CICS::equals)
.findFirst()
.stream()
.map(type -> new Tuple2<>(Name.ACCESS_CICS, TypeName.PROGRAM_TYPE))
.collect(Collectors.toList());
}
return Collections.emptyList();
}
}
|
GenAI Context Retriever
Before a prompt is sent to a GenAI model, additional context is added to the prompt by context retrievers. It is possible to add custom context retrievers with a mining extension.
To do so you need to add a class implementing one of the ContextRetriever interfaces to the innowake.mining.server.genai.context package:
-
GenAiAnnotationContextRetriever
-
GenAiModuleContextRetriever
-
GenAiFunctionalBlockContextRetriever
-
GenAiReachabilityBlockContextRetriever
-
GenAiDataDictionaryContextRetriever
It is also possible to define multiple retrievers for the same kind of entity. Each retriever is supposed to return the context of the previously executed retriever + the newly retrieved context.
Example
If you wanted to add the custom properties of an annotation to the annotation description prompt, the context retriever could look like this:
| package innowake.mining.server.genai.context;
import innowake.mining.shared.entities.AnnotationPojoPrototype;
import org.springframework.stereotype.Service;
@Service
public class ExampleAnnotationContextRetriever implements GenAiAnnotationContextRetriever {
@Override
public AiGenerationContext retrieve(final AnnotationContext context) {
final AiGenerationContext previousContext = context.getGenerationContext();
final AnnotationPojoPrototype annotationPojoPrototype = context.getAnnotation();
if (annotationPojoPrototype.customProperties.isDefined()) {
/* Return context containing context from previous retrievers + new context */
return new AiGenerationContext(previousContext.getAdditionalPromptContext()
+ annotationPojoPrototype.customProperties.get().toString());
}
/* No custom properties are available -> we return the previous context */
return context.getGenerationContext();
}
}
|
The String value that gets passed to the AiGenerationContext constructor will get added to the prompt.
Request UUID
-
If you don't use GenAI capabilities for the context retrieval you can ignore this
-
The context object (f.e. AnnotationContext) which gets passed to retrieve has a method getRequestUUID
-
Purpose: associate the context retrieval with the description generation for token usage tracing purposes
-
If you access f.e. the Knowledge Service in your context retriever, you should add the UUID to the request as part of the requestMetadata map at key "uuid"