Architecture¶

gCube CMS suite : General Architecture

The conceptual architecture behind gCube CMS Suite is schematized in the figure above.

Here we describe the rationale behind the architecture :

Clients perform both CRUD and Query operations against the service both via using the client library or by directly interacting with the service REST interface.
They interact with the service producing the custom metadata document (stored in a Document Store DB) and registering related Filesets (stored in gCube StorageHub).
Clients sends steps (pre configured in the collection UCD) requests to the service, which in turn delegate Lifecycle operations to configured implementations.
Clients may also directly consume materialized projects in ad hoc engines (e.g. SDI or DBMS). For this purpose, the service exposes information needed by the client in order to properly consume the project collection.

In the figure below we further illustrate the various components involved in the typical execution of Projects’ lifecycle

Service¶

gCube CMS Service is a gCube REST-like Application developped on top of gCube SmartGears.

It implements the core business logic of gCube CMS Suite, exposing methods for the management of Projects, Use Case Descriptors and available server plugins.

It is built on top of Jersey Framework, it uses a MongoDB Document Store for JSON archiving and Querying. It exploits the infrastructure StorageHub for archiving FileSets and the Information System in order to discover available resources.

Interface¶

Note

The service interface is available for testing here.

Projects¶

BASE PATH : ../projects/{ucid}/

Projects interface offers the following methods over a collection (see {ucid} in path)

CRUD operations (create, deleteById, getById, updateById)
FileSet Management
Query operations (powered by Mongo DB capabiliteis) allowing for filtering, pagination, projection.
STEP invocation
Project configuration, reporting the generated resources for the collection and how to access them.

FileSet Management¶

Clients can register Filesets with JSON requests with the following fields :

fieldName : target field name
parentPath : the target field’s parent
fieldDefinitionPath : JSONPath of the target field definition in UCD schema
streams : list of JSON objects representing the fileset’s payload and filenames. Payload can be specified either as gCube Storage Hub volatile area ID or as URL
clashOptions : one of REPLACE_EXISTING, MERGE_EXISTING, APPEND
toSetAccess [optional]: to set access policy for the fileset. Default is taken from the UCD.
attributes [optional] : additional attributes to set to the FileSet.

Following example is taken from Concessioni use case submission :

Querying¶

The service supports querying against Projects collections. Clients can submit query requests as a JSON object with the following fields :

ordering [optional] : bean with both direction and fields list
paging [optional]: bean with both offset and limit
projection [optional] (NB see mongoDB aggregation pipelines): the project fields to be returned
filtering [optional] (NB see mongoDB aggregation pipelines): the condition to evaluate

Following are examples of queries towards collections (in some of the following example we consider structure and lifecycle from “Concessioni” UCD) :

Query lifecycleInformation ordered by submission¶

::

{: “ordering” :{“direction” : “DESCENDING”, “fields” :[“_info._creationInfo._instant”]}, “projection” :{“_lifecycleInformation._phase”:1,”_theDocument.nome”:1,”_lifecycleInformation._lastOperationStatus”:1,”_lifecycleInformation._errorMessages”:1,”_info._creationInfo._instant”:1}

}

Query filesets ordered by submission¶

::

{: “ordering” :{“direction” : “DESCENDING”, “fields” :[“_info._creationInfo._instant”]}, “projection” :{“_theDocument.abstractRelazione.fileset._payloads” : 1,”_theDocument.immaginiRappresentative.fileset._payloads” : 1, “_theDocument.pianteFineScavo.fileset._payloads” : 1,”_theDocument.posizionamentoScavo.fileset._payloads” : 1}

}

Query projects with “follow” relationship ordered by name¶

::

{: “ordering” :{“direction” : “ASCENDING”, “fields” :[“_theDocument.nome”]}, “filter” : {“_relationships._relationshipName” : {“$eq”:”follows”}}

}

STEP Invocation¶

Lifecycle STEPs can be invoked by submitting JSON requests with the following fields :

stepID : the ID of the STEP to be executed.
options : the parameters needed for the STEP exewcution.

Note

You can check the supported STEPs along with its parameters, just by interacting with the Plugins interface described below.

Project configuration¶

Some resources are generated by the service in the normal execution of Projects lifecycle (e.g. indexes). The service exposes the following informations in order to allow clients (e.g. GUIs) to properly consume these resources.

indexes : A set of generated indexes (e.g. SDI centroid layers)
archives : A set of generated archive (e.g. Document archive) along with related information.
errorMessages : A set of error messages reporting errors in generating the current response
warningMessages : A set of warning messages reporting unexpected behaviour in generating the current response
status : the global status.

Note

In the current implementation all collection uses the same DOCUMENT-STORE-COLLECTION (based on MongoDB) and W-STORAGE (based on gCube StorageHub).

Note

DOCUMENT-STORE-COLLECTION archive also reports aggregated Projects count by PHASE and STATUS.

Following is an example of the returned information

{
    "profile_id": "profiledConcessioni",
    "context": "/pred4s/preprod/preVRE",
    "last_updated_time": "2022-11-17T14:58:59.026",
    "indexes": [
        {"_type": "GIS-CENTROIDS",
            "layer": {"_type": "gcube-sdi-layer",
                "_platformInfo": [{
                        "_type": "Geoserver",
                        "workspace": "profiledconcessioni_prevre",
                        "storeName": "profiledconcessioni_prevre_centroids",
                        "_host": "geoserver1-t.pre.d4science.org",
                        "layerName": "profiledconcessioni_prevre_centroids"
                    }],
                "_bbox": {"_maxX": 180.0,"_minX": -180.0,"_maxY": 90.0,"_minY": -90.0},
                "_ogcLinks": {
                    "wms": "https://geoserver1-t.pre.d4science.org/geoserver/profiledconcessioni_prevre/wms?service=WMS&version=1.1.0&request=GetMap&layers=profiledconcessioni_prevre:profiledconcessioni_prevre_centroids&styles=&bbox=-180.000000,-90.000000,180.000000,90.000000&srs=EPSG:4326&format=application/openlayers&width=400&height=400"
                }
            },
            "indexName": "profiledconcessioni_prevre_centroids",
            "records": 1,
            "crossReferencedLayers": {},
            "flag": "public"
        }],
    "archives": [
        {
            "_type": "DOCUMENT-STORE-COLLECTION",
            "count": 225,
            "collection_name": "profiledConcessioni",
            "countByPhase": [
                {"_id": {"phase": "DRAFT","status": "OK"},"count": 19},
                {"_id": {"phase": "PUBLISHED","status": "OK"},"count": 3},
                {"_id": {"phase": "Pending Approval","status": "ERROR"},"count": 11},
                ...
            ]
        },
        {
            "_type": "W-STORAGE",
            "folder_id": "ded73475-f9b1-46c8-94f8-5b35c60667ce"
        }
    ],
    "errorMessages": null,
    "warningMessages": null,
    "status": "OK"
}

UCD Management¶

Current approach uses the class org.gcube.application.geoportal.service.engine.providers.ucd.SingleISResourceUCDProvider as a UCD Provider.

This class asks the gCube Information System a Generic Resource with the following coordinates :

SecondaryType : CMS
Name : UCDs

It expects the body to be like the following example

<UCDs>
   <record label="Human readable name" ucdUrl="some url" defaultForRegex="[opt] regex"/>
   <record label="Another readable name" ucdUrl="some other url" defaultForRegex="[opt] regex"/>
</UCDs>

Project Management¶

Projects metadata are stored as JSON document in a Document Store DB (powered by MongoDB) called Project archive. The core logic of Project management lies in the class org.gcube.application.geoportal.service.engine.mongo.ProfiledMongoManager, which is the only component in charge of accessing the Project archive, in order to grant robustness via isolation.

Note

In order to execute both lifecyle EVENTs and STEPs, the service locates the configured LifecycleManager handler and passes to them requests with all the required information (check Plugin Framework below).

ProfiledMongoManager class implements the following lock mechanism for all writing executions (here represented for the STEP execution scenario for completeness).

Plugin Framework¶

In order to isolate logic, Plugins are discovered via reflection. Plugins are thus expected to implement the proper interface, choosing between the ones defined in the framework and listed below.

The framework provides also : * Definition of all involved Requests, Response and Faults beans * Basic plugin implementations for common use cases, that can be extended for custom solutions (e.g. AbstractLifeCycleManager, SingleLifecycleManager, Default3PhaseManager) * Common utilities for plugin discovery (e.g. for Lifecycle Manager to invoke Materializers and Indexers) * Engine Provider in order to allow plugins to access implementations provided by the service * Common utilities (e.g. serialization)

Engine Provider¶

org.gcube.application.cms.implementations.ImplementationProvider class represents a common point of implementations reference for both the service and plugins.

It exposes registered implementations by asking for an Engine (generic interface *org.gcube.application.cms.caches.Engine*<T>) responsible to provide a specific object class.

The service (or a startup environment, e.g. for testing purposes) is expected to register the implemented Engines at startup.

In the following list there are all Engine<T> implementations provided by the service and available to Plugins :

MongoClientProvider provided class T : org.gcube.application.geoportal.service.model.internal.db.Mongo
ProjectAccessProvider provided class T : org.gcube.application.cms.implementations.ProjectAccess
StorageClientProvider provided class T : org.gcube.application.geoportal.common.utils.StorageUtils
SingleISResourceUCDProviderp rovided class T : org.gcube.application.geoportal.service.engine.providers.ucd.ProfileMap
PluginManager provided class T : org.gcube.application.geoportal.service.engine.providers.PluginManager.PluginMap
UCDManager provided class T : org.gcube.application.geoportal.service.engine.mongo.UCDManagerI
ConfigurationCacheprovided class T : org.gcube.application.geoportal.service.engine.providers.ConfigurationCache.ConfigurationMap

Plugin Interfaces¶

The following is a list of defined plugin interfaces, which implementations are expected to implement in order to make themself available.

Following diagram shows the interfaces defined in the framework (package org.gcube.application.cms.plugins)

Note

Plugin interface as root of the above hierarchy grants common behaviour to all plugins, i.e. the exposition of PluginDescriptor object, which information are returned by the service Plugin REST interface.

Plugin Development support¶

In order to facilitate the development of plugins, the following solutions may be of use to the developer :

extension of an already provided LifecycleManager in order to override default behaviour (e.g. as per Concessioni UCD, see suite module concessioni-lifecycle)
extension of org.gcube.application.cms.plugins.AbstractPlugin class in order to leverage on common behaiovur (e.g. configuration management, error handling)
extension of org.gcube.application.cms.caches.AbstractScopedMap or org.gcube.application.cms.caches.TimedMap for easy implementation of caches (respectively by context and global)
leverage on module cms-test-commons for exploitation of context management, basic plugin testing (both static and in a gCube context)

Architecture¶

Service¶

Interface¶

Projects¶

FileSet Management¶

Querying¶

Query lifecycleInformation ordered by submission¶

Query filesets ordered by submission¶

Query projects with “follow” relationship ordered by name¶

STEP Invocation¶

Project configuration¶

UCD Management¶

Project Management¶

Plugin Framework¶

Engine Provider¶

Plugin Interfaces¶

Plugin Development support¶

Client Library¶