Skip to content
This repository has been archived by the owner on Nov 4, 2024. It is now read-only.

Commit

Permalink
Merge pull request #254 from galasa-dev/mcobbett-1889-cache-rest-cps-…
Browse files Browse the repository at this point in the history
…properties

Cached CPS can be enabled/disabled using a CPS property.
  • Loading branch information
techcobweb authored Jul 25, 2024
2 parents ad727f6 + 742185c commit cf6ce1a
Show file tree
Hide file tree
Showing 9 changed files with 794 additions and 23 deletions.
14 changes: 2 additions & 12 deletions build-locally.sh
Original file line number Diff line number Diff line change
Expand Up @@ -70,7 +70,6 @@ Environment variables used:
DEBUG - Optional. Valid values "1" (on) or "0" (off). Defaults to "0" (off).
SOURCE_MAVEN - Optional. Where maven/gradle can look for pre-built development levels of things.
Defaults to https://development.galasa.dev/main/maven-repo/framework/
LOGS_DIR - Optional. Where logs are placed. Defaults to creating a temporary directory.
EOF
}
Expand Down Expand Up @@ -140,17 +139,8 @@ else
info "SOURCE_MAVEN set to ${SOURCE_MAVEN} by caller."
fi

# Create a temporary dir.
# Note: This bash 'spell' works in OSX and Linux.
if [[ -z ${LOGS_DIR} ]]; then
export LOGS_DIR=$(mktemp -d 2>/dev/null || mktemp -d -t "galasa-logs")
info "Logs are stored in the ${LOGS_DIR} folder."
info "Over-ride this setting using the LOGS_DIR environment variable."
else
mkdir -p ${LOGS_DIR} 2>&1 > /dev/null # Don't show output. We don't care if it already existed.
info "Logs are stored in the ${LOGS_DIR} folder."
info "Over-ridden by caller using the LOGS_DIR variable."
fi
export LOGS_DIR=$BASEDIR/temp
mkdir -p $LOGS_DIR

info "Using source code at ${source_dir}"
cd ${BASEDIR}/${source_dir}
Expand Down
13 changes: 11 additions & 2 deletions galasa-extensions-parent/dev.galasa.cps.rest/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ The configuration on the ecosystem is read-only. Set and Delete operations are n

To configure Galasa to load and use this adapter:

The galasactl tool can be configured to communicate with that CPS (see the latest docs on the galasactl tool.
The galasactl tool can be configured to communicate with that CPS (see the latest docs on the galasactl tool).

To do this, assuming `https://myhost/api/bootstrap` can be used to
communicate with the remote server, add the following to your `bootstrap.properties` file,
Expand All @@ -18,4 +18,13 @@ communicate with the remote server, add the following to your `bootstrap.propert
# https://myhost/api is the location of the Galasa REST API endpoints.
framework.config.store=galasacps://myhost/api
# Tells the framework to load this extension, so it can register to react when the `galasacps` URL scheme is used.
framework.extra.bundles=dev.galasa.cps.rest
framework.extra.bundles=dev.galasa.cps.rest
```

The CPS over REST feature has a cache which can be turned on using the `framework.cps.rest.cache.is.enabled` property.
- Set it to `true` to enable caching of CPS properties on the client-side, with an agressive cache-priming which loads all
CPS properties into the cache at the start.
- Set it to `false` or don't have that property in your CPS store, and the caching will be disabled.



Original file line number Diff line number Diff line change
@@ -0,0 +1,212 @@
/*
* Copyright contributors to the Galasa project
*
* SPDX-License-Identifier: EPL-2.0
*/
package dev.galasa.cps.rest;

import java.util.*;
import java.util.Map.Entry;

import javax.validation.constraints.NotNull;
import javax.validation.constraints.Null;
import dev.galasa.extensions.common.api.LogFactory;

import dev.galasa.framework.spi.ConfigurationPropertyStoreException;
import dev.galasa.framework.spi.IConfigurationPropertyStore;

import org.apache.commons.logging.Log;

/**
* This class is a CPS implementation that delegates calls to the child CPS it gets passed.
* But it caches responses.
*
* Up-front the implementation reads the entire contents from the CPS and caches it.
*
* Set and Delete of properties are deleted, and the cache state is maintained.
*
* The cache is turned on using the 'framework.cps.rest.cache.is.enabled' property.
* - true : The cacheing is turned on.
* - false : Calls pass directly through to the child CPS implementation.
* Default value: false.
*/
public class CacheCPS implements IConfigurationPropertyStore {

// The key map key is the fully qualified property name
// The value is the value of the property.
private Map<String,String> propertyCache ;

// We use this flag so that we don't try to prime the cache twice.
private boolean isCachePrimed = false ;

private IConfigurationPropertyStore childCPS ;

private Log log ;

private boolean isCacheEnabled = false;

/**
* The CPS property which this extension draws from to control whether the cache is enabled or not.
*/
public static final String FEATURE_FLAG_CPS_PROP_CACHED_CPS_ENABLED = "framework.cps.rest.cache.is.enabled";


public CacheCPS( IConfigurationPropertyStore childCPS , LogFactory logFactory) throws ConfigurationPropertyStoreException {

this.log = logFactory.getLog(this.getClass());
this.propertyCache = new HashMap<String,String>();
this.childCPS = childCPS ;
}


private synchronized void primeCaches(IConfigurationPropertyStore childCPS ) throws ConfigurationPropertyStoreException {

// Don't re-prime the caches if they are not primed already.
if (this.isCachePrimed==false) {

// Only prime the cache once
this.isCachePrimed = true ;

String isEnabledPropValue = childCPS.getProperty(FEATURE_FLAG_CPS_PROP_CACHED_CPS_ENABLED);
if ((isEnabledPropValue==null)||(isEnabledPropValue.isBlank())) {
log.info("CPS Cache property "+FEATURE_FLAG_CPS_PROP_CACHED_CPS_ENABLED+" not found in child CPS.");
this.isCacheEnabled = false ;
} else {
log.info("CPS Cache property "+FEATURE_FLAG_CPS_PROP_CACHED_CPS_ENABLED+" has a value of "+isEnabledPropValue);
this.isCacheEnabled = Boolean.parseBoolean(isEnabledPropValue);
}

if (!this.isCacheEnabled) {
log.info("CPS Cache is not enabled...");
} else {

log.info("CPS Cache is enabled, and being primed...");
List<String> namespaces = childCPS.getNamespaces();

for( String namespace : namespaces ) {

if (!namespace.equals("secure")) {
Map<String, String> propertiesFromNamespace = childCPS.getPropertiesFromNamespace(namespace);

for( Entry<String,String> propertyInNamespace : propertiesFromNamespace.entrySet()){
String propertyValue = propertyInNamespace.getValue();
String longPropertyName = propertyInNamespace.getKey();

propertyCache.put(longPropertyName,propertyValue);
}
}
}
}
log.info("CPS Cache primed with "+Integer.toString(propertyCache.size())+" properties.");

}
}

@Override
public List<String> getNamespaces() throws ConfigurationPropertyStoreException {
primeCaches(childCPS);
List<String> results ;
if (isCacheEnabled) {
results = new ArrayList<String>();

// Gather a set of the namespaces, so there are no duplicates.
Set<String> namespacesSet = new HashSet<>();
for (Map.Entry<String,String> entry : propertyCache.entrySet()){
String propertyName = entry.getKey();
String[] parts = propertyName.split("\\.");
if (parts.length > 1) {
String namespace = parts[0];
namespacesSet.add(namespace);
}
}

results.addAll(namespacesSet);
} else {
results = this.childCPS.getNamespaces();
}
return results;
}

@Override
public @Null String getProperty(@NotNull String fullyQualifiedPropertyName) throws ConfigurationPropertyStoreException {
primeCaches(childCPS);
String result ;
if (isCacheEnabled) {
result = propertyCache.get(fullyQualifiedPropertyName);
} else {
result = this.childCPS.getProperty(fullyQualifiedPropertyName);
}
return result ;
}

@Override
public void setProperty(@NotNull String key, @NotNull String value) throws ConfigurationPropertyStoreException {
primeCaches(childCPS);

// Delegate the set of the property to the child CPS
childCPS.setProperty(key, value);

if (isCacheEnabled) {
// The child changed the property value ok, so we should change the cache version also.
propertyCache.put(key,value);
}
}

@Override
public @NotNull Map<String, String> getPrefixedProperties(@NotNull String prefix)
throws ConfigurationPropertyStoreException {

primeCaches(childCPS);

Map<String, String> results;
if (isCacheEnabled) {
results = new HashMap<String, String>();
for( Entry<String,String> property : this.propertyCache.entrySet() ){
String propName = property.getKey();
if( propName.startsWith(prefix)) {
String propValue = property.getValue();
results.put(propName, propValue);
}
}
} else {
results = this.childCPS.getPrefixedProperties(prefix);
}
return results;
}

@Override
public void deleteProperty(@NotNull String key) throws ConfigurationPropertyStoreException {

primeCaches(childCPS);

// Delegate the delete to the underlying CPS.
this.childCPS.deleteProperty(key);

if (this.isCacheEnabled) {
// Keep our cache in step.
propertyCache.remove(key);
}
}

@Override
public Map<String, String> getPropertiesFromNamespace(String namespace) throws ConfigurationPropertyStoreException {

primeCaches(childCPS);

Map<String, String> results ;
if (this.isCacheEnabled) {
results = getPrefixedProperties(namespace);
} else {
results = childCPS.getPropertiesFromNamespace(namespace);
}
return results;
}

@Override
public void shutdown() throws ConfigurationPropertyStoreException {

// Delegate this stimulus to the child, to give that a chance of closing resources.
childCPS.shutdown();
}

}
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,7 @@ private boolean isPropertyRedacted(String fullyQualifiedPropertyName) {
// So we use the /cps/{namespace}/properties?prefix=xxxx so that if the endpoint isn't available, we get 404,
// and if the property doesn't exist, then we get null in the map.
// Although it's not as efficient on the server-side, performance isn't everything in this case, as local runs
// can be slower/less performant than the ecosyste runs.
// can be slower/less performant than the ecosystem runs.
Map<String,String> properties = getPrefixedProperties(fullyQualifiedPropertyName);
propertyValueResult = properties.get(fullyQualifiedPropertyName);
}
Expand Down Expand Up @@ -441,7 +441,7 @@ private Map<String,String> propertiesToMap(GalasaProperty[] properties) throws C
GalasaPropertyData data = property.getData();
String value = data.getValue();

log.info("galasacps: over rest (with prefix): "+fullyQualifiedPropName+" : "+value);
// log.info("galasacps: over rest (with prefix): "+fullyQualifiedPropName+" : "+value);

if (!isPropertyRedacted(fullyQualifiedPropName)) {
results.put(fullyQualifiedPropName, value);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
import dev.galasa.extensions.common.impl.HttpClientFactoryImpl;
import dev.galasa.extensions.common.impl.LogFactoryImpl;
import dev.galasa.framework.spi.ConfigurationPropertyStoreException;
import dev.galasa.framework.spi.IConfigurationPropertyStore;
import dev.galasa.framework.spi.IConfigurationPropertyStoreRegistration;
import dev.galasa.framework.spi.IFrameworkInitialisation;

Expand Down Expand Up @@ -74,13 +75,17 @@ public void initialise(@NotNull IFrameworkInitialisation frameworkInitialisation
throw new ConfigurationPropertyStoreException(msg,ex);
}

IConfigurationPropertyStore baseCPS = new RestCPS(
ecosystemRestApi,
httpClientFacotory,
jwtProvider,
logFactory
);

IConfigurationPropertyStore cacheCPS = new CacheCPS(baseCPS, logFactory);

frameworkInitialisation.registerConfigurationPropertyStore(
new RestCPS(
ecosystemRestApi,
httpClientFacotory,
jwtProvider,
logFactory
)
cacheCPS
);
}
}
Expand Down
Loading

0 comments on commit cf6ce1a

Please sign in to comment.