Step 5. Real-World Plugin Project

[choonho]

Code Analysis of plugin-http-file-cost-datasource

Understanding API call flow

Service	Resource	Verb	Description
cost_analysis	DataSource	register	Register Cost Analysis Plugin
cost_analysis	DataSource	enable	Enable Plugin
cost_analysis	DataSource	sync	Start to sync data of plugin

Clone Cost Analysis Micro Service

To analysis cost-analysis service, clone source code with specific branch

git clone https://github.com/cloudforet-io/cost-analysis.git
cd cost-analysis
git branch -r
git switch -c origin/release-1.12
git checkout release-1.12
tree src/  -L 3

# source code directory
choonhoson@Choonhoui-MacBookAir cost-analysis % tree src/ -L 3
src/
├── setup.py
└── spaceone
    ├── __init__.py
    └── cost_analysis
        ├── __init__.py
        ├── conf
        ├── connector
        ├── error
        ├── info
        ├── interface
        ├── lib
        ├── manager
        ├── model
        └── service

Current master branch is cloudforet-2.x version, but our installation version is release-1.12

Understand Backend Framework

Cloudforet core team developed Python-based gRPC framework like django or Java Spring framework.

Layer	Description	Base Class	Implementation Directory
Interface	Entry point of Service request	core/api.py	project/interface/interface type/
Handler	Pre, Post processing before Service call
Service	Business logic of service	core/service.py	project/service/
Cache	Caching for manager function(optional)	core/cache/
Manager	Unit operation for each service function	core/manager.py	project/manager/
Connector	Interface for Data Source(ex. DB, Other micro services)

gRPC client is either Web browser(console-api micro service) or spacectl CLI tool.

Understanding API call by example

Let's debug API call flow based on register API.

The command spacectl exec register cost_analysis.DataSource -f will request

Interface layer

Each Resource has its own source file like data_source.py for DataSource resource.

File path: src/spaceone/cost_analysis/interface/grpc/v1/data_source.py

choonhoson@Choonhoui-MacBookAir src % tree spaceone/cost_analysis/interface/grpc/v1
spaceone/cost_analysis/interface/grpc/v1
├── __init__.py
├── budget.py
├── budget_usage.py
├── cost.py
├── cost_query_set.py
├── data_source.py
├── data_source_rule.py
├── job.py
└── job_task.py

When client requests register API call, the entry point is def register( ) function.
The received gRPC message structure from the client is automatically changed to python dictionary data structure by framework.

params contains the real data from client, metadata contains extra data such as api token, transaction_id and client information like HTTP header data.

 14     def register(self, request, context):
 15         params, metadata = self.parse_request(request, context)
 16
 17         with self.locator.get_service('DataSourceService', metadata) as data_source_service:
 18             return self.locator.get_info('DataSourceInfo', data_source_service.register(params))

The main functionality of interface layer is receiving request, then transfer to proper service layer. And respond the result to client.

We provide factory pattern for triggering service or manager by self.locator. If you want to trigger the DataSourceService, just call like self.locator.get_service(SERVICE_NAME, metadata)

# Create Service Instance
svc = self.locator.get_service('DataSourceService', metadata)

# Call method
svc.register(params)

Service layer

Service layer is a main function of API body. So service layer contains a business logic of API function.
Framework provides automatic pre-condition and post-condition check by handlers. The technology is python decorator, a.k.a. @

There are two different decorators like

• class decorator
• function decorator

when a service is triggered, class decorator is executed automatically.

For example, when the DataSource service is triggered, following handlers are automatically executed,

1. event_handler: send the event to audit system
2. mutation_handler: update the parameters or metadata if needed, such as adding extra parameter like domain_id
3. authorization_handler: verify the request has permission to execute or not
4. authentication_handler: verify the call is valid or not, such as checking the api token

File path: src/spaceone/cost_analysis/service/data_source_service.py

 18 @authentication_handler
 19 @authorization_handler
 20 @mutation_handler
 21 @event_handler
 22 class DataSourceService(BaseService):
 23
 24     def __init__(self, *args, **kwargs):
 25         super().__init__(*args, **kwargs)

After passing all these class handlers, there is additional pre-condition check, before entering the method

check_required checks the parameter must contains these fields. In this example, register function required at least name, data_source_type, and domain_id.

transaction appends more metadata, such as execution scope. scope is a barrier of data range. There is DOMAIN or PROJECT.

 32     @transaction(append_meta={'authorization.scope': 'DOMAIN'})
 33     @check_required(['name', 'data_source_type', 'domain_id'])
 34     def register(self, params):
 35         """Register data source
 36
 37         Args:
 38             params (dict): {
 39                 'name': 'str',
 40                 'data_source_type': 'str',
 41                 'secret_type': 'str',
 42                 'secret_filter': 'dict',

For example, the business logic of register DataSource in cost_analysis are

1. check the plugin exist or not
2. If exist init plugin
3. save result to database

Each sub logic is implemented at Manager layer. If you want to check plugin, call repository service via RepositoryManager.

Calling Manager layer is also implemented by factory pattern by self.locator

For checking plugin exist or not, call RepositoryManager, then check plugin_id exist or not.

 66             self._validate_plugin_info(plugin_info, secret_type)
 67             self._check_plugin(plugin_info['plugin_id'], domain_id)

442     def _check_plugin(self, plugin_id, domain_id):
443         repo_mgr: RepositoryManager = self.locator.get_manager('RepositoryManager')
444         repo_mgr.get_plugin(plugin_id, domain_id)

Manager layer

Manager layer is an implementation layer for unit function. There are three main functions for manager layer.

• DB manipulation
• interact with outer systems
• Interact with other micro services

DB manipulation is create, update, delete or list of Database entity. This is abstracted by Model class similar with sqlalchemy. Currently we support only mongodb model.

For example, registering new data source in a database is calling the create function of model.

File path: src/spaceone/cost_analysis/manager/data_source_manager.py

 15     def register_data_source(self, params):
 16         def _rollback(data_source_vo):
 17             _LOGGER.info(f'[register_data_source._rollback] '
 18                          f'Delete data source : {data_source_vo.name} '
 19                          f'({data_source_vo.data_source_id})')
 20             data_source_vo.delete()
 21
 22         data_source_vo: DataSource = self.data_source_model.create(params)
 23         self.transaction.add_rollback(_rollback, data_source_vo)
 24
 25         return data_source_vo

Interact with other micro services is one of key technologies for MSA(Micro Service Architecture). For interact with other micro services, it needs

• Service discovery
• client SDK

In the Cloudforet framework, we provide SpaceConnector coding pattern for easy communication between two micro services.

For example, to check plugin-http-cost-datasource plugin exist or not, cost-analysis have to request to repository service.

File path: src/spaceone/cost_analysis/manager/repository_manager.py

In the line number 14, service discovery is easily by creating SpaceConnector with service name.

  9
 10 class RepositoryManager(BaseManager):
 11
 12     def __init__(self, *args, **kwargs):
 13         super().__init__(*args, **kwargs)
 14         self.repo_connector: SpaceConnector = self.locator.get_connector('SpaceConnector', service='repository')

To call API of other micro service, the method pattern is connector.dispatch(RESOURCE.VERB, parameters)

Line number 17 calls get method of repository. Line number 20 calls get_versions method of repository.

 16     def get_plugin(self, plugin_id, domain_id):
 17         return self.repo_connector.dispatch('Plugin.get', {'plugin_id': plugin_id, 'domain_id': domain_id})
 18
 19     def check_plugin_version(self, plugin_id, version, domain_id):
 20         response = self.repo_connector.dispatch('Plugin.get_versions', {'plugin_id': plugin_id, 'domain_id': domain_id})
 21         versions = response.get('results', [])

API document of version 1.12 is at https://cloudforet-io.github.io/api-doc-1-12/repository/v1/Plugin/

Assignment

• Update DataSource at Step 4.
  • change base_url with your own csv file (using spacectl CLI)
  • The API is update_plugin at https://cloudforet-io.github.io/api-doc-1-12/cost_analysis/v1/DataSource/#update_plugin
• Report for sync API call flow
  • based on interface, service, manager layer
  • How cost-analysis service and plugin-http-file-cost-datasource interact

[py-dev-nandini-12]

Hi @choonho ,
For the above assignment , are we suppose to modify the update_plugin function in datasource.py file in the interface layer with these payloads?{
"data_source_id": "ds-085d1e872789",
"version": "1.0.4",
"options": {},
"upgrade_mode": "AUTO"
} using command "spacectl exec update_plugin cost_analysis.DataSource -f --data_source_id=ds-085d1e872789 --version=1.0.4 --upgrade_mode=AUTO" . Please let me know.

[Cvr421]

Hi @choonho , For the above assignment , are we suppose to modify the update_plugin function in datasource.py file in the interface layer with these payloads?{ "data_source_id": "ds-085d1e872789", "version": "1.0.4", "options": {}, "upgrade_mode": "AUTO" } using command "spacectl exec update_plugin cost_analysis.DataSource -f --data_source_id=ds-085d1e872789 --version=1.0.4 --upgrade_mode=AUTO" . Please let me know.
@py-dev-nandini-12 i think when you change the base url with your csv file and then you get the new Data_source_id and then that id you have to replace here {data_source_id": "ds-085d1e872789"}

[arup1221]

@Cvr421 do you think to modify the source code ?

like this function

def update_plugin(self, request, context):
        params, metadata = self.parse_request(request, context)
        with self.locator.get_service('DataSourceService', metadata) as data_source_service:
            return self.locator.get_info('DataSourceInfo', data_source_service.update_plugin(params)) 

[choonho]

Hi all,
It is not a change of code. You can update information by API call, which was already configured at Step 4.

[choonho]

For example, the data only exists at Sep. 2023, fill data of 2023-10, 2023-11, and 2024-12 with your own data set.
In that case, you have to own your own csv file instead of sample csv file.

[Cvr421]

Do this @arup1221

[arup1221]

yes

[Cvr421]

Hi all, It is not a change of code. You can update information by API call, which was already configured at Step 4.
That's correct

[choonho]

In Step 2, we already studied how to install web service with NGINX.
Run your own NGINX pod with your own csv file, then change plugin information with new csv file url.

[Cvr421]

Hello @choonho when i am running the minikube cluster after sometime my whole laptop is freezes and hang and nothing is working untill i restart my computer .I have tried multiple solution like increasing minikube memory size and and cpus and i have solove increase the docker resource limit but nothing is working . Can you please suggest some solution
Few day back i have run minikube cluster and it worked perfectly fine . but now it just freezes whole system
i am using ubuntu OS
every time stuck at this step

[SpringWiz11]

@Cvr421
I too faced that problem, but unfortunately, I think its the issue with the hardware / Kubernetes. In my case it was the problem wrt the memory constraints. You can stop the other concurrent processes and then try it again.

[arup1221]

if you using docker desktop increase its memory.

[Cvr421]

Thanks everyone i'll look into this.

[Cvr421]

if you using docker desktop increase its memory.

@arup1221 I have increased the docker memory size but still it's happening.

[Cvr421]

@Cvr421 ContainerCreating is not an application problem, it is a problem of kubernetes. In this case, it is needed to check the events of kubernetes. See the events of 1) kubernetes itself, 2) pod event. and 3) OS system step by step.

Thanks @choonho i'll try this.

[arup1221]

@choonho I have submitted the task in pdf file. Could you please check!

[Cvr421]

@choonho I have uploaded the PDF of task-5. Please Review .

[choonho]

Best Submission

Arup Gope

[arup1221]

Thank you @choonho

[Cvr421]

Best Submission

Arup Gope

Nice work @arup1221

[arup1221]

Thank you

[py-dev-nandini-12]

@choonho uploaded the last assignment .

Thank you
Nandini

[choonho]

@py-dev-nandini-12 Hi, you did very well. Congratulate finishing all tasks.

[py-dev-nandini-12]

Thank you so much for being such a great mentor.

[Suraj-kumar00]

I have completed all the tasks thanks to @choonho sir for 1:1 mentorship.