Skip to content

Commit d425829

Browse files
tanwarshtanwarsh
authored
Updated Keras and TensorFlow Task Runner and related workspaces. (#1174)
* keras and tf updated Signed-off-by: yes <shailesh.tanwar@intel.com> * skipped rebuild model for round 0 Signed-off-by: yes <shailesh.tanwar@intel.com> * formatting issues fixed Signed-off-by: yes <shailesh.tanwar@intel.com> * added keras in workflow Signed-off-by: yes <shailesh.tanwar@intel.com> * pipeline tensorflowversion check Signed-off-by: yes <shailesh.tanwar@intel.com> * revert changes Signed-off-by: yes <shailesh.tanwar@intel.com> * revert changes Signed-off-by: yes <shailesh.tanwar@intel.com> * removed extra line Signed-off-by: yes <shailesh.tanwar@intel.com> * workspace changes Signed-off-by: yes <shailesh.tanwar@intel.com> * removed python 3.8 for taskrunner workflow Signed-off-by: yes <shailesh.tanwar@intel.com> * updated python version to 3.9 Signed-off-by: yes <shailesh.tanwar@intel.com> * updated python version to 3.9 Signed-off-by: yes <shailesh.tanwar@intel.com> * saved model name changes in tests Signed-off-by: yes <shailesh.tanwar@intel.com> * code change to save model Signed-off-by: yes <shailesh.tanwar@intel.com> * keras_nlp workspace changes Signed-off-by: yes <shailesh.tanwar@intel.com> * keras and tf updated Signed-off-by: yes <shailesh.tanwar@intel.com> * skipped rebuild model for round 0 Signed-off-by: yes <shailesh.tanwar@intel.com> * formatting issues fixed Signed-off-by: yes <shailesh.tanwar@intel.com> * added keras in workflow Signed-off-by: yes <shailesh.tanwar@intel.com> * pipeline tensorflowversion check Signed-off-by: yes <shailesh.tanwar@intel.com> * revert changes Signed-off-by: yes <shailesh.tanwar@intel.com> * revert changes Signed-off-by: yes <shailesh.tanwar@intel.com> * removed extra line Signed-off-by: yes <shailesh.tanwar@intel.com> * workspace changes Signed-off-by: yes <shailesh.tanwar@intel.com> * removed python 3.8 for taskrunner workflow Signed-off-by: yes <shailesh.tanwar@intel.com> * updated python version to 3.9 Signed-off-by: yes <shailesh.tanwar@intel.com> * updated python version to 3.9 Signed-off-by: yes <shailesh.tanwar@intel.com> * saved model name changes in tests Signed-off-by: yes <shailesh.tanwar@intel.com> * code change to save model Signed-off-by: yes <shailesh.tanwar@intel.com> * keras_nlp workspace changes Signed-off-by: yes <shailesh.tanwar@intel.com> * removed duplicate code Signed-off-by: yes <shailesh.tanwar@intel.com> * removed oython 3.8 from taskrunnner e2e workflow Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes as per comments Signed-off-by: yes <shailesh.tanwar@intel.com> * fix for formatting issue Signed-off-by: yes <shailesh.tanwar@intel.com> * keras cnn with compression code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * remove version changes Signed-off-by: yes <shailesh.tanwar@intel.com> * revert version changes Signed-off-by: yes <shailesh.tanwar@intel.com> * keep keras runner for tensorflow workspaces Signed-off-by: yes <shailesh.tanwar@intel.com> * formatting issue Signed-off-by: yes <shailesh.tanwar@intel.com> * removed keras as ke Signed-off-by: yes <shailesh.tanwar@intel.com> * comment changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> --------- Signed-off-by: yes <shailesh.tanwar@intel.com>
1 parent ac9ff39 commit d425829

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

76 files changed

+373
-3003
lines changed

‎docs/developer_ref/troubleshooting.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@
1010
The following is a list of commonly reported issues in Open Federated Learning (|productName|). If you don't see your issue reported here, please submit a `Github issue
1111
<https://github.com/intel/openfl/issues>`_ or contact us directly on `Slack <https://join.slack.com/t/openfl/shared_invite/zt-ovzbohvn-T5fApk05~YS_iZhjJ5yaTw>`_.
1212

13-
1. I see the error :code:`Cannot import name TensorFlowDataLoader from openfl.federated`
13+
1. I see the error :code:`Cannot import name KerasDataLoader from openfl.federated`
1414

1515
|productName| currently uses conditional imports to attempt to be framework agnostic. If your task runner is derived from `KerasTaskRunner` or `TensorflowTaskRunner`, this error could come up if TensorFlow\*\ was not installed in your collaborator's virtual environment. If running on multi-node experiment, we recommend using the :code:`fx workspace export` and :code:`fx workspace import` commands, as this will ensure consistent modules between aggregator and collaborators.
1616

‎openfl-workspace/keras_cnn_with_compression/.workspace renamed to ‎openfl-workspace/keras_2dunet/.workspace

File renamed without changes.

‎openfl-workspace/tf_2dunet/README.md renamed to ‎openfl-workspace/keras_2dunet/README.md

File renamed without changes.

‎openfl-workspace/tf_2dunet/plan/cols.yaml renamed to ‎openfl-workspace/keras_2dunet/plan/cols.yaml

File renamed without changes.

‎openfl-workspace/tf_cnn_histology/plan/data.yaml renamed to ‎openfl-workspace/keras_2dunet/plan/data.yaml

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
# Copyright (C) 2020-2021 Intel Corporation
1+
# Copyright (C) 2020-2024 Intel Corporation
22
# Licensed subject to the terms of the separately executed evaluation license agreement between Intel Corporation and you.
33

44
# all keys under 'collaborators' corresponds to a specific colaborator name the corresponding dictionary has data_name, data_path pairs.
@@ -8,4 +8,4 @@
88
# collaborator_name,data_directory_path
99

1010
one,1
11-
two,2
11+
two,2

‎openfl-workspace/keras_cnn_with_compression/plan/defaults renamed to ‎openfl-workspace/keras_2dunet/plan/defaults

File renamed without changes.

‎openfl-workspace/tf_2dunet/plan/plan.yaml renamed to ‎openfl-workspace/keras_2dunet/plan/plan.yaml

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@ aggregator :
55
defaults : plan/defaults/aggregator.yaml
66
template : openfl.component.Aggregator
77
settings :
8-
init_state_path : save/tf_2dunet_brats_init.pbuf
9-
last_state_path : save/tf_2dunet_brats_latest.pbuf
10-
best_state_path : save/tf_2dunet_brats_best.pbuf
8+
init_state_path : save/init.pbuf
9+
last_state_path : save/latest.pbuf
10+
best_state_path : save/best.pbuf
1111
rounds_to_train : 10
1212
db_store_rounds : 2
1313

@@ -20,7 +20,7 @@ collaborator :
2020

2121
data_loader :
2222
defaults : plan/defaults/data_loader.yaml
23-
template : src.tfbrats_inmemory.TensorFlowBratsInMemory
23+
template : src.dataloader.KerasBratsInMemory
2424
settings :
2525
batch_size: 64
2626
percent_train: 0.8
@@ -29,7 +29,7 @@ data_loader :
2929

3030
task_runner :
3131
defaults : plan/defaults/task_runner.yaml
32-
template : src.tf_2dunet.TensorFlow2DUNet
32+
template : src.taskrunner.Keras2DUNet
3333

3434
network :
3535
defaults : plan/defaults/network.yaml
@@ -38,7 +38,30 @@ assigner :
3838
defaults : plan/defaults/assigner.yaml
3939

4040
tasks :
41-
defaults : plan/defaults/tasks_tensorflow.yaml
41+
aggregated_model_validation:
42+
function : validate_task
43+
kwargs :
44+
batch_size : 32
45+
apply : global
46+
metrics :
47+
- acc
48+
49+
locally_tuned_model_validation:
50+
function : validate_task
51+
kwargs :
52+
batch_size : 32
53+
apply : local
54+
metrics :
55+
- acc
56+
57+
train:
58+
function : train_task
59+
kwargs :
60+
batch_size : 32
61+
metrics :
62+
- loss
63+
epochs : 1
64+
4265

4366
compression_pipeline :
44-
defaults : plan/defaults/compression_pipeline.yaml
67+
defaults : plan/defaults/compression_pipeline.yaml

‎openfl-workspace/tf_2dunet/requirements.txt renamed to ‎openfl-workspace/keras_2dunet/requirements.txt

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
keras==3.6.0
12
nibabel
23
setuptools>=65.5.1 # not directly required, pinned by Snyk to avoid a vulnerability
3-
tensorflow==2.13
4+
tensorflow==2.18.0

‎openfl-workspace/keras_cnn_with_compression/src/__init__.py renamed to ‎openfl-workspace/keras_2dunet/src/__init__.py

File renamed without changes.

‎openfl-workspace/tf_2dunet/src/brats_utils.py renamed to ‎openfl-workspace/keras_2dunet/src/brats_utils.py

File renamed without changes.

‎openfl-workspace/tf_2dunet/src/tfbrats_inmemory.py renamed to ‎openfl-workspace/keras_2dunet/src/dataloader.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@
33

44
"""You may copy this file as the starting point of your own model."""
55

6-
from openfl.federated import TensorFlowDataLoader
6+
from openfl.federated import KerasDataLoader
77
from .brats_utils import load_from_nifti
88

99

10-
class TensorFlowBratsInMemory(TensorFlowDataLoader):
11-
"""TensorFlow Data Loader for the BraTS dataset."""
10+
class KerasBratsInMemory(KerasDataLoader):
11+
"""Keras Data Loader for the BraTS dataset."""
1212

1313
def __init__(self, data_path, batch_size, percent_train=0.8, pre_split_shuffle=True, **kwargs):
1414
"""Initialize.

‎openfl-workspace/tf_2dunet/src/nii_reader.py renamed to ‎openfl-workspace/keras_2dunet/src/nii_reader.py

File renamed without changes.

‎openfl-workspace/keras_2dunet/src/taskrunner.py

Lines changed: 158 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,158 @@
1+
# Copyright (C) 2020-2021 Intel Corporation
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
"""You may copy this file as the starting point of your own model."""
5+
6+
import tensorflow as tf
7+
import keras
8+
9+
from openfl.federated import KerasTaskRunner
10+
11+
12+
class Keras2DUNet(KerasTaskRunner):
13+
"""Initialize.
14+
15+
Args:
16+
**kwargs: Additional parameters to pass to the function
17+
18+
"""
19+
20+
def __init__(self, **kwargs):
21+
"""Initialize.
22+
23+
Args:
24+
**kwargs: Additional parameters to pass to the function
25+
26+
"""
27+
super().__init__(**kwargs)
28+
29+
self.model = self.build_model(self.data_loader.get_feature_shape(), use_upsampling=True, **kwargs)
30+
self.model.summary(print_fn=self.logger.info, line_length=120)
31+
self.initialize_tensorkeys_for_functions()
32+
33+
34+
def build_model(self, input_shape,
35+
use_upsampling=False,
36+
n_cl_out=1,
37+
dropout=0.2,
38+
activation_function='relu',
39+
seed=0xFEEDFACE,
40+
depth=5,
41+
dropout_at=None,
42+
initial_filters=32,
43+
batch_norm=True,
44+
**kwargs):
45+
"""Define the TensorFlow model.
46+
47+
Args:
48+
input_shape: input shape of the model
49+
use_upsampling (bool): True = use bilinear interpolation;
50+
False = use transposed convolution (Default=False)
51+
n_cl_out (int): Number of channels in input layer (Default=1)
52+
dropout (float): Dropout percentage (Default=0.2)(Default = True)
53+
activation_function: The activation function to use after convolutional layers (Default='relu')
54+
seed: random seed (Default=0xFEEDFACE)
55+
depth (int): Number of max pooling layers in encoder (Default=5)
56+
dropout_at: Layers to perform dropout after (Default=[2,3])
57+
initial_filters (int): Number of filters in first convolutional layer (Default=32)
58+
batch_norm (bool): True = use batch normalization (Default=True)
59+
**kwargs: Additional parameters to pass to the function
60+
61+
"""
62+
if dropout_at is None:
63+
dropout_at = [2, 3]
64+
65+
inputs = keras.layers.Input(shape=input_shape, name='Images')
66+
67+
if activation_function == 'relu':
68+
activation = tf.nn.relu
69+
elif activation_function == 'leakyrelu':
70+
activation = tf.nn.leaky_relu
71+
72+
params = {
73+
'activation': activation,
74+
'kernel_initializer': keras.initializers.he_uniform(seed=seed),
75+
'kernel_size': (3, 3),
76+
'padding': 'same',
77+
}
78+
79+
convb_layers = {}
80+
81+
net = inputs
82+
filters = initial_filters
83+
for i in range(depth):
84+
name = f'conv{i + 1}a'
85+
net = keras.layers.Conv2D(name=name, filters=filters, **params)(net)
86+
if i in dropout_at:
87+
net = keras.layers.Dropout(dropout)(net)
88+
name = f'conv{i + 1}b'
89+
net = keras.layers.Conv2D(name=name, filters=filters, **params)(net)
90+
if batch_norm:
91+
net = keras.layers.BatchNormalization()(net)
92+
convb_layers[name] = net
93+
# only pool if not last level
94+
if i != depth - 1:
95+
name = f'pool{i + 1}'
96+
net = keras.layers.MaxPooling2D(name=name, pool_size=(2, 2))(net)
97+
filters *= 2
98+
99+
# do the up levels
100+
filters //= 2
101+
for i in range(depth - 1):
102+
if use_upsampling:
103+
up = keras.layers.UpSampling2D(
104+
name=f'up{depth + i + 1}', size=(2, 2))(net)
105+
else:
106+
up = keras.layers.Conv2DTranspose(
107+
name='transConv6', filters=filters,
108+
kernel_size=(2, 2), strides=(2, 2), padding='same')(net)
109+
net = keras.layers.concatenate(
110+
[up, convb_layers[f'conv{depth - i - 1}b']],
111+
axis=-1
112+
)
113+
net = keras.layers.Conv2D(
114+
name=f'conv{depth + i + 1}a',
115+
filters=filters, **params)(net)
116+
net = keras.layers.Conv2D(
117+
name=f'conv{depth + i + 1}b',
118+
filters=filters, **params)(net)
119+
filters //= 2
120+
net = keras.layers.Conv2D(name='Mask', filters=n_cl_out,
121+
kernel_size=(1, 1),
122+
activation='sigmoid')(net)
123+
model = keras.models.Model(inputs=[inputs], outputs=[net])
124+
125+
126+
self.optimizer = keras.optimizers.RMSprop(1e-2)
127+
model.compile(
128+
loss=self.dice_coef_loss,
129+
optimizer=self.optimizer,
130+
metrics=["acc"]
131+
)
132+
133+
return model
134+
135+
def dice_coef_loss(self, y_true, y_pred, smooth=1.0):
136+
"""Dice coefficient loss.
137+
138+
Calculate the -log(Dice Coefficient) loss
139+
140+
Args:
141+
y_true: Ground truth annotation array
142+
y_pred: Prediction array from model
143+
smooth (float): Laplace smoothing factor (Default=1.0)
144+
Returns:
145+
float: -log(Dice cofficient) metric
146+
"""
147+
intersection = tf.reduce_sum(y_true * y_pred, axis=(1, 2, 3))
148+
149+
term1 = -tf.math.log(tf.constant(2.0) * intersection + smooth)
150+
term2 = tf.math.log(tf.reduce_sum(y_true, axis=(1, 2, 3))
151+
+ tf.reduce_sum(y_pred, axis=(1, 2, 3)) + smooth)
152+
153+
term1 = tf.reduce_mean(term1)
154+
term2 = tf.reduce_mean(term2)
155+
156+
loss = term1 + term2
157+
158+
return loss

‎openfl-workspace/keras_cnn_mnist/plan/plan.yaml

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,9 +5,9 @@ aggregator :
55
defaults : plan/defaults/aggregator.yaml
66
template : openfl.component.Aggregator
77
settings :
8-
init_state_path : save/keras_cnn_mnist_init.pbuf
9-
best_state_path : save/keras_cnn_mnist_best.pbuf
10-
last_state_path : save/keras_cnn_mnist_last.pbuf
8+
init_state_path : save/init.pbuf
9+
best_state_path : save/best.pbuf
10+
last_state_path : save/last.pbuf
1111
rounds_to_train : 10
1212

1313
collaborator :
@@ -19,15 +19,15 @@ collaborator :
1919

2020
data_loader :
2121
defaults : plan/defaults/data_loader.yaml
22-
template : src.tfmnist_inmemory.TensorFlowMNISTInMemory
22+
template : src.dataloader.KerasMNISTInMemory
2323
settings :
2424
collaborator_count : 2
2525
data_group_name : mnist
2626
batch_size : 256
2727

2828
task_runner :
2929
defaults : plan/defaults/task_runner.yaml
30-
template : src.keras_cnn.KerasCNN
30+
template : src.taskrunner.KerasCNN
3131

3232
network :
3333
defaults : plan/defaults/network.yaml
@@ -40,3 +40,7 @@ tasks :
4040

4141
compression_pipeline :
4242
defaults : plan/defaults/compression_pipeline.yaml
43+
# To use different Compression Pipeline, uncomment the following lines
44+
# template : openfl.pipelines.KCPipeline
45+
# settings :
46+
# n_clusters : 6

‎openfl-workspace/keras_cnn_mnist/requirements.txt

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,3 @@
1-
numpy==1.23.5
2-
tensorflow==2.13
1+
keras==3.6.0
2+
tensorflow==2.18.0
3+

‎openfl-workspace/keras_cnn_mnist/src/tfmnist_inmemory.py renamed to ‎openfl-workspace/keras_cnn_mnist/src/dataloader.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,12 +3,12 @@
33

44
"""You may copy this file as the starting point of your own model."""
55

6-
from openfl.federated import TensorFlowDataLoader
6+
from openfl.federated import KerasDataLoader
77
from .mnist_utils import load_mnist_shard
88

99

10-
class TensorFlowMNISTInMemory(TensorFlowDataLoader):
11-
"""TensorFlow Data Loader for MNIST Dataset."""
10+
class KerasMNISTInMemory(KerasDataLoader):
11+
"""Data Loader for MNIST Dataset."""
1212

1313
def __init__(self, data_path, batch_size, **kwargs):
1414
"""

‎openfl-workspace/keras_cnn_mnist/src/keras_cnn.py renamed to ‎openfl-workspace/keras_cnn_mnist/src/taskrunner.py

Lines changed: 9 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,12 @@
1-
# Copyright (C) 2020-2021 Intel Corporation
1+
# Copyright (C) 2020-2024 Intel Corporation
22
# SPDX-License-Identifier: Apache-2.0
33

44
"""You may copy this file as the starting point of your own model."""
55

6-
import tensorflow.keras as ke
7-
from tensorflow.keras import Sequential
8-
from tensorflow.keras.layers import Conv2D
9-
from tensorflow.keras.layers import Dense
10-
from tensorflow.keras.layers import Flatten
6+
from keras.models import Sequential
7+
from keras.layers import Conv2D
8+
from keras.layers import Dense
9+
from keras.layers import Flatten
1110

1211
from openfl.federated import KerasTaskRunner
1312

@@ -50,7 +49,7 @@ def build_model(self,
5049
num_classes (int): The number of classes of the dataset
5150
5251
Returns:
53-
tensorflow.python.keras.engine.sequential.Sequential: The model defined in Keras
52+
keras.models.Sequential: The model defined in Keras
5453
5554
"""
5655
model = Sequential()
@@ -72,14 +71,8 @@ def build_model(self,
7271

7372
model.add(Dense(num_classes, activation='softmax'))
7473

75-
model.compile(loss=ke.losses.categorical_crossentropy,
76-
optimizer=ke.optimizers.legacy.Adam(),
77-
metrics=['accuracy'])
78-
79-
# initialize the optimizer variables
80-
opt_vars = model.optimizer.variables()
81-
82-
for v in opt_vars:
83-
v.initializer.run(session=self.sess)
74+
model.compile(loss="categorical_crossentropy",
75+
optimizer="adam",
76+
metrics=["accuracy"])
8477

8578
return model

‎openfl-workspace/keras_cnn_with_compression/plan/cols.yaml

Lines changed: 0 additions & 5 deletions
This file was deleted.

‎openfl-workspace/keras_cnn_with_compression/plan/data.yaml

Lines changed: 0 additions & 7 deletions
This file was deleted.

0 commit comments

Comments
 (0)