Port MobileNet #2049

pkgoogle · 2025-01-17T23:15:32Z

Continuation of #1908

Updating the implementation of mobilenet as per torch, and then will add the timm conversion script to port weights from hf

Gist Example: gist

BytePairTokenizer must not split sequences of \n (keras-team#1910) * fix for loading of special tokens in Llama tokenizer * fix for Llama tokenizer which can have multiple end tokens * bug fix * adding some missing tokens to Llama3 tokenizer * fixed tests and Llama3Tokenizer init. * now loading correct eos_token config from Hugging Face checkpoint. Using hack for Keras checkpoint because it does not have this info * fix for BytePairTokenizer to make Lllama3-instruct work in chat: \n\n sequences are significant in the chat template and must be preserved by the tokenizer --------- Co-authored-by: Martin Görner <martin@huggingface.co> fix for generation that never stops in Llama3-Instruct variants (keras-team#1904) * fix for loading of special tokens in Llama tokenizer * fix for Llama tokenizer which can have multiple end tokens * bug fix * adding some missing tokens to Llama3 tokenizer * fixed tests and Llama3Tokenizer init. * now loading correct eos_token config from Hugging Face checkpoint. Using hack for Keras checkpoint because it does not have this info --------- Co-authored-by: Martin Görner <martin@huggingface.co> fix failing JAX GPU test (keras-team#1911) * fix tests * fix test Refactor `MMDiT`, add `ImageToImage` and `Inpaint` for SD3 (keras-team#1909) * Refactor `MMDiT` and add `ImageToImage` * Update model version * Fix minor bugs. * Add `Inpaint` for SD3. * Fix warnings of MMDiT. * Addcomment to Inpaint * Simplify `MMDiT` implementation and info of `summary()`. * Refactor `generate()` API of `TextToImage`, `ImageToImage` and `Inpaint`. Minor bug fix (keras-team#1915) Change to image_converter.image_size since it is a tuple and it's not a callable function. [Mix Transformer] Add Presets for MiTB0...MiTB5 (keras-team#1893) * add presets for mit * add standin paths * register presets in __init__.py * fix op in overlapping patching and embedding, start adding conversion utils * style * add padding to MiT patchingandembedding * update to support other presets * update conversin script * fix link for b5 * add cityscapes weights * update presets * update presets * update conversion script to make directories * use save_preset * change name of output dir * add preprocessor flow * api gen and add preprocessor to mits * conform to new image classifier style * format * resizing image converter -> ImageConverter * address comments refactoring remove default resizing for vision backbones (keras-team#1916) * remove defailt resizing * fix GPU test Update VGG model to be compatible with HF and add conversion scripts (keras-team#1914) Deeplab presets (keras-team#1918) * add preset configurations for deeplabv3 * fix uri * Add training details update presets to point to the main Keras Kaggle page (keras-team#1921) * update presets to point to the main keras page * update mit path Added test for the way BytePairTokenizer handles the \n\n sequence, which is important in Lama chat templates (keras-team#1912) * added test for the way BytePairTokenizer handles the \n\n sequence, which is important in Lama chat templates * un commented the test lines that were commented by mistake * fixed linter errors Task models fix (keras-team#1922) * added test for the way BytePairTokenizer handles the \n\n sequence, which is important in Lama chat templates * fix for wrongly configured task models LLama, PaliGemma, Mistral and Phi3 + test * comments * un commented the test lines that were commented by mistake * fixed linter errors adding option strip_prompt to generate() (keras-team#1913) * added test for the way BytePairTokenizer handles the \n\n sequence, which is important in Lama chat templates * un commented the test lines that were commented by mistake * fixed linter errors * added options strip_prompt to generate() * fix for tensorflow: the compiled version of generate(strip_prompt=True) now works + code refactoring to make it more understandable * added test for generate(strip_prompt=True) * minor edits Layout map for Llama (keras-team#1923) * added test for the way BytePairTokenizer handles the \n\n sequence, which is important in Lama chat templates * un commented the test lines that were commented by mistake * fixed linter errors * added default layout map for Llama * minor fixes in tests Update deeplab_v3_presets.py (keras-team#1924) Add paths to get SAM weights from (keras-team#1925) Two fixes for image resizing in preprocessing (keras-team#1927) 1. Properly display when are not resizing the input image in `model.summary()` 2. Allow setting the `image_size` directly on a preprocessing layer. 2. is just to allow a more consistent way to set the input shape across tasks. We now have: ```python text_classifier = keras_hub.models.TextClassifer.from_preset( "bert_base_en", ) text_classifier.preprocessor.sequence_length = 256 image_classifier = keras_hub.models.TextClassifer.from_preset( "bert_base_en", ) image_classifier.preprocessor.image_size = (256, 256) multi_modal_lm = keras_hub.models.CausalLM.from_preset( "some_preset", ) multi_modal_lm.preprocessor.sequence_length = 256 multi_modal_lm.preprocessor.image_size = (256, 256) ``` add back default image resizing (keras-team#1926) Update deeplab_v3_presets.py (keras-team#1928) * Update deeplab_v3_presets.py * Update deeplab_v3_presets.py Update PaliGemma to remove `include_rescaling` arg (keras-team#1917) * update PaliGemma * update conversion script * fix GPU tests fix path (keras-team#1929) * fix path * nit Fix paligemma checkpoint conversion script (keras-team#1931) * add back default image resizing * fix bug in image converter * fix paligemma checkpoint conversion file * fix preset name * remove debug code * revert unintended changes update preset path to point to latest version of models (keras-team#1932) Update sdv3 path (keras-team#1934) update sam docstring to show correct backbone in docstring (keras-team#1936) Convert input dict to tensors during train_on_batch (keras-team#1919) Register VGG presets. (keras-team#1935) * register vgg preset * nit * nit * nit Add ResNetVD presets (keras-team#1897) * Add ResNetVD presets * Updated Kaggle handles * Add weight conversion script for ResNet_vd * Add usage rebase conflict resolved conflict resolve Update sam_presets.py (keras-team#1940) Update vit_det_backbone.py (keras-team#1941) fix gpu test (keras-team#1939) * fix gpu test * cast input * update dtype * change to resnet preset * remove arg Added Support for Returning Attention Scores in TransformerEncoder call (keras-team#1879) * Added: Return attention scores argument to transformer encoder * Added: docstring for return_attention_scores and added a test to chek the working of the argument * Fixed: Test case by removing print stmts and using self.assertAllEqual * Fixed: Linting Mark preset tests as large (keras-team#1942) * fix tests * fix test * Update preset_utils_test.py version bump to 0.17.0.dev0 (keras-team#1944) Update stable_diffusion_3_presets.py (keras-team#1946) [Semantic Segmentation] - Add SegFormer Architecture, Weight Conversion Script and Presets (keras-team#1883) * initial commit - tf-based, kcv * porting to keras_hub structure - removing aliases, presets, etc. * enable instantiation of segformer backbone with custom MiT backbone * remove num_classes from backbone * fix input * add imports to __init__ * update preset * update docstrings * add basic tests * remove redundant imports * update docstrings * remove unused import * running api_gen.py * undo refactor of mit * update docstrings * add presets for mit * add standin paths * add presets for segformer backbone * register presets in __init__.py * addressing comments * addressing comments * addressing comments * update most tests * add remaining tests * remove copyright * fix test * override from_config * fix op in overlapping patching and embedding, start adding conversion utils * style * add padding to MiT patchingandembedding * update to support other presets * update conversin script * fix link for b5 * add cityscapes weights * update presets * update presets * update conversion script to make directories * use save_preset * change name of output dir * add preprocessor flow * api gen and add preprocessor to mits * conform to new image classifier style * format * resizing image converter -> ImageConverter * merge mit branch into segformer branch * add preprocessor and converter * address comments * clarify backbone usage * add conversion script * numerical equivalence changes * fix numerical inaccuracies * update conversion script * update conversion script * remove transpose * add preprocessor to segformer class * fix preset path * update test shape * update presets * update test shape * expand docstrings * add rescaling and normalization to preprocessor * remove backbone presets, remove copyrights, remove backbone cls from segmenter * remove copyright and unused import * apply same transformation to masks as input images * fix import * fix shape in tests Update readme (keras-team#1949) * Update README.md * Update README.md Update llama_backbone.py docstring (keras-team#1950) Update path (keras-team#1953) Update preset path for keras.io. There is no LLaMA2 in keras.io https://keras.io/api/keras_hub/models/llama2 This is the actual link: https://keras.io/api/keras_hub/models/llama2 For Vicuna it does not have it's own model direcotry, since it is also the part of Llama,, updated the path. Update SD3 init parameters (replacing `height`, `width` with `image_shape`) (keras-team#1951) * Replace SD3 `height` and `width` with `image_shape` * Update URI * Revert comment * Update SD3 handle * Replace `height` and `width` with `image_shape` * Update docstrings * Fix CI Update docstring (keras-team#1954) AudioConverter is registered as "keras_hub.layers.WhisperAudioConverter" and not as part of models. updated Mobilenet backbone to match it with torch implementation timm script added checkpoint conversion added Refactoring

divyashreepathihalli · 2025-01-21T21:53:23Z

keras_hub/src/models/mobilenet/conv_bn_act_block.py

@@ -0,0 +1,57 @@
+import keras
+
+BN_EPSILON = 1e-5


lets move all of the blocks and layer implementations into the mobilenet_backbone.py file

divyashreepathihalli · 2025-01-21T21:54:08Z

keras_hub/src/models/mobilenet/conv_bn_act_block.py

+
+
+class ConvBnActBlock(keras.layers.Layer):
+    def __init__(


add docstring

divyashreepathihalli · 2025-01-21T21:54:36Z

keras_hub/src/models/mobilenet/depthwise_conv_block.py

+    batch normalization and relu6 activation.
+
+    Args:
+        x: Input tensor of shape `(rows, cols, channels)


the docstring is not matching the init args

divyashreepathihalli · 2025-01-21T21:55:23Z

keras_hub/src/models/mobilenet/inverted_residual_block.py

+    """An Inverted Residual Block.
+
+    Args:
+        expansion: integer, the expansion ratio, multiplied with infilters to


the doc string is not matching the init args

divyashreepathihalli · 2025-01-21T21:56:52Z

keras_hub/src/models/mobilenet/mobilenet_backbone.py


-BN_EPSILON = 1e-3
-BN_MOMENTUM = 0.999
+BN_EPSILON = 1e-5


where are the new values from?

1e-5 is the Pytorch default, since we are porting a TIMM Pytorch implementation the weights don't work unless they match the original setting here. Momentum is only relevant during training so I decided to "just leave it", but presumably if they don't match the TIMM version the models will start to diverge when training/fine-tuning on the same data.

divyashreepathihalli · 2025-01-21T21:58:55Z

keras_hub/src/models/mobilenet/mobilenet_backbone.py

-        stackwise_activation: list of activation functions, for each inverted
-             residual block in the model.
+        stackwise_activation: list of list of activation functions, for each
+            inverted residual block in the model.
        image_shape: optional shape tuple, defaults to (224, 224, 3).
        depth_multiplier: float, controls the width of the network.


depth_multiplier arg has been removed in init args. In general can you please check all the init args and docstring to make sure they are consistent?

divyashreepathihalli

Thanks for the PR @pkgoogle Can you please add a demo colab to verify the outputs?

Thanks!

divyashreepathihalli · 2025-01-21T23:52:21Z

keras_hub/src/models/mobilenet/conv_bn_act_block.py

+            use_bias=False,
+            name=f"{name}_conv",
+        )
+        self.bn = keras.layers.BatchNormalization(


bn->batch_normalization

divyashreepathihalli · 2025-01-21T23:54:09Z

keras_hub/src/models/mobilenet/depthwise_conv_block.py

+
+
+class DepthwiseConvBlock(keras.layers.Layer):
+    """


add a one like comment and then a description block like this
"""Implementation od DepthWiseConvolution block.

add more description here...
...

Args:
Example:
etc
"""

I wasn't quite sure what the ask was here.. did you want me to add Examples to all doc strings? Please review the current state and let me know. Thanks.

divyashreepathihalli · 2025-01-21T23:55:08Z

keras_hub/src/models/mobilenet/depthwise_conv_block.py

+            of the convolution along the width and height.
+            Can be a single integer to specify the same value for
+            all spatial dimensions. Specifying any stride value != 1 is
+            incompatible with specifying any `dilation_rate` value != 1.


what dilation_rate?

This is technically true but since we don't expose dilation_rate in the block constructor this never matters as dilation rate is defaulted to 1 for all conv2d layers. So I removed the reference to it now.

divyashreepathihalli · 2025-01-21T23:56:01Z

keras_hub/src/models/mobilenet/depthwise_conv_block.py

+            Can be a single integer to specify the same value for
+            all spatial dimensions. Specifying any stride value != 1 is
+            incompatible with specifying any `dilation_rate` value != 1.
+        block_id: Integer, a unique identification designating the block number.


block_id not used. besides name is what we will be using for the layer

done (removed)

divyashreepathihalli · 2025-01-21T23:56:35Z

keras_hub/src/models/mobilenet/depthwise_conv_block.py

+        filters,
+        kernel_size=3,
+        stride=2,
+        se=None,


what is se? lets keep the variable name complete and not abbreviations squeeze_excite

divyashreepathihalli · 2025-01-21T23:58:49Z

keras_hub/src/models/mobilenet/inverted_residual_block.py

+        filters,
+        kernel_size,
+        stride,
+        se_ratio,


divyashreepathihalli · 2025-01-21T23:59:28Z

keras_hub/src/models/mobilenet/conv_bn_act_block.py

+            momentum=BN_MOMENTUM,
+            name=f"{name}_bn",
+        )
+        self.act = keras.layers.Activation(activation)


divyashreepathihalli · 2025-01-22T00:01:02Z

keras_hub/src/models/mobilenet/inverted_residual_block.py

+            use_bias=False,
+            name=f"{name}_conv3",
+        )
+        self.bn3 = keras.layers.BatchNormalization(


divyashreepathihalli · 2025-01-22T00:04:07Z

keras_hub/src/models/mobilenet/inverted_residual_block.py

+
+    def build(self, input_shape):
+        if self.name is None:
+            self.name = keras.backend.get_uid("block0")


you should build the layers in this method and set self.built= True

divyashreepathihalli · 2025-01-22T00:05:53Z

keras_hub/src/models/mobilenet/mobilenet_backbone_test.py

+            init_kwargs=self.init_kwargs,
+            input_data=self.input_data,
+            expected_output_shape=(2, 7, 7, 288),
+            run_mixed_precision_check=False,


run tests with these set to True

I have set it to this... I'm running into a very weird issue when it tries to create a revitalized model. (The initial construction is successful and looks as expected).

pytest keras_hub/src/models/mobilenet/mobilenet_backbone_test.py

output:

_______________________________________________________________________________________________________________________________________________________________ MobileNetBackboneTest.test_backbone_basics _______________________________________________________________________________________________________________________________________________________________ self = <keras_hub.src.models.mobilenet.mobilenet_backbone_test.MobileNetBackboneTest testMethod=test_backbone_basics> def test_backbone_basics(self): > self.run_vision_backbone_test( cls=MobileNetBackbone, init_kwargs=self.init_kwargs, input_data=self.input_data, expected_output_shape=(2, 7, 7, 288), run_mixed_precision_check=True, run_data_format_check=False, ) keras_hub/src/models/mobilenet/mobilenet_backbone_test.py:62: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ keras_hub/src/tests/test_case.py:517: in run_vision_backbone_test self.run_backbone_test( keras_hub/src/tests/test_case.py:445: in run_backbone_test self.run_serialization_test(backbone) keras_hub/src/tests/test_case.py:282: in run_serialization_test revived_instance = cls.from_config(cfg) keras_hub/src/models/backbone.py:115: in from_config return cls(**config) keras_hub/src/models/mobilenet/mobilenet_backbone.py:698: in __init__ self.output_shape = keras.ops.shape(x) keras_hub/src/models/backbone.py:71: in __setattr__ return super().__setattr__(name, value) ../../miniforge3/envs/keras_migration/lib/python3.10/site-packages/keras/src/layers/layer.py:1449: in __setattr__ return super().__setattr__(name, value) ../../miniforge3/envs/keras_migration/lib/python3.10/site-packages/keras/src/backend/tensorflow/trackable.py:28: in __setattr__ super().__setattr__(name, value) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <[AttributeError("'MobileNetBackbone' object has no attribute 'name'") raised in repr()] MobileNetBackbone object at 0x15f258040>, name = 'output_shape', value = (None, 7, 7, 288) def __setattr__(self, name, value): """Support self.foo = trackable syntax.""" try: if getattr(self, name) is value: # Short circuit for `self.$x = self.$x`. return except AttributeError: pass if getattr(self, "_self_setattr_tracking", True): value = data_structures.sticky_attribute_assignment( trackable=self, value=value, name=name) > super(AutoTrackable, self).__setattr__(name, value) E AttributeError: can't set attribute 'output_shape' ../../miniforge3/envs/keras_migration/lib/python3.10/site-packages/tensorflow/python/trackable/autotrackable.py:70: AttributeError ======================================================================================================================================================================== short test summary info ========================================================================================================================================================================= FAILED keras_hub/src/models/mobilenet/mobilenet_backbone_test.py::MobileNetBackboneTest::test_backbone_basics - AttributeError: can't set attribute 'output_shape'

I had to add the output shape to resolve a different error. Any idea on how to move past this?

I should note, the conversion workflow is fine and the colab linked up there also works fine.

I checked all the remaining test errors, they all stem from this one actually.

divyashreepathihalli · 2025-01-22T00:07:40Z

formatting needs updating, pip install -U ruff and reformat

pkgoogle · 2025-01-24T00:29:41Z

formatting needs updating, pip install -U ruff and reformat

The remaining formatting issue is unsatisfiable, if I correct the long line -- there is no syntax which is accepted by the format checker. So I decided to keep the long line as that already exists in the repo.

ushareng and others added 21 commits October 1, 2024 03:41

kaggle weights

0dd12dd

Merge branch 'keras-team:master' into mobilenet_weights

304ff63

updated Mobilenet backbone to match it with torch implementation

d359d51

Deleted presets

966384d

Mobilenet preset deleted

38fa5d3

code reformat

16d81ba

padding changed

72982b8

downsample_padding

78411c9

typo fixed

e859fb4

timm script added

35217ba

checkpoint conversion added

2e1e9c0

preset added

c40088d

rebase done

2d5568d

code formatting

9f4c7a3

Merge branch 'master' into mobilenet_weights

23dce2a

preset path updated

eb90095

WIP mobilenet fixes, subblock refactoring

7cbfebc

WIP refactored, classifier/task changes

cfe4a4f

matched mobilenetv3 inference, working now

a5a0bb3

Merge branch 'master' into port_mobilenet

2413f1e

pkgoogle requested a review from divyashreepathihalli January 17, 2025 23:25

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 17, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 17, 2025

format pass

cc72ba4

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 17, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 17, 2025

actual format pass

c4214d5

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 17, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 17, 2025

fix import

3c8cbd5

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 18, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 18, 2025

update test, attempting to fix format issue

d1cebdf

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 21, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 21, 2025

fix format back to original style

5eb4743

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 21, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 21, 2025

divyashreepathihalli reviewed Jan 21, 2025

View reviewed changes

divyashreepathihalli requested changes Jan 21, 2025

View reviewed changes

divyashreepathihalli reviewed Jan 22, 2025

View reviewed changes

pkgoogle added 6 commits January 23, 2025 13:51

review updates, format fixes etc.

433db39

update fix DepthwiseConvBlock args

77784e8

implement compute output shape for squeeze_and_excite layer

8146370

update arguments to IR Block

42e7901

explicitly build head before transfer

19b8f2d

updates, fixes to ensure colab workflow works

330e1ab

pkgoogle added the kokoro:force-run Runs Tests on GPU label Jan 24, 2025

kokoro-team removed the kokoro:force-run Runs Tests on GPU label Jan 24, 2025

Port MobileNet #2049

Are you sure you want to change the base?

Port MobileNet #2049

Conversation

pkgoogle commented Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

divyashreepathihalli Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

divyashreepathihalli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pkgoogle Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pkgoogle Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

divyashreepathihalli commented Jan 22, 2025

pkgoogle commented Jan 24, 2025

pkgoogle commented Jan 17, 2025 •

edited

Loading

divyashreepathihalli Jan 21, 2025 •

edited

Loading

pkgoogle Jan 24, 2025 •

edited

Loading

pkgoogle Jan 24, 2025 •

edited

Loading