NNSmith 0.1.0

Overview

In this release, NNSmith is made more robust, systematic and extensible. It also supports a wider range of operators, data types and frontends/backends. We upstreamed the main infrastructure improvements done during the NeuRI project (algorithmic parts will be upstreamed in incoming releases), making it "a fuzzer infrastructure" in addition to simply "a fuzzer" for DL frameworks. Open-source developers can easily build or extend their own fuzzers based on NNSmith now. The documentation and GitHub experience are also improved to make it more approachable.

Notable Changes

• Adding more ops for TF (#55)
• Add UB and vulnerable ops filter (#56)
• Resuming coverage instrumentation (#60)
• GraphIR for systematic model manipulation (#61, #64, #66)
• Adding versioning information for nnsmith/model/backend (#63)
• Concolic generation (#66)
• Sharpening and testing user-defined constraints (#72)
• Include and exclude ops by name and data types (#72)
• Adding more complete data types (#74, #76)
• Deprecate support for iree (#74)
• Supporting torchjit as backend (#79)
• Run eager mode on GPU if the backend target is CUDA (#89)
• Fixing or improving operator rules: Pool2d, InterpBase, ReduceBase (#66), Reshape (11a29ef), broadcast, Abs, ReflectPad, ArgMin/Max (#74), TFMatMul (#95)
• Adapting TF's naming convention in GraphIR's var names (#66)
• Harnessing hydra dependencies (#66, #68)
• Documentation (#57, #58, #60, bf1f592, dce784a, ba486ca)
• Fixing and supporting fp16 (#85, #89)
• Contribution guidance (#86)
• Code of conduct (#82)
• Issue templates (#80)
• Bug report links are being listed and maintained (#90)
• Defaultify to symbolic-cinit as generation mode (#96)

See the full commit list: a8307f2...b4b9fac

PyPI Release

https://pypi.org/project/nnsmith/0.1.0/

Contributors

@Co1lin @ganler @jakc4103

Thanks, @soodoshll and many others for issue reporting.

NNSmith 0.1.0-rc2

Notable Improvements

• Adding more ops for TF (#55)
• Add UB and vulnerable ops filter (#56)
• Resuming coverage instrumentation (#60)
• GraphIR for systematic model manipulation (#61, #64, #66)
• Adding versioning information for nnsmith/model/backend (#63)
• Concolic generation (#66)
• Sharpening and testing user-defined constraints (#72)
• Include and exclude ops by name and data types (#72)
• Adding more complete data types (#74, #76)
• Deprecate support for iree (#74)
• Supporting torchjit as backend (#79)
• Run eager mode on GPU if the backend target is CUDA (#89)
• Fixing or improving operator rules: Pool2d, InterpBase, ReduceBase (#66), Reshape (11a29ef), broadcast, Abs, ReflectPad, ArgMin/Max (#74), TFMatMul (#95)
• Adapting TF's naming convention in GraphIR's var names (#66)
• Harnessing hydra dependencies (#66, #68)
• Documentation (#57, #58, #60, bf1f592, dce784a, ba486ca)
• Fixing and supporting fp16 (#85, #89)
• Contribution guidance (#86)
• Code of conduct (#82)
• Issue templates (#80)

See the full commit list: a8307f2...8cee6f6

PyPI Release

https://pypi.org/project/nnsmith/0.1.0rc2/

Contributors

@Co1lin @ganler @jakc4103

Thanks, @soodoshll and many others for issue reporting.

NNSmith 0.1.0-rc1

Notable Improvements

• Adding more ops for TF (#55)
• Add UB and vulnerable ops filter (#56)
• Resuming coverage instrumentation (#60)
• GraphIR for systematic model manipulation (#61, #64, #66)
• Adding versioning information for nnsmith/model/backend (#63)
• Concolic generation (#66)
• Sharpening and testing user-defined constraints (#72)
• Include and exclude ops by name and data types (#72)
• Adding more complete data types (#74, #76)
• Deprecate support for iree (#74)
• Supporting torchjit as backend (#79)
• Run eager mode on GPU if the backend target is CUDA (#89)
• Fixing or improving operator rules: Pool2d, InterpBase, ReduceBase (#66), Reshape (11a29ef), broadcast, Abs, ReflectPad, ArgMin/Max (#74)
• Adapting TF's naming convention in GraphIR's var names (#66)
• Harnessing hydra dependencies (#66, #68)
• Documentation (#57, #58, #60, bf1f592, dce784a, ba486ca)
• Fixing and supporting fp16 (#85, #89)
• Contribution guidance (#86)
• Code of conduct (#82)
• Issue templates (#80)

See the full commit list: a8307f2...8cee6f6

PyPI Release

https://pypi.org/project/nnsmith/0.1.0rc1/

Contributors

@Co1lin @ganler @jakc4103

Thanks, @soodoshll and many others for issue reporting.

NNSmith 0.0.1

Release 0.0.1

This release mainly stabilizes the v0.0.0 (which is a huge refactor for code quality and extensibility from research code) and improves the usability of NNSmith.

Major enhancements

• Support IREE backend;
• Backend-oriented spec constraint patching and rewriting;
• Extensible filtering rules;
• Fuzzing loop crash safety and timeout oracle;
• Adaptive GPU allocation in TF;
• Remove default backend and model types -- specify them explicitly;

Bug fixes

• Fix (Fuzzing Loop) and enhance (others) determinism;
• Remove "ints" in torch.mean signature spec as it is not supported.
• Replace unstable PyTorch GEMM spec (actually addmm) with MatMul spec (torch.matmul).
• Fix tf.Variable device placement consistency requirement.
• Fix by adapting torch.sum(int32) -> int64;
• Fix time budge/constraint syntax sugar of "s/m/min/h/hr" in CLI;

Other noticeable changes

• Suppress warnings from tvm;
• Improve argument type checking;
• Use setuptools_scm to determine package versioning;
• Option to check input/output types during executing PyTorch SymbolNet;
• NNSmith LOGO;

Other releases

• PyPI: https://pypi.org/project/nnsmith/0.0.1/

Contributors

@Co1lin @ganler